waggle-os/docs/research/PAPER-2-CONCEPT_gepa-evolution.md

Paper 2 — Closed-Loop Prompt Evolution: Can Smaller Models Match Flagships?

Status: CONCEPT — section skeleton + key claims. Full write-up after v2 experiment ($200 budget approved). Target: arXiv cs.AI preprint → ICLR / NeurIPS workshop Authors: Marko Markovic (Egzakta Group) Drafted: 2026-04-15

---

Thesis

A closed-loop prompt-evolution system — combining GEPA (Genetic-Pareto prompt optimization) with schema-level behavioral constraints — can produce prompts for smaller open-weight models (Gemma 4 31B) that match or exceed the performance of flagship commercial models (Opus 4.6) on domain-specific tasks, at 10-50x lower inference cost.

---

Section Skeleton

1. Abstract (~250 words)

• Problem: organizations pay flagship-model prices for tasks where a tuned smaller model suffices
• Approach: closed-loop system that traces agent behavior → builds eval datasets → evolves prompts → gates deployments → deploys to production
• v1 result: 108.8% C/A ratio (Gemma 4 + evolved prompt vs raw Opus 4.6, n=10, 4 blind judges)
• v2 design: n=60, 3 domains, 4-judge multi-vendor pool, bootstrap CI, pre-committed negative-result publication
• Placeholder: insert v2 numbers after experiment

2. Introduction

• The cost problem: flagship models at $15-75/M tokens for tasks that don't require frontier reasoning
• The prompt-engineering bottleneck: human prompt engineers are expensive and iterate slowly
• Our approach: automated prompt evolution with production-grade safety gates
• Key insight: the evolution system runs on production traces, not synthetic data — the prompts are evolved against real user behavior

3. Background and Related Work

• GEPA (Agrawal et al., arXiv:2507.19457, ICLR 2026 Oral): Genetic-Pareto prompt evolution, beats RL (GRPO) by +6% avg / +20% max with ≤35x fewer rollouts, beats MIPROv2 by >10%. Integrated into DSPy 3.0.
• ACE (Zhang et al., arXiv:2510.04618, Stanford/SambaNova): Agentic Context Engineering — closest public analog to our EvolveSchema approach for behavioral-spec mutation
• Reflection 70B (Shumer, Sept 2024): cautionary tale — "small beats big" claims require extreme rigor
• DSPy (Khattab et al.): programmatic prompt optimization framework
• TextGrad (Yuksekgonul et al.): gradient-based prompt optimization
• Our positioning: we're not a prompt optimizer — we're a closed-loop production system that happens to use prompt evolution as one component

4. System Architecture

• 6-stage pipeline (ASCII diagram):

  Traces → Dataset → Compose → Gates → Deploy → Monitor
           ↑                                    ↓
           └────────── Feedback Loop ───────────┘
  

• TraceRecorder: captures every agent turn (model, tokens, tools, human action)
• EvalDatasetBuilder: builds train/test splits from production traces
• LLM-as-Judge: multi-vendor blind judging with rubric
• GEPA + EvolveSchema composition: GEPA mutates the prompt, EvolveSchema mutates behavioral rules, feedback separation ensures they don't interfere
• Constraint gates: 4 categories (safety, quality, cost, behavioral) with configurable thresholds
• Evolution deploy: atomic persona override + behavioral-spec override with backup/rollback
• Boot-time merge: accepted overrides merge into active behavioral spec on server start

5. The Evolution Pipeline in Detail

5.1 Trace Collection

• Automatic via wireAgentLoopCallbacks — every tool call, every model response
• Stored in execution_traces table with session scoping

5.2 Dataset Construction

• Group traces by tool patterns (e.g., "save_memory → search_memory" sequences)
• Score by outcome quality (human approval rate, task completion)
• Split: 50/50 train/test with reproducible seed

5.3 GEPA Iterative Evolution

• Population of prompt candidates
• Pareto frontier across objectives (quality, cost, length)
• Iteration cap: 500 iterations OR cost ceiling
• Early-abort: plateau detection (50 consecutive iterations below threshold)

5.4 EvolveSchema Integration

• Behavioral-spec mutations: add/remove/modify rules
• Feedback separation: GEPA feedback on prompt quality vs EvolveSchema feedback on behavioral compliance
• Compose: merge prompt mutations and schema mutations into a single candidate

5.5 Constraint Gates

• Safety gate: injection scan score must not degrade
• Quality gate: judge scores must meet minimum threshold
• Cost gate: token count must not exceed budget
• Behavioral gate: all critical behavioral rules must pass

5.6 Deploy and Monitor

• Atomic file writes with backup/rollback
• Hot-reload via event emission (behavioral-spec:reloaded)
• Production monitoring via the same TraceRecorder

6. Experimental Design

6.1 v1 (completed, preliminary)

• Setup: 10 coder questions, Gemma 4 31B + Waggle-evolved prompt vs raw Opus 4.6
• Judges: 4 blind judges (Opus, Sonnet, Haiku, GPT-4o)
• Result: C/A ratio 108.8% — evolved-Gemma beat raw-Opus on per-judge mean
• Limitations: n=10, one domain, eval=train, Opus-as-judge bias, no CI
• Placeholder: v1 results table

6.2 v2 (designed, budget approved, not yet run)

• Dataset: 60 examples (30 train + 30 test), 3 domains (writer/analyst/researcher), 10 per domain per stratum
• Arms:
  • A: raw Opus 4.6 (flagship reference)
  • B: Gemma 4 + human-engineered prompt (100 tokens)
  • C: Gemma 4 + GEPA-evolved prompt (from training run)
• Judges: 4-judge multi-vendor pool:
  • Sonnet 4.6 (Anthropic)
  • Haiku 4.5 (Anthropic)
  • GPT-5 (OpenAI)
  • Gemini 2.5 Pro (Google)
• Rotation: each test example judged by all 4, randomized arm-letter assignment
• Statistics: bootstrap 95% CI + permutation test at alpha=0.05
• Hypotheses:
  • H1: C/A >= 0.95 for 2+ of 3 domains
  • H2: C/A >= 1.00 for 1+ domain (replicates v1 headline)
  • H3: C/A < 0.90 for 2+ domains → publish negative (pre-committed)
• Budget: $200 hard cap ($80 training + $80 evaluation + $40 buffer)
• Placeholder: v2 results table, per-domain breakdown, judge agreement, CI intervals

7. Results (TO BE COMPLETED)

• Placeholder: v2 primary results table
• Placeholder: per-domain breakdown
• Placeholder: inter-judge agreement (Krippendorff's alpha)
• Placeholder: bootstrap CI visualization
• Placeholder: training curve (score per iteration)
• Placeholder: prompt length analysis (tokens added vs quality gained)
• Placeholder: cost analysis (evolution cost vs inference savings)

8. Discussion

• What the results mean for the cost-quality tradeoff
• When evolution works (domain-specific, well-defined tasks) vs when it doesn't (open-ended reasoning)
• The role of judge diversity in credibility
• Production implications: how often to re-evolve, drift detection

9. Limitations

• v1 sample size (n=10) is underpowered
• Single-organization traces (Waggle users, not a general population)
• GEPA is domain-specific — global prompts may not generalize across all task families
• Judge-based evaluation inherits judge biases
• No human evaluation (judges are all LLMs)
• Evolution cost is amortized but non-trivial ($80-150 per run)

10. Reproducibility

• Split seed committed to repo
• GEPA config, judge prompts, dataset — all published
• Raw results JSON available
• Cost tracking via CostTracker

11. Conclusion

• Closed-loop evolution is a viable alternative to paying flagship prices
• The system is production-grade, not a research prototype
• Pre-committed negative-result publication builds trust

---

Key Claims (must be defensible with data)

#	Claim	Evidence needed	Status
1	Evolved Gemma 4 matches Opus 4.6 (C/A >= 0.95)	v2 experiment (30 test, 4 judges)	PLANNED
2	Evolution cost ($80-150) is amortized over thousands of inferences	Cost analysis	PLANNED
3	Multi-vendor judges agree on ranking (alpha > 0.6)	Inter-judge agreement stats	PLANNED
4	Constraint gates prevent quality regressions	Gate pass/fail rates from training	PLANNED
5	System handles negative results gracefully	Pre-committed H3 publication	DESIGNED
6	Evolution converges within 500 iterations	Training curve analysis	PLANNED

---

What Needs to Happen Before Full Write-Up

1. Pre-flight checklist — verify API keys for all 4 judges + Gemma endpoint
2. Write-up skeleton — abstract + methods with [X.X%] placeholders (this document)
3. Run v2 training — GEPA on 30 train examples ($80 budget)
4. Run v2 evaluation — 30 test x 3 arms x 4 judges ($80 budget)
5. Statistical analysis — bootstrap CI, permutation test, inter-judge agreement
6. Fill in results — tables, charts, per-domain breakdown
7. External reviewer pass — one ML peer reads methods + results
8. Publish — arXiv preprint + reproducibility repo