moving

docs/research/04-gepa-public-reveal-strategy.md

@@ -0,0 +1,392 @@
+# GEPA + ACE Self-Improvement Proof — Public Reveal Strategy
+
+**Author:** Waggle OS research series (4 of 7)
+**Drafted:** 2026-04-15 (overnight batch)
+**Context:** Waggle's v1 evolution hypothesis produced a 108.8% C/A ratio (Gemma 4 31B + Waggle-evolved prompt vs raw Opus 4.6 on 10 coder questions, 4 blind judges). V2 scales to 60 examples × 3 domains × 3 baselines with a hard train/test split. Q1-Q5 decisions are captured in `docs/hypothesis-v2-decisions.md`; execution runbook in `docs/hypothesis-v2-execution-plan.md`. This document is the **public-communication strategy** for that result — how to reveal it without it getting pattern-matched as hype.
+
+---
+
+## ⚠ Critical corrections from research-agent returns (post-draft)
+
+- **GEPA stands for "Genetic-Pareto"**, not "Goal-driven Evolution of Prompts Algorithm." Paper: Agrawal et al., arXiv:2507.19457, ICLR 2026 Oral. Core algorithm is reflective prompt evolution with Pareto frontier of candidates across objectives, beats RL (GRPO) by +6 % avg / +20 % max with ≤35× fewer rollouts, and beats MIPROv2 by >10 %. Integrated into DSPy 3.0 as `dspy.GEPA` and into MLflow prompt-opt APIs. Repo: github.com/gepa-ai/gepa.
+- **"EvolveSchema" by Mikhail" could not be pinned down.** Best candidate as a public analog: **ACE — Agentic Context Engineering** (Zhang et al., Stanford/SambaNova, arXiv:2510.04618). No Mikhail on author list. Recommend dropping the Mikhail attribution in public publication unless the original internal source is located.
+- **Gemma 4 31B** was released by Google on April 2, 2026 under Apache 2.0 and currently sits at Arena #3 open model (1452 Elo). Our v1 headline rides a very recent wave rather than fighting against it.
+- **Reflection 70B (Matt Shumer, Sept 2024)** is the canonical cautionary tale — any "small beats big" claim is pattern-matched against it. Lesson applied in §5.1 below: publish reproducibility first, headline second.
+
+---
+
+## TL;DR
+
+This is a **high-beta** moment for Waggle. A public "small model beats flagship" claim either compounds into a year of inbound enterprise leads, research credibility, and technical-brand equity — or gets shredded by a hostile HN thread and hurts Waggle's standing for 12 months afterward.
+
+The difference is **almost entirely about credibility signals.** The result itself is defensible (multi-vendor judges, no Opus self-bias, train/test split committed, negative-result publication pre-committed per Q5). The failure mode is sloppy framing, premature victory-lap tone, or skipping the rigor narrative to front-load the "mind-blown" headline.
+
+**Strategy:** ship a full arXiv-style research note *first*, then a Twitter thread + LinkedIn long-form + HN post *second* pointing at the note. The note earns the trust; the short-form captures the virality.
+
+**Non-negotiable:** reproducibility repo live on day one. Split seed, eval dataset, evolved prompt, judge prompts, raw results JSON. The first skeptic who can re-run and see the same numbers does more marketing than any post.
+
+---
+
+## 1. The claim space
+
+### 1.1 What v1 proved
+
+- **On 10 curated coder questions judged by 4 independent blind judges, Gemma 4 31B with a Waggle-evolved prompt beat raw Opus 4.6 at per-judge mean C/A ratio of 108.8%.**
+- The Opus judge itself ranked evolved-Gemma above raw-Opus — the opposite of self-preservation bias.
+- The evolved prompt added only +91 tokens over baseline.
+
+### 1.2 What v1 didn't prove
+
+Per `docs/evolution-hypothesis-v2-plan.md`:
+- n=10 is underpowered
+- one-domain (coder) coverage
+- weak baselines ("Answer clearly.")
+- eval = train (GEPA and judges saw same examples)
+- Opus-as-judge caveat (Opus is also Arm A)
+- no confidence interval or significance test
+
+### 1.3 What v2 will prove
+
+- n=60 (30 train + 30 test), stratified 3 domains × 2 strata
+- Hard train/test split, reproducible seed
+- 3 graded baselines: weak prompt / human-engineered prompt / GEPA-evolved prompt
+- 4-judge multi-vendor pool (Anthropic Sonnet + Anthropic Haiku + OpenAI GPT-5 + Google Gemini 2.5 Pro) — Opus dropped entirely
+- Bootstrap 95% CI + permutation test at α=0.05
+- **H₁:** C/A ≥ 0.95 for 2+ of 3 domains
+- **H₂:** C/A ≥ 1.00 for 1+ domain (replicates v1 headline)
+- **H₃:** C/A < 0.90 for 2+ domains → publish negative (pre-committed per Q5)
+
+This is the right rigor level for a credible publication.
+
+---
+
+## 2. Audiences and what each needs
+
+### 2.1 AI researchers (for trust / credibility)
+
+**What they need to believe it:**
+- Methods section written like an arXiv paper (methods / results / limitations / related work)
+- Reproducibility repo (dataset, split seed, judge prompts, raw scores)
+- Pre-committed negative-result publication
+- Named judges (specific model versions + dates)
+- Inter-judge agreement statistics
+
+**Channel:** arXiv preprint + Twitter thread by a named author + HN post that links to the preprint *not* to the Waggle product
+
+**Outcome if they nod:** retweets, podcast invites, academic citations in Q3-Q4 2026 follow-ups
+
+### 2.2 Technical buyers — CTOs, VPs of Engineering, AI platform leads (for sales)
+
+**What they need to believe it:**
+- The same rigor above, *and* an answer to "can this run on my infrastructure with my data?"
+- A cost comparison: inference cost of Gemma 4 31B at $X/Mtok vs Opus 4.6 at $Y/Mtok with the implicit "self-host and you save ~90%"
+- A deployment story that matches their constraints (on-prem, private-VPC, etc.)
+
+**Channel:** LinkedIn long-form post by Marko as founder; case-study PDF one-pager; direct outreach to 20-30 named targets with a personalized version
+
+**Outcome if they nod:** KVARK enterprise conversations → EUR 1.2M+ pipeline expansion
+
+### 2.3 Prompt engineers + applied ML engineers (for virality)
+
+**What they need to believe it:**
+- Side-by-side output examples (concrete, not abstract)
+- The evolved prompt visible in full (they want to copy it)
+- A reproducibility-to-the-minute guide
+- A clear "this is not magic, here's how" explanation of GEPA + ACE mechanics
+
+**Channel:** Twitter thread with screenshots; explainer blog post; YouTube walkthrough if we're ambitious
+
+**Outcome if they nod:** tens of thousands of impressions, ecosystem amplification, some try Waggle themselves
+
+### 2.4 The skeptic class (to survive)
+
+Every rigorous result gets three kinds of pushback on HN/Twitter:
+- **"They cherry-picked the questions"** → rebut with: train/test split, domain stratification, reproducibility repo with exact dataset
+- **"The judges are biased"** → rebut with: 4-vendor pool, published judge prompts, inter-judge agreement statistics
+- **"It's just prompt engineering, any tech-savvy person could do this"** → rebut with: compare against the human-engineered baseline (which is in the v2 arm list) and show that automated evolution *beats* human-engineered
+- **"Won't replicate"** → rebut with: reproducibility repo + one-line docker command to re-run
+- **"The 108.8% is within measurement noise"** → rebut with: bootstrap CI + permutation test p-value
+
+Pre-emptively address all five in the methods + limitations section. Don't wait for them to be raised; lead with them.
+
+---
+
+## 3. The research note — structure
+
+Target: ~15 pages, arXiv-style. Saves to `docs/research/evolution-hypothesis-v2-note-<date>.md` in the repo + mirror public URL at `waggle-os.ai/research/v2`.
+
+### 3.1 Abstract (200 words)
+
+> We report on a replication of Waggle OS's v1 evolution result at a 6× larger scale. In our v1 experiment (n=10, coder domain, 4 judges), Gemma 4 31B with a prompt evolved by Waggle's GEPA + ACE loop scored 108.8% of raw Claude Opus 4.6 per blind judge. To test whether this generalizes, we ran a v2 evaluation on 60 examples across 3 domains (writer, analyst, researcher), with a held-out 30-example test set, 3 graded baselines (weak / human-engineered / GEPA-evolved), and a 4-vendor judge pool (Anthropic Sonnet, Anthropic Haiku, OpenAI GPT-5, Google Gemini 2.5 Pro). We report C/A ratio of [RESULT] on the test set, 95% bootstrap CI [LO, HI], permutation test p=[P]. [H₁ / H₂ / H₃ verdict.] Full dataset, split seed, evolved prompts, judge prompts, and raw scores are available at [reproducibility repo URL].
+
+### 3.2 Introduction (1 page)
+
+- The problem: can a general-purpose evolution loop take an open-weight model to flagship quality on a user's task distribution?
+- What we built: GEPA + ACE integrated, closed-loop on execution traces
+- What we tested: v1 result replication at scale with a much tighter methodology
+- Why it matters: local-first AI systems can achieve competitive quality without customer data crossing the training-loop boundary
+
+### 3.3 Methods (3 pages)
+
+- Dataset construction (60 examples × 3 domains × 2 strata, selection criteria)
+- Split procedure (fixed seed, domain-balanced)
+- Arms (A = raw Opus 4.6, B₁ = weak prompt, B₂ = human-engineered, B₃ = GEPA-evolved; all non-A arms run on Gemma 4 31B)
+- GEPA iteration procedure (max 500 iterations or $80, early-abort on plateau)
+- Judge pool, letter-to-arm randomization per example to prevent positional bias
+- Scoring rubric (1-5 per arm per judge)
+- Statistical tests (bootstrap CI, permutation test)
+
+### 3.4 Results (3 pages)
+
+- Primary: mean per-judge C/A ratio, aggregated median across judges
+- Secondary: per-domain breakdown, per-arm raw rating distribution
+- Inter-judge agreement statistics
+- Cost / token-count per arm
+- Evolution training curve (score-per-iteration)
+
+### 3.5 Limitations (1 page)
+
+- Sample size still modest (60)
+- Domain coverage (3 out of potentially infinite task families)
+- Judge model pool (4, all commercial; no open-weight judge)
+- English-language only
+- Gemma 4 31B specifically; we don't claim generalization to other 30B-class models without additional runs
+- Time-bounded snapshot of model versions (Opus 4.6 as of April 2026)
+
+### 3.6 Related work (1 page)
+
+- GEPA original paper (cite; details from overnight research agent)
+- EvolveSchema (Mikhail et al., cite)
+- DSPy, TextGrad, OPRO, Promptbreeder, APE — positional context
+- Orca 2 / Phi-3 / small-model-beats-flagship claims — prior art
+- Multi-judge evaluation methodology papers
+
+### 3.7 Conclusion (0.5 page)
+
+- What we learned
+- What we'd run next (per-domain evolution, longer context tasks, code generation with execution verification)
+- Call to action: reproducibility repo is up, run it yourself
+
+### 3.8 Appendices
+
+- A: Full evolved prompt (B₃)
+- B: Human-engineered baseline prompt (B₂)
+- C: Judge rubric + prompts verbatim
+- D: Example questions per domain (5 per domain sampled)
+- E: Full results table (anonymized if needed)
+
+---
+
+## 4. The Twitter thread (12-18 tweets)
+
+### Example draft — tweets 1-5
+
+1. "Can a prompt-evolution loop take an open-weight 31B model past Claude Opus 4.6 on your tasks? We ran the test. (v1 replicated at 6× scale, rigorous methodology, multi-vendor judges, reproducibility repo public.) 🧵"
+
+2. "v1 result: 108.8% C/A on 10 coder questions. 4 blind judges. No self-bias. Nice, but underpowered and single-domain. Critics would rightly call that."
+
+3. "v2: 60 examples × 3 domains (writer/analyst/researcher) × 3 baselines (weak / human-engineered / GEPA-evolved). Train/test split with a fixed seed. 4-judge pool (Sonnet + Haiku + GPT-5 + Gemini 2.5). Opus *dropped* as a judge to kill the self-bias criticism dead."
+
+4. "Result: [insert headline numbers + CI + p-value]. [Verdict on H₁/H₂/H₃.]"
+
+5. "Reproducibility: [repo URL]. One command to re-run. Same seed → same result."
+
+...continuing with domain-specific numbers, the evolved prompt, a side-by-side example, credits, call to action.
+
+### Thread-writing rules
+
+- Every claim has a number or a link
+- No "mind-blowing" / "game-changing" / "insane" / any superlative that makes serious people bounce
+- Thread author is Marko — personal account of the founder, not a corporate account
+- Pin the thread
+- Quote-retweet one thoughtful skeptic's objection with a calm response
+- Engage with the thread for 48h minimum
+
+---
+
+## 5. The LinkedIn long-form (2000-3000 words)
+
+Written in Marko's voice (see memory: `marko-markovic-style` skill). Executive / product-owner tone. Audience: CTOs, VPs Eng, enterprise buyers.
+
+Outline:
+
+1. **The problem** — "We kept running into the same thing at Egzakta. Our enterprise customers loved Claude Opus's quality, but couldn't use it on sensitive data. Self-hosted open-weight models were a step behind. We wanted to close that gap *without* asking customers to send data to the cloud."
+
+2. **The hypothesis** — "What if prompt evolution — specifically an integrated GEPA + ACE loop — could take Gemma 4 31B running on a customer's infrastructure to the point where it beats raw Opus on the customer's actual tasks?"
+
+3. **The v1 experiment** — tell the story of the 10-question test, the 108.8% result, the surprised-us-too moment
+
+4. **Why we doubted it** — the 6 objections from the v2 plan
+
+5. **The v2 experiment** — methodology, arms, judges, stats
+
+6. **Results** — the numbers, with charts
+
+7. **What this means for enterprise AI** — sovereignty is compatible with flagship quality if you have a working evolution loop. Your data stays inside; the prompts evolve against it. Your open-weight model gets smarter about *your* stack than any closed flagship can be.
+
+8. **KVARK and Waggle** — this runs on your Kubernetes via KVARK; it runs on your laptop via Waggle. Same stack, different deployment.
+
+9. **Call to action** — read the research note, clone the repo, re-run the experiment. If you want this on your stack, [contact/demo link].
+
+---
+
+## 6. The HN post
+
+Title draft: "Show HN: We evolved a 31B model prompt past Claude Opus 4.6 — methodology + reproducibility repo"
+
+Why HN: HN is where a rigorous result either gets celebrated or destroyed. We need HN to celebrate, which requires:
+- **Title is honest, not hype-y.** "We evolved" not "BREAKING" not "insane result"
+- **First comment is by the author** (Marko), explaining the setup + linking the research note + acknowledging the v1 caveats openly
+- **Engage every top-level reply within 30 minutes** for first 4 hours
+- **Have named senior ML people read the note pre-publication** — not necessarily endorse, but no surprises (see §8)
+
+---
+
+## 7. Demo assets
+
+Prepare BEFORE the research note drops:
+
+| Asset | Purpose | Format |
+|---|---|---|
+| Research note PDF | arXiv-style, 15 pages | Markdown → Pandoc → PDF |
+| Reproducibility repo | Clone-and-run | `waggle-os/evolution-hypothesis-v2` GitHub repo |
+| Side-by-side example | Visceral impact | Screenshot or GIF comparing arm outputs |
+| Evolution training curve | "Here's the learning happening" | Line chart, 2D axes (iteration, score) |
+| Cost comparison | Business case | Table: arm → $/1k requests at scale |
+| Inter-judge agreement heatmap | Rigor signal | Heatmap across 4 judges × 3 arms |
+| 60-second explainer video | Social shareable | Loom or OBS screencast |
+| Evolved prompt PDF | Download-and-try | 1-page PDF with the evolved prompt verbatim |
+| Demo workspace | Try-it-yourself | Pre-seeded Waggle workspace users can import |
+
+---
+
+## 8. Pre-publication warm list
+
+Email 5-10 named people 72 hours before public post. Not for endorsement — for **no surprises**. Give them a heads-up + PDF draft + link to repo. Ask only: "Any methodological issues I should address before this goes out? I want to catch issues privately before critics catch them publicly."
+
+Suggested warm list (refined based on who owns the prompt-evolution narrative):
+
+- **Omar Khattab** (Stanford / DSPy) — the academic center of gravity for this line of work; GEPA was contributed into DSPy 3.0
+- **Lakshya Agrawal** — first author on the GEPA paper (arXiv:2507.19457, ICLR 2026 Oral); courtesy brief
+- **Krista Opsahl-Ong** — MIPROv2 author; competitor but credible gatekeeper
+- **Qizheng Zhang** / **James Zou** (Stanford/SambaNova) — ACE paper authors; relevant to the structural-evolution claim
+- **Simon Willison** (simonwillison.net) — kingmaker blog; one post moves the narrative
+- **Nathan Lambert** (Interconnects / AI2) — researcher-practitioner, high-credibility amplifier
+- **Swyx** (Latent Space) — podcast distribution if they bite
+- **Jim Fan** (NVIDIA) — high-reach amplifier for agent-world results
+- 1-2 Egzakta-network enterprise AI leaders (not at Anthropic / OpenAI / Google to avoid awkwardness)
+- 1 technical journalist who covers AI rigorously (Import AI, The Batch)
+
+Marko to cross-check against his actual network and adjust.
+
+The warm-list response often catches 1-2 non-obvious issues that would otherwise be HN-critique #1. Worth the 72 hours.
+
+---
+
+## 9. Handling criticism and follow-up
+
+### 9.1 Likely objections and ready responses
+
+(All to be drafted into an FAQ or pinned thread replies)
+
+| Objection | Response |
+|---|---|
+| "Judges are hallucinating their scores" | Inter-judge agreement stats show α = [X]. Human spot-check confirms in appendix D. |
+| "Judges are biased toward verbose answers" | B₁ (weak prompt) often produces terse answers; it did *not* win by brevity. Length not correlated with score per judge (see appendix). |
+| "Opus 4.6 is getting worse over time / not the current frontier" | We tested in [specific model version, dated]. Anthropic has not announced deprecation. |
+| "Gemma 4 31B has seen benchmark contamination" | Eval examples were hand-authored, not drawn from MMLU/HellaSwag/etc. Split seed shows this. |
+| "108.8% → 0.5% is all noise" | Bootstrap 95% CI is [LO, HI]; excludes zero / doesn't cross 1.0 for [X of Y] domains. |
+| "Why should I trust your evolution loop vs DSPy/TextGrad?" | We don't claim unique novelty in the core algorithm; we claim the *integration* (GEPA + ACE on live user traces) works at product scale. DSPy path comparison is future work. |
+| "This only works because you cherry-picked task types" | 3 domains (writer, analyst, researcher) span different reasoning styles; per-domain breakdowns in §3.4 show [XYZ]. |
+
+### 9.2 If H₃ fires (negative result)
+
+Per Q5 commitment: publish anyway. Title adjusts to "Our v2 hypothesis test did not replicate v1 — here's what we learned."
+
+Outline:
+- What we hoped
+- What we got
+- What that tells us about when evolution helps and when it doesn't
+- What we'd try next
+
+**This is worth publishing because:**
+- Research integrity → trust compounds
+- "We tried and it didn't work" is a legitimate contribution
+- Pre-committing to publish either way is itself a marketing signal (savvy buyers notice)
+
+### 9.3 If H₁ passes weakly (within noise of 0.95)
+
+Publish with the softer framing. H₂ becomes follow-up work. Set expectations honestly.
+
+### 9.4 If H₂ passes strongly (>1.0 across domains)
+
+This is the jackpot scenario. Execute the full Twitter + LinkedIn + HN triple cycle. Book a podcast circuit.
+
+---
+
+## 10. Timing and cadence
+
+**Target publication window:** within 14 days of Marko approving Q1-Q5 decisions. Per execution plan: ~4 days run + ~3 days write + 2 days external review + 2 days final polish + 2-3 days for warm-list response and surgery = ~14 days.
+
+**Rollout sequence:**
+
+- **T-72h:** warm-list emails go out
+- **T-48h:** warm-list feedback incorporated
+- **T-24h:** final research note published to repo (but not announced)
+- **T-0:** public reveal — Twitter thread + LinkedIn post + HN post all within 2-hour window
+- **T+24h:** engage every substantive reply; write FAQ update if patterns emerge
+- **T+48h:** reach out to podcast circuit / journalists who engaged
+- **T+7d:** retrospective on response, patterns, leads generated
+
+**Avoid:** publishing on Monday (HN dies by Tue) or Friday (dies over weekend). Target Tuesday or Wednesday morning US Pacific time.
+
+---
+
+## 11. Success criteria
+
+- [ ] Research note publishes within 14 days of greenlight
+- [ ] Reproducibility repo is live and has ≥1 public third-party reproduction within 14 days of publication
+- [ ] Twitter thread ≥100k impressions
+- [ ] LinkedIn post ≥20k impressions + ≥5 CTO-level comments
+- [ ] HN post reaches front page (top 30)
+- [ ] ≥2 inbound enterprise conversations within 30 days citing the note
+- [ ] ≥1 podcast invitation
+- [ ] Zero successful "methodology-busting" critiques — if there are real issues, we catch them in warm-list surgery first
+
+---
+
+## 12. Waggle's narrative after the reveal
+
+The reveal is not the destination; it's a node in a longer narrative arc:
+
+- **Q2 2026:** v2 reveal → "Waggle can make your open-weight model competitive with flagships."
+- **Q3 2026:** case study with a named enterprise pilot deployment → "Here's how Egzakta's customer X did it on-prem."
+- **Q4 2026:** follow-up paper or extended note with additional domains / models → "It keeps working at wider scope."
+- **Q1 2027:** open-source the evolution stack portions as part of the `hive-mind` OSS push (see report 01) → "Now you can run this yourself."
+- **Q2 2027:** reference-customer Anna-Ska-style testimonial + Gartner cover-quote → "Enterprise adoption is real."
+
+Every node reinforces the others. The reveal is the flywheel's first hard push.
+
+---
+
+## 13. Open decisions for Marko
+
+1. **Warm list** — draft it together before sending; I have suggestions in §8 but you have the actual relationships
+2. **Author voice** — single-author (you as founder) or dual-byline (you + one researcher)? Single is simpler; dual adds perceived rigor
+3. **Repo org** — `waggle-os/evolution-hypothesis-v2` on the existing org, or a research-specific org? I recommend the former unless a research org is planned for other papers
+4. **Podcast strategy** — opportunistic or actively pitched? I recommend opportunistic for the first appearance; if there's interest, pitch 2-3 specific shows
+5. **Localization** — publish first in English; do we also do Serbian/EU-regional spins for Egzakta's home market? Defer to you; if yes, this doubles the rollout effort
+
+If those five get quick answers, the entire plan is executable end-to-end in 14 days from greenlight.
+
+---
+
+## Closing
+
+This is Waggle's single highest-leverage moment until the next headline. The research is real; the rigor is there; the story is clear. What the reveal needs is **patience** (wait until the rigor is complete, don't front-run with v1-only framing) and **discipline** (publish the research note *first*, then short-form, not the other way around).
+
+Get this right and it accelerates KVARK enterprise pipeline, OSS `hive-mind` launch momentum, and Waggle-product consumer adoption simultaneously. Get it wrong and we spend 12 months rebuilding technical credibility.
+
+The execution plan is in `docs/hypothesis-v2-execution-plan.md`. The decisions are in `docs/hypothesis-v2-decisions.md`. This document is the publication plan.
+
+**Ready to execute on Marko's greenlight.**