waggle-os/docs/plans/HERMES-40-PREREG-2026-05-19.md

Pre-Registration — Hermes "~40% faster" closed-loop claim (R6)

Date: 2026-05-19 · Status: LOCKED before any LLM spend · Repo @ c87e5b7 Discipline: research eval — strict pre-registration, no revisit (feedback_production_vs_research_cost_discipline.md). Any post-data change = documented amendment with rationale, never a silent edit. This file's committed content is the contract.

1. Claim under test

Rubric line 7 / D1: a Waggle agent that has autonomously distilled a reusable skill from a successful complex task completes a similar later task materially faster than a fresh instance — Hermes Agent's benchmarked ~40% faster on research tasks. We test Waggle's now-wired R1 loop (planSkillDistillation seam c87e5b7 + behavioral rule + real create_skill).

2. Metrics (locked)

• Primary: tool-calls to a grader-correct completion = AgentResponse.toolsUsed.length.
• Secondary (reported, not gating): assistant turns; total tokens (input+output).
• A faster wrong answer does not count — speed is measured only among correct completions (§6 grader).

3. Pre-registered success criterion (locked)

Paired unit = a "second similar task" t_b run twice: once with no skill (baseline), once with the family's distilled skill in context (treatment). Per pair, reduction r = (tc_baseline − tc_treatment) / tc_baseline, counted only when both runs are grader-PASS.

• PROVEN ⇔ escalated N=20 run has median r ≥ 0.40 AND a one-sided sign test (H0: P(treatment<baseline) ≤ 0.5; H1: >0.5) over PASS–PASS pairs with p < 0.05 (ties dropped; exact binomial).
• Anything else after escalation = NOT-PROVEN (report effect size + CI honestly).

4. Model (pinned)

qwen/qwen3-30b-a3b-instruct-2507 via OpenRouter (key hydrated from VaultStore, as prompt-assembler-v5-eval.ts). Temperature 0 (determinism where supported). Mandatory pre-run slug probe (trivial call, maxTokens=8): if the model is unreachable the run ABORTS — no fallback substitution (v5-eval deviation policy). Within-model paired design ⇒ absolute model competence does not bias the relative effect.

5. Arms (per task family i)

1. A0 — distill source: fresh Orchestrator, fixed tool set, empty skill scope, task t_a (a real ≥5-tool research task). Run real runAgentLoop. If grader-PASS and ≥5 tool calls → the R1 loop directive is applied and the agent authors skill_i via the real create_skill tool (faithful to the wired loop; skill .md lands in an isolated scope dir).
2. baseline_b: fresh Orchestrator, fixed tool set, empty skill scope, sibling task t_b (same family/method, different specifics). Record tool-calls/turns/tokens; grade.
3. treatment_b: fresh Orchestrator, fixed tool set, skill scope containing only skill_i (surfaced via list_skills/search_skills exactly as the product does), task t_b. Record; grade.

Pair = t_b: baseline_b vs treatment_b. The only difference is the presence of the self-distilled skill. Skill isolation is asserted at runtime (baseline scope dir empty; treatment scope dir contains exactly skill_i); a violation aborts the family.

Pre-data amendment (2026-05-19, before any spend): create_skill/skill_lookup are exercised as in-harness tools with byte-identical on-disk semantics to production skill-tools.ts (LLM authors the markdown; written to a per-arm scope dir; later runs discover it by reading that dir). Reason: deterministic per-arm scope isolation without coupling the eval to Orchestrator/vault/starter-skills/marketplace. The R1 distillation directive is generated by the real planSkillDistillation() (c87e5b7), tying the eval to the shipped artifact. Metric, threshold, model, N, caps, gate, and analysis (§§2–4, 8–10) are unchanged — this note records a harness-construction fidelity choice, not an outcome-affecting revision.

6. Grader (deterministic, code-based)

Each t_b ships a required-facts checklist (string/regex must-appear in the final answer — specific file.ts names + specific facts). PASS = all required facts present in the final assistant message. Only PASS–PASS pairs enter the metric. Grader is code, not a model (no LLM-judge cost/variance in the gate).

7. Task families (pre-specified; corpus = this repo @ c87e5b7 + memory substrate)

Real, ≥5-tool, reproducible (Grep/Read/recall over fixed local content; no live web):

• F1 trace-a-wired-behavior — t_a: trace how recallMemory excludes temporary; t_b: trace how the autoSave sign-gate coerces self-incapacity. Required facts: orchestrator.ts + memory-sign-gate.ts + the importance values.
• F2 audit-for-a-pattern — t_a: enumerate every != 'temporary' recall filter; t_b: enumerate every scanForInjection call site. Required facts: the specific files/paths.
• F3 summarize-a-subsystem-from-source — t_a: the evolution stack; t_b: the harvest stack. Required facts: ≥3 specific module names each.

Pilot uses F1–F3 (N=3). Powered run reuses the same 3 families × repeated sibling instances drawn from a fixed pre-listed pool (N=20 total pairs; pool enumerated in the harness, not improvised post-hoc).

8. Cost governance (hard)

CostTracker hard mode. dailyBudgetUsd = 5 pilot / 45 combined cap. Per-response tokens → USD via the pinned model's OpenRouter price (recorded in manifest output). BudgetExceededError aborts immediately — overspend is structurally impossible. Additional ceilings: maxTurns ≤ 25 per agentic run; global LLM-call ceiling; slug-probe before spend.

9. Gate — pilot → escalation (pre-registered, no discretion)

After N=3 pilot, ESCALATE to N=20 iff all:

1. median r ≥ 0.40 over pilot PASS–PASS pairs, and
2. ≥ 2 of 3 pilot families are PASS–PASS (model can actually do the tasks — guards a false-negative from model-floor), and
3. projected powered cost = (pilot_spend / 3) × 20 × 1.3 safety ≤ 40 remaining.

Else STOP → emit INCONCLUSIVE-STOPPED with pilot numbers + cost projection + recommended amendment (PM-memo pattern). No metric swap, no re-run, no threshold move.

10. Pre-registered outcomes (all valid; none hidden)

• PROVEN — escalated, §3 met. Rubric D1 may move 2→3 with this as evidence.
• NOT-PROVEN — escalated, §3 not met. Rubric D1 stays 2; record honest effect size.
• INCONCLUSIVE-STOPPED — pilot gate (§9) failed (effect <40% directional, model-floor <2/3 PASS, or cost projection >cap). Rubric D1 stays 2, flagged "directional pilot only, not proven"; manifest the projection + amendment ask.

11. Anti-p-hacking

Single confirmatory analysis (§3). No optional stopping beyond the one pre-registered gate (§9). No post-hoc family/metric selection. Nondeterminism is handled by the sign test + explicit small-N pilot caveat; the pilot is explicitly underpowered and cannot itself say "proven" (§10). Results doc reports the verdict verbatim against this file.

Amendment 2 (2026-05-19 PM, post-pilot, user-directed)

Change: pinned model §4 qwen/qwen3-30b-a3b-instruct-2507 → qwen/qwen3-30b-a3b-thinking-2507 (the project's QWEN_THINKING_MODEL).

Rationale (documented per §11, no silent revisit): Pilot 1 (39e2f49) returned INCONCLUSIVE-STOPPED with a precisely-located cause: the instruct (non-reasoning) variant did not act on the create_skill distillation directive even when the real R1 gate would fire (F3, 6 tools). The instruct variant also under-planned multi-tool work (F1/F2 resolved in <5 tools). A reasoning variant deliberates over the in-context behavioral rule before acting and plans deeper tool use — directly targeting cause #2 and improving the odds of cause #1. Decided by the user (instruction priority); this is the manifest §10 "amendment ask" being answered, not a goalpost move.

Unchanged (no p-hacking): metric (§2), success threshold + sign test (§3), N + caps + gate (§§8–9), arms + isolation + grader (§§5–6), task families + pool (§7), outcomes (§10). Only the model identity changes. Pilot 1's verbatim INCONCLUSIVE result stands in the record (HERMES-40-RESULTS-2026-05-19.md); the thinking-variant run is reported as Pilot 2 under the same locked design. Spend continues against the same $5 pilot / $45 combined caps (Pilot 1 spent $0.0048).

Amendment 3 (2026-05-19 PM, post-Pilot-2, user-directed: option A)

Change: §7 task environment only — replace the synthetic corpus + families with a forcing environment: a project-specific, non-guessable multi-file subsystem where a correct answer requires ≥8 grounded tool calls (registry→entry→chained stage files→gate→config), so the real planSkillDistillation ≥5 gate actually fires and a skill is authored. System prompt firmed to require tool-grounded evidence (corpus is fictional/project-specific → cannot be answered from model priors).

Rationale (§11): Pilots 1+2 were INCONCLUSIVE for one model-invariant reason — tasks resolved in 1–6 tool calls, below the ≥5 trigger (shipped R1 correctly gated off every time). The bottleneck is task difficulty, not model or claim. Amendment 3 fixes exactly that.

Unchanged (no p-hacking): §2 metric, §3 threshold + one-sided sign test, §§5–6 arms/isolation/grader-mechanism, §§8–9 caps + pre-registered gate, §10 outcomes, §4 model (qwen-thinking, Amdt 2). Only §7's corpus/family content changes (the grader still = "all required facts present"; required-facts are now scattered ≥1-per-file to force traversal). Pilots 1+2 stand verbatim in the record; the forcing-corpus run = Pilot 3 under the same locked machinery, same $5/$45 caps (cum spent $0.0141).

Amendment 4 (2026-05-19 PM, post-Pilot-3, user-directed: option B + harness-bug retraction)

RETRACTION: The "decisive cross-pilot finding — 30B won't autonomously self-distil (6/6→0)" recorded after Pilots 1–3 is WITHDRAWN. Root cause was a harness defect, not model behavior: the distill phase was a single runAgentLoop turn. The model correctly traced the task and emitted its final answer; the loop then exited (no tool_calls). It was never given the post-task turn in which production R1 actually distils — in prod the chat.ts seam computes planSkillDistillation after the turn and surfaces .directive into a subsequent turn. The model never declined create_skill; it was never asked at a point it could act. (The user flagged this: qwen demonstrably can call tools — every distill run passed the grader, which requires successful tool calls.) Any claim about model self-distillation propensity from Pilots 1–3 is void.

Changes: (a) harness correctness fix — two-phase distill: Phase 1 runs task_a clean (no in-turn distill rule), then Phase 2 replays production R1 (continue the conversation: task → answer → the real planSkillDistillation().directive) with create_skill available — the turn the model can actually act on. (b) Per user option B: model §4 → anthropic/claude-sonnet-4.6 (frontier agentic, OpenRouter $3/$15 per M). (c) §8 powered cap → $40 (user's ≤$40).

Unchanged (no p-hacking): §2 metric, §3 threshold + sign test, §§5–6 isolation/grader, §9 gate logic, §10 outcomes, §7 forcing corpus (Amdt 3, validated). Sequence stays T3: a fresh $5 pilot validates the fixed harness, then — only if the pre-registered gate passes — the powered N=20 under the $40 combined cap (user-authorized B). Pilots 1–3 remain in the record as harness-development history with this retraction attached.