waggle-os/docs/plans/HERMES-40-RESULTS-2026-05-19.md

# Results — Hermes "~40% faster" closed-loop claim (R6 pilot)

**Verdict: `INCONCLUSIVE-STOPPED`** (pre-registered outcome, manifest §10)
**Spend:** $0.0048 / $5 pilot cap · **Wall:** ~50s · **Model:** `qwen/qwen3-30b-a3b-instruct-2507` (OpenRouter)
**Contract:** `docs/plans/HERMES-40-PREREG-2026-05-19.md` @ `a7b844a` · **Harness:** `f9de7ae`
Reported verbatim against the manifest. No goalpost moving (§11).

## What happened

| Family | task_a tools | R1 trigger (real `planSkillDistillation`) | Skill authored? | Pair |
|---|---|---|---|---|
| F1 trace-wired-behavior | 4 | gated-off (<5, correct) | no | not formed |
| F2 audit-pattern | 2 | gated-off (<5, correct) | no | not formed |
| F3 summarize-subsystem | 6 | **would-fire** (≥5) | **no — model ignored `create_skill`** | not formed |

0/3 families PASS–PASS → pre-registered gate §9 (`passFamilies ≥ 2`) failed → **STOP**. All 3 distill-source runs *passed the code grader* (the model is genuinely agentic over the tools; grader/cost-cap/gate/sign-test machinery all functioned).

## Honest reading

The pilot did **not** measure the Hermes effect and find it absent — it **never formed a measurable pair**. The claim is **neither supported nor refuted**. Two precisely-located, distinct causes — both *experiment construction*, not evidence about the claim:

1. **Corpus too small to exercise the loop.** Tasks resolve in 2–4 tool calls, below the ≥5 distillation threshold. The *real* `planSkillDistillation` correctly returned `null` for F1/F2 (gated-off) — shipped R1 behaving exactly as specified, just under test conditions that never reach it.
2. **30B model under-complies with the meta-directive.** F3 reached 6 tools (R1 *would* fire) yet the model did not call `create_skill` despite the shipped behavioral-rule text in context. A 30B instruct model under-follows a secondary "now distil a skill" instruction.

This is the T3 tier working as designed: **$0.0048 bought the finding that the experiment is underpowered by construction**, instead of $40 on a doomed N=20.

## Effect on the rubric

D1 (closed learning loop) **stays 2** ("solid", wired + deterministically triggered + unit-proven). The Hermes "~40% faster" benchmark remains the open gap between D1=2 and a real premium D1=3 — unchanged from the R5 honest state. Nothing in this pilot lets us claim 3.

What the pilot *did* add (a real, if narrow, datum on the loop's autonomous half): a small instruct model, given the shipped behavioral distillation rule and a qualifying ≥5-tool success, did **not** self-distil. That argues the production loop's reliability depends on either model strength or a more deterministic surfacing than behavioral-prose — relevant to R5b's design (the seam emits a `step`, but authoring still depends on the model acting).

## Pre-registered amendment ask (manifest §10)

To actually measure the speed claim the experiment needs amendment (documented, user-decided per cost-discipline — not a silent re-run):
- **A. Forcing corpus** — larger/deeper corpus + tasks engineered so a correct answer *requires* ≥5–10 tool calls (reliably trips the real R1 gate).
- **B. Stronger model** — a model that complies with the `create_skill` directive (cost ↑ per the pinned-model amendment process), keeping authoring LLM-side (Hermes-faithful).
- **C. Deterministic distillation arm** — harness mechanically distils a skill from task_a's successful trace, testing reuse-speedup (claim part ii) while *separately* reporting model self-distillation compliance (claim part i). Cheapest path to a real speed number; explicitly decouples the two halves of the Hermes claim.
- **D. Stop here** — record as honestly unproven (rubric already states this); spend nothing further.

No option is taken without an explicit pre-registered amendment + (for B) a cost-cap decision.

## Pilot 3 — Amendment 3 (forcing corpus, qwen-thinking)

**Verdict: `INCONCLUSIVE-STOPPED`** · spend $0.0244 (cum **$0.0385 / $5**) · ~2m15s.

| Family | task_a tools | distill grader | R1 trigger | create_skill called? | Pair |
|---|---|---|---|---|---|
| F1 ingest→export | **5** | **PASS** | **would-fire** | **no** | not formed |
| F2 audit→ingest | **5** | **PASS** | **would-fire** | **no** | not formed |
| F3 export→audit | **5** | **PASS** | **would-fire** | **no** | not formed |

**Amendment 3 succeeded at its purpose.** The forcing corpus reliably produced genuine ≥5-tool, grader-correct successes where the real `planSkillDistillation` **would fire** (in-data, 3/3). The task-difficulty bottleneck (Pilots 1–2) is solved.

> **⚠ RETRACTED 2026-05-19 PM (Amendment 4).** The "decisive" finding below is **WITHDRAWN**. It was a **harness artifact**: the distill phase was a single `runAgentLoop` turn that ended at the model's answer (no tool_calls → loop exits), so the model was **never given the post-task turn** where production R1 actually distils (`chat.ts` seam fires `planSkillDistillation` *after* the turn → directive surfaced into a *subsequent* turn). The model never declined `create_skill` — it was never asked where it could act. qwen demonstrably *can* call tools (every distill run passed the grader, which requires tool calls). Pilots 1–3 measured an incomplete harness, not model self-distillation propensity. Fixed via two-phase distill + re-run (Amendment 4). The section is kept for history only.

## Decisive cross-pilot finding (half i of the Hermes claim) — RETRACTED, see banner above

The blocker is now isolated and **model-behavioral**: given a real qualifying success **and** the shipped behavioral distillation rule in context, the **30B model does not call `create_skill`**. Replicated across both variants and the forcing corpus: **6/6 qualifying opportunities → 0 autonomous distillations** (Pilot 1 instruct F3 @6 tools; Pilot 3 thinking @5 tools ×3).

This is a real, citable result, not a null. The Hermes claim has two halves:
- **(i) the loop autonomously distils on success** — **empirically negative on a 30B model.** The trigger is correctly wired (R5b) and *would* fire; the model simply does not act on the in-context directive. Confirms the R5b open concern verbatim: the seam emits a `step`/directive but authoring still depends on the model *acting*.
- **(ii) reuse of a distilled skill → ~40% faster** — **still unmeasured**, blocked behind (i): no skill is ever authored, so no treatment arm forms.

**Rubric impact:** D1 stays 2. New durable datum: a robust closed loop cannot depend on model goodwill to call `create_skill` — premium D1=3 likely requires the seam to *deterministically* distil (or compel it), not merely emit a directive. Directly informs a future R5b hardening.

## Decision after Pilot 3 (user-decided; no autonomous re-run)

- **B. Stronger model** — a frontier agentic model likely complies with `create_skill`; tests whether *both* halves hold. Real $ + bigger build.
- **C. Deterministic distillation arm** — harness mechanically distils a skill from task_a's PASS trace (no reliance on model volunteering), measures half (ii) directly, and separately reports half (i) = the strong negative above. Cheapest path to an actual speed number; also a prototype of the more robust production seam. *(Recommended.)*
- **D. Stop** — record as-is: half (i) empirically negative on 30B (valuable, honest), half (ii) undetermined; D1=2.

## Pilot 4 — Amendment 4 (fixed two-phase harness + sonnet-4.6) — VALID RESULT, R6 CONCLUDES

**Verdict: `INCONCLUSIVE-STOPPED`** (pre-registered gate §9.1) · spend $0.4597 (cum **~$0.50 / $5**) · ~4m45s · `anthropic/claude-sonnet-4.6`.

**The harness fix worked — this result is valid, not an artifact.** All **3/3 families formed PASS–PASS pairs**: the model traced task_a, authored a skill on the faithful post-task distill turn (production-mirroring Phase 2), and both baseline_b and treatment_b passed the grader.

| Family | tcBase | tcTreat | reduction |
|---|---|---|---|
| F1 ingest→export | 7 | 8 | **−14%** (skill *added* a lookup call) |
| F2 audit→ingest | 7 | 7 | **0%** |
| F3 export→audit | 7 | 7 | **0%** |

**median reduction = 0%**, sign-test p = 1. Gate: passFamilies ✓, cost ✓, **median ✗ (0 < 0.40)** → no escalation. The gate correctly **halted before the $40 powered run** rather than spend it confirming a null.

### Conclusion (R6, valid, final under the locked manifest)

The Hermes "~40% faster" speed claim is **NOT reproduced** in this controlled setting: with a fixed harness and a frontier agentic model, a self-distilled skill yielded **~0% median tool-call reduction** (range −14%…0%). This is a real measured negative.

**Why — and the actual finding about when the closed loop pays off:** the forcing corpus is a clean linear chain whose *optimal* path is short (~7 grounded calls). A strong model already walks it near-optimally **without** the skill, so there is no wasted exploration for a distilled recipe to eliminate (it can even cost one extra `skill_lookup`). Self-distilled skills accelerate tasks where the **baseline floundered** (dead-ends, re-derivation); they cannot speed up a task that is already a short deterministic traversal for a capable model. Hermes's ~40% presumably comes from workloads with genuine exploratory waste — not from clean, well-specified lookups.

**Rubric:** D1 **stays 2** — and is now *better characterized*: the R1 loop is wired + unit/integration-proven (R5b) and, with the fixed harness, the model **does** autonomously distil on a qualifying success (the Pilot 1–3 negative was retracted as a harness bug). The remaining gap to a premium D1=3 is not "does the loop work" but "does reuse pay off" — which is **workload-dependent**, ~0% on already-optimal tasks. Citing a flat "~40% faster" would be unsupported by this evidence.

**Cost discipline outcome:** total R6 spend ≈ $0.50 of the $5 pilot budget; the $40 powered budget was **correctly never spent** — the T3 pilot→gate design prevented a $40 confirmation of a null. R6 concludes here under the pre-registered no-revisit rule; any "tasks-with-genuine-floundering" follow-up is a *new* pre-registered experiment, user-initiated, not an autonomous re-run.

## Pilot 2 — Amendment 2 (qwen-thinking, user-directed)

**Verdict: `INCONCLUSIVE-STOPPED`** · spend $0.0093 (cumulative **$0.0141 / $5**) · ~56s · `qwen/qwen3-30b-a3b-thinking-2507`.

| Family | task_a tools | distill grader | R1 trigger | Pair |
|---|---|---|---|---|
| F1 | 2 | PASS | gated-off (<5, correct) | not formed |
| F2 | 1 | **FAIL** | gated-off | not formed |
| F3 | 2 | **FAIL** | gated-off | not formed |

Pattern *inverted* vs Pilot 1 (instruct: 4/2/6 tools, all distill-PASS, no skill authored): the thinking variant used **fewer** tool calls and failed 2/3 graders. Not a measurement bug — F1 passed the grader, so the `content` field is read correctly for the thinking model; F2/F3 were genuinely under-grounded.

## Cross-pilot conclusion (binding)

**Two pilots, two models, identical structural verdict.** The limiting factor is **experiment construction, model-invariant**: the synthetic corpus is small enough that a 30B model (reasoning or not) resolves these tasks in ≤6 tool calls — below the ≥5 distillation threshold. The real `planSkillDistillation` correctly gated-off on every family (shipped R1 working as designed; the test never reaches it). The Hermes "~40% faster" claim is **neither supported nor refuted**. **D1 stays 2.** Total spend $0.0141 of $5 — the T3 tier did its job: ~1.4 cents bought a decisive structural finding instead of $40 on a doomed powered run.

To measure the effect at all, the *task environment* must force ≥5–10 grounded tool calls (amendment A-class). That is a design change with cost implications and is a **user decision** — autonomous re-engineering + re-run would be the goalpost-moving the cost-discipline rule bans. Decision options surfaced to the user; no further spend without an explicit Amendment 3.