97 lines
12 KiB
Markdown
97 lines
12 KiB
Markdown
# Pre-Registration — Hermes "~40% faster" closed-loop claim (R6)
|
||
|
||
**Date:** 2026-05-19 · **Status:** LOCKED before any LLM spend · **Repo @** `c87e5b7`
|
||
**Discipline:** research eval — strict pre-registration, no revisit (`feedback_production_vs_research_cost_discipline.md`). Any post-data change = documented amendment with rationale, never a silent edit. This file's committed content is the contract.
|
||
|
||
## 1. Claim under test
|
||
|
||
Rubric line 7 / D1: a Waggle agent that has autonomously distilled a reusable skill from a successful complex task completes a *similar later task* materially faster than a fresh instance — Hermes Agent's benchmarked **~40% faster on research tasks**. We test Waggle's now-wired R1 loop (`planSkillDistillation` seam `c87e5b7` + behavioral rule + real `create_skill`).
|
||
|
||
## 2. Metrics (locked)
|
||
|
||
- **Primary:** tool-calls to a *grader-correct* completion = `AgentResponse.toolsUsed.length`.
|
||
- **Secondary (reported, not gating):** assistant turns; total tokens (input+output).
|
||
- A faster *wrong* answer does not count — speed is measured only among correct completions (§6 grader).
|
||
|
||
## 3. Pre-registered success criterion (locked)
|
||
|
||
Paired unit = a "second similar task" `t_b` run twice: once with no skill (baseline), once with the family's distilled skill in context (treatment). Per pair, reduction `r = (tc_baseline − tc_treatment) / tc_baseline`, counted **only when both runs are grader-PASS**.
|
||
|
||
- **PROVEN** ⇔ escalated N=20 run has **median r ≥ 0.40** AND a **one-sided sign test** (H0: P(treatment<baseline) ≤ 0.5; H1: >0.5) over PASS–PASS pairs with **p < 0.05** (ties dropped; exact binomial).
|
||
- Anything else after escalation = **NOT-PROVEN** (report effect size + CI honestly).
|
||
|
||
## 4. Model (pinned)
|
||
|
||
`qwen/qwen3-30b-a3b-instruct-2507` via **OpenRouter** (key hydrated from `VaultStore`, as `prompt-assembler-v5-eval.ts`). Temperature **0** (determinism where supported). Mandatory pre-run **slug probe** (trivial call, maxTokens=8): if the model is unreachable the run **ABORTS** — no fallback substitution (v5-eval deviation policy). Within-model paired design ⇒ absolute model competence does not bias the *relative* effect.
|
||
|
||
## 5. Arms (per task family `i`)
|
||
|
||
1. **A0 — distill source:** fresh `Orchestrator`, fixed tool set, **empty** skill scope, task `t_a` (a real ≥5-tool research task). Run real `runAgentLoop`. If grader-PASS **and** ≥5 tool calls → the R1 loop directive is applied and the agent authors `skill_i` via the **real `create_skill`** tool (faithful to the wired loop; skill `.md` lands in an isolated scope dir).
|
||
2. **baseline_b:** fresh `Orchestrator`, fixed tool set, **empty** skill scope, sibling task `t_b` (same family/method, different specifics). Record tool-calls/turns/tokens; grade.
|
||
3. **treatment_b:** fresh `Orchestrator`, fixed tool set, skill scope containing **only `skill_i`** (surfaced via `list_skills`/`search_skills` exactly as the product does), task `t_b`. Record; grade.
|
||
|
||
Pair = `t_b`: `baseline_b` vs `treatment_b`. The only difference is the presence of the self-distilled skill. Skill isolation is asserted at runtime (baseline scope dir empty; treatment scope dir contains exactly `skill_i`); a violation aborts the family.
|
||
|
||
**Pre-data amendment (2026-05-19, before any spend):** `create_skill`/`skill_lookup` are exercised as in-harness tools with **byte-identical on-disk semantics to production `skill-tools.ts`** (LLM authors the markdown; written to a per-arm scope dir; later runs discover it by reading that dir). Reason: deterministic per-arm scope isolation without coupling the eval to Orchestrator/vault/starter-skills/marketplace. The R1 distillation directive is generated by the **real `planSkillDistillation()`** (`c87e5b7`), tying the eval to the shipped artifact. Metric, threshold, model, N, caps, gate, and analysis (§§2–4, 8–10) are unchanged — this note records a harness-construction fidelity choice, not an outcome-affecting revision.
|
||
|
||
## 6. Grader (deterministic, code-based)
|
||
|
||
Each `t_b` ships a required-facts checklist (string/regex must-appear in the final answer — specific `file.ts` names + specific facts). **PASS** = all required facts present in the final assistant message. Only PASS–PASS pairs enter the metric. Grader is code, not a model (no LLM-judge cost/variance in the gate).
|
||
|
||
## 7. Task families (pre-specified; corpus = this repo @ `c87e5b7` + memory substrate)
|
||
|
||
Real, ≥5-tool, reproducible (Grep/Read/recall over fixed local content; no live web):
|
||
|
||
- **F1 trace-a-wired-behavior** — `t_a`: trace how `recallMemory` excludes `temporary`; `t_b`: trace how the autoSave sign-gate coerces self-incapacity. Required facts: `orchestrator.ts` + `memory-sign-gate.ts` + the importance values.
|
||
- **F2 audit-for-a-pattern** — `t_a`: enumerate every `!= 'temporary'` recall filter; `t_b`: enumerate every `scanForInjection` call site. Required facts: the specific files/paths.
|
||
- **F3 summarize-a-subsystem-from-source** — `t_a`: the evolution stack; `t_b`: the harvest stack. Required facts: ≥3 specific module names each.
|
||
|
||
Pilot uses F1–F3 (N=3). Powered run reuses the same 3 families × repeated sibling instances drawn from a fixed pre-listed pool (N=20 total pairs; pool enumerated in the harness, not improvised post-hoc).
|
||
|
||
## 8. Cost governance (hard)
|
||
|
||
`CostTracker` **hard** mode. `dailyBudgetUsd = 5` pilot / **`45` combined** cap. Per-response tokens → USD via the pinned model's OpenRouter price (recorded in manifest output). `BudgetExceededError` aborts immediately — overspend is structurally impossible. Additional ceilings: `maxTurns ≤ 25` per agentic run; global LLM-call ceiling; slug-probe before spend.
|
||
|
||
## 9. Gate — pilot → escalation (pre-registered, no discretion)
|
||
|
||
After N=3 pilot, **ESCALATE to N=20** iff **all**:
|
||
1. **median r ≥ 0.40** over pilot PASS–PASS pairs, **and**
|
||
2. **≥ 2 of 3** pilot families are PASS–PASS (model can actually do the tasks — guards a false-negative from model-floor), **and**
|
||
3. projected powered cost `= (pilot_spend / 3) × 20 × 1.3 safety ≤ 40` remaining.
|
||
|
||
Else **STOP** → emit `INCONCLUSIVE-STOPPED` with pilot numbers + cost projection + recommended amendment (PM-memo pattern). No metric swap, no re-run, no threshold move.
|
||
|
||
## 10. Pre-registered outcomes (all valid; none hidden)
|
||
|
||
- **PROVEN** — escalated, §3 met. Rubric D1 may move 2→3 with this as evidence.
|
||
- **NOT-PROVEN** — escalated, §3 not met. Rubric D1 stays 2; record honest effect size.
|
||
- **INCONCLUSIVE-STOPPED** — pilot gate (§9) failed (effect <40% directional, model-floor <2/3 PASS, or cost projection >cap). Rubric D1 stays 2, flagged "directional pilot only, not proven"; manifest the projection + amendment ask.
|
||
|
||
## 11. Anti-p-hacking
|
||
|
||
Single confirmatory analysis (§3). No optional stopping beyond the one pre-registered gate (§9). No post-hoc family/metric selection. Nondeterminism is handled by the sign test + explicit small-N pilot caveat; the pilot is explicitly underpowered and cannot itself say "proven" (§10). Results doc reports the verdict verbatim against this file.
|
||
|
||
## Amendment 2 (2026-05-19 PM, post-pilot, user-directed)
|
||
|
||
**Change:** pinned model §4 `qwen/qwen3-30b-a3b-instruct-2507` → **`qwen/qwen3-30b-a3b-thinking-2507`** (the project's `QWEN_THINKING_MODEL`).
|
||
|
||
**Rationale (documented per §11, no silent revisit):** Pilot 1 (`39e2f49`) returned INCONCLUSIVE-STOPPED with a precisely-located cause: the *instruct* (non-reasoning) variant did not act on the `create_skill` distillation directive even when the real R1 gate would fire (F3, 6 tools). The instruct variant also under-planned multi-tool work (F1/F2 resolved in <5 tools). A reasoning variant deliberates over the in-context behavioral rule before acting and plans deeper tool use — directly targeting cause #2 and improving the odds of cause #1. Decided by the user (instruction priority); this is the manifest §10 "amendment ask" being answered, not a goalpost move.
|
||
|
||
**Unchanged (no p-hacking):** metric (§2), success threshold + sign test (§3), N + caps + gate (§§8–9), arms + isolation + grader (§§5–6), task families + pool (§7), outcomes (§10). Only the model identity changes. Pilot 1's verbatim INCONCLUSIVE result stands in the record (`HERMES-40-RESULTS-2026-05-19.md`); the thinking-variant run is reported as **Pilot 2** under the same locked design. Spend continues against the same $5 pilot / $45 combined caps (Pilot 1 spent $0.0048).
|
||
|
||
## Amendment 3 (2026-05-19 PM, post-Pilot-2, user-directed: option A)
|
||
|
||
**Change:** §7 task environment only — replace the synthetic corpus + families with a **forcing** environment: a project-specific, non-guessable multi-file subsystem where a correct answer **requires ≥8 grounded tool calls** (registry→entry→chained stage files→gate→config), so the real `planSkillDistillation` ≥5 gate actually fires and a skill is authored. System prompt firmed to require tool-grounded evidence (corpus is fictional/project-specific → cannot be answered from model priors).
|
||
|
||
**Rationale (§11):** Pilots 1+2 were INCONCLUSIVE for one model-invariant reason — tasks resolved in 1–6 tool calls, below the ≥5 trigger (shipped R1 correctly gated off every time). The bottleneck is task difficulty, not model or claim. Amendment 3 fixes exactly that.
|
||
|
||
**Unchanged (no p-hacking):** §2 metric, §3 threshold + one-sided sign test, §§5–6 arms/isolation/grader-mechanism, §§8–9 caps + pre-registered gate, §10 outcomes, §4 model (qwen-thinking, Amdt 2). Only §7's corpus/family *content* changes (the grader still = "all required facts present"; required-facts are now scattered ≥1-per-file to force traversal). Pilots 1+2 stand verbatim in the record; the forcing-corpus run = **Pilot 3** under the same locked machinery, same $5/$45 caps (cum spent $0.0141).
|
||
|
||
## Amendment 4 (2026-05-19 PM, post-Pilot-3, user-directed: option B + harness-bug retraction)
|
||
|
||
**RETRACTION:** The "decisive cross-pilot finding — 30B won't autonomously self-distil (6/6→0)" recorded after Pilots 1–3 is **WITHDRAWN**. Root cause was a **harness defect, not model behavior**: the distill phase was a *single* `runAgentLoop` turn. The model correctly traced the task and emitted its final answer; the loop then exited (no tool_calls). It was **never given the post-task turn** in which production R1 actually distils — in prod the `chat.ts` seam computes `planSkillDistillation` *after* the turn and surfaces `.directive` into a *subsequent* turn. The model never declined `create_skill`; it was never asked at a point it could act. (The user flagged this: qwen demonstrably *can* call tools — every distill run passed the grader, which requires successful tool calls.) Any claim about model self-distillation propensity from Pilots 1–3 is void.
|
||
|
||
**Changes:** (a) **harness correctness fix** — two-phase distill: Phase 1 runs task_a clean (no in-turn distill rule), then Phase 2 replays production R1 (continue the conversation: task → answer → the *real* `planSkillDistillation().directive`) with `create_skill` available — the turn the model can actually act on. (b) Per user option B: model §4 → **`anthropic/claude-sonnet-4.6`** (frontier agentic, OpenRouter $3/$15 per M). (c) §8 powered cap → **$40** (user's ≤$40).
|
||
|
||
**Unchanged (no p-hacking):** §2 metric, §3 threshold + sign test, §§5–6 isolation/grader, §9 gate logic, §10 outcomes, §7 forcing corpus (Amdt 3, validated). Sequence stays T3: a fresh **$5 pilot** validates the fixed harness, then — only if the pre-registered gate passes — the **powered N=20** under the $40 combined cap (user-authorized B). Pilots 1–3 remain in the record as harness-development history with this retraction attached.
|