moving

benchmarks/preregistration/manifest-v5-preregistration.md

@@ -0,0 +1,356 @@
+# Manifest v5 — Task 2.5 Stage 3 N=400 Pre-Registration (RPM-Throttled)
+
+**Manifest version:** v5.0.0-preregistration
+**Manifest type:** `stage_3_n400_preregistration_v5_rpm_throttled`
+**Preregistered date:** 2026-04-24
+**Authority:** PM (Marko Marković) — P4 path (manifest v5 + concurrency=1) ratified 2026-04-24 on §1.3b IN_SCOPE verdict following §1.3 Gate P+ probe FAIL. Inherits §1.1 lock-semantics waiver + §1.2 RCA ratifications.
+**Branch:** `feature/c3-v3-wrapper`
+**Code freeze:** HEAD `373516c2784807da8536dbc0c194c54f4e4cd4be` (short `373516c`) — frozen EXCEPT the single §5.2 addendum edit to `litellm-config.yaml` permitted by Step 2 of the v5 emission path (see §0.5 Delta Log + §11).
+**Supersedes:** Manifest v4 (anchor commit `dedd698`, obsoleted by this v5 emission after §1.3 probe FAIL revealed Google per-model 25-RPM preview cap on `gemini-3.1-pro-preview`). v4 pre-registration remains a predecessor audit artefact; v5 governs all Stage 3 re-kick forward.
+**Inherits:** Bench-Spec LOCK v1 (`decisions/2026-04-22-bench-spec-locked.manifest.yaml`).
+**Machine-readable twin:** [`manifest-v5-preregistration.yaml`](manifest-v5-preregistration.yaml)
+
+---
+
+## 0. Status
+
+**PRE-REGISTERED — PENDING PM RATIFICATION OF v5 EMISSION.**
+
+This document supersedes manifest v4 (anchor `dedd698`). v5's trigger and scope deltas are enumerated in §0.5 below. All other sections inherit v4 verbatim for tamper-evident standalone audit. Any change to v5's success criteria, statistical tests, sample design, or scope after v5's anchor commit invalidates the pre-registration and requires a new PM-ratified decision document (manifest v6+).
+
+**Anchor commit SHA:** recorded in the git commit that adds these files.
+**Manifest SHA-256 (MD + YAML bytes):** computed at anchor-commit time via
+`sha256sum benchmarks/preregistration/manifest-v5-preregistration.{md,yaml}` — recorded
+in the commit message body for tamper-evident audit trail.
+
+---
+
+## 0.5. v5 Delta Log (NEW — v5-specific)
+
+### Trigger
+
+**§1.3 Gate P+ pre-flight probe (anchor `66dcd5a`, 2026-04-24) empirical finding:** 24 / 50 HTTP 429 on `gemini-3.1-pro-preview` via LiteLLM alias, at a steady 1.67 RPS submission rate. Google 429 body (call 25) cites:
+
+> `Quota exceeded for metric: generativelanguage.googleapis.com/generate_requests_per_model, limit: 25, model: gemini-3.1-pro`.
+
+Root cause: Google PRODUCT POLICY — preview models have per-model sub-caps (here **25 RPM**) that do NOT scale with account billing tier. Tier 2's 1000 RPM (Egzakta billing ID 01DBA5-921E58-9DAF46) applies account-wide across models, not per model.
+
+**§1.3b scope audit (anchor `69a14708`, 2026-04-24):** applying P2 (LiteLLM `rpm: 20`) inline to `litellm-config.yaml:361-364` (gemini-3.1-pro-preview alias block) is IN_SCOPE of manifest v4 §11 frozen paths list under both strict YAML and narrow MD readings. P2 terminally blocked; fallback to P4 per PM decision tree.
+
+### Changes from v4
+
+| # | Section | v4 | v5 |
+|---|---------|-----|-----|
+| §3 | Sample concurrency | `--parallel-concurrency 2`; three batches (cells 1+2, 3+4, 5) | **`--parallel-concurrency 1`; five sequential cell invocations (cells 1→2→3→4→5)** |
+| §5.2 | Judge ensemble | triple with no rate limits in LiteLLM config | **addendum: `rpm: 20` on `gemini-3.1-pro-preview` alias** (the ONLY §11 exception in v5) |
+| §8 §7.4 | Runner lock | `concurrent_runners: FORBIDDEN (cross-process)` per §1.1 waiver; intra-wrapper parallel allowed | **`concurrent_runners: SEQUENTIAL, parallel-concurrency=1`**; moot by sample-design change, but §1.1 cross-process waiver still applies at PID level |
+| §11 | Code freeze | HEAD 373516c; no file modifications during run | **HEAD 373516c except the ONE permitted `litellm-config.yaml` §5.2 addendum** (`rpm: 20` on gemini-3.1-pro-preview alias). All other frozen paths from v4 unchanged. |
+| §14 | Budget | $30 cap / $28 halt / ~$23 expected; wall-clock ~40-60 min (optimistic) | $30 cap / $28 halt / ~$23 expected UNCHANGED; wall-clock re-estimated under concurrency=1 (see §14) |
+
+### UNCHANGED from v4 (verbatim inheritance)
+
+**§1** primary hypothesis (Fisher one-sided p<0.10 on retrieval−no-context ≥ 5pp); **§2** secondary endpoints (S1–S5); **§4** dataset (LoCoMo 1531 instances, SHAs `79fa87e9…` + `39e415e2…`); **§6** substrate (HybridSearch conv-scope top-K=20, nomic-embed-text); **§7** SYSTEM_AGENTIC verbatim bytes (SHA-256 `6facae6d…`, 1467 bytes); **§9** post-hoc exclusion policy NONE; **§10** deviation policy (halt + restart-required); **§12** scope boundaries (claim/not-claim + SOTA composition reserved for PM); **§13** PM gates (Gate P + Gate D structure retained); **§15** related artefacts (predecessor).
+
+### Parent chain
+
+v4 predecessor pre-registration: anchor commit `dedd698`. v5 supersedes v4 for Stage 3 N=400 governance. v4's artefacts (probe FAIL at `66dcd5a`, scope audit at `69a14708`) inform v5 but do NOT re-pre-register.
+
+### LiteLLM alias naming reconciliation
+
+LiteLLM alias naming reconciliation: Step 2 `rpm: 20` edit applied to
+`gemini-3.1-pro` alias (`litellm-config.yaml:361-364`), which routes to
+upstream `gemini/gemini-3.1-pro-preview`. Brief §5.2 phrased this as
+"`gemini-3.1-pro-preview` alias"; terminology refers to upstream model
+name, not LiteLLM alias key. Both aliases share the same Google upstream
+bucket; see Fold-in 3.5b for sibling alias treatment.
+
+---
+
+## 1. Primary hypothesis (directional, confirmatory)
+
+_Inherited verbatim from manifest v4 §1. No change._
+
+> **Memory-lift at conv-scope retrieval exceeds zero-memory baseline.**
+>
+> `retrieval_judge_accuracy − no-context_judge_accuracy ≥ 5pp`
+>
+> evaluated at **Fisher exact one-sided** p-value **< 0.10**.
+
+**One-sided justification:** theory-driven directional claim; ex-ante
+scaffolding from Gate B whole-corpus-leak dry-run (8/20 other-conv leak)
++ Gate C monotonicity chain (no-context 0.10 < retrieval 0.35 < agentic
+0.40 < oracle 0.55 at N=20).
+
+**Failure mode:** <2% probability at N=400 given Gate C's +25pp effect
+size (5× threshold). If primary fails despite coherent chain → PM
+adjudication on power-vs-signal question.
+
+---
+
+## 2. Secondary endpoints (ex-ante, non-blocking on primary)
+
+_Inherited verbatim from manifest v4 §2. No change._
+
+| # | Endpoint | Direction | Threshold | Test |
+|---|----------|-----------|-----------|------|
+| S1 | no-context ≤ retrieval | positive | ≥ 0pp | Fisher one-sided p < 0.20 |
+| S2 | retrieval ≤ agentic | positive | ≥ 0pp | Fisher one-sided p < 0.20 |
+| S3 | agentic ≤ oracle-context | positive | ≥ 0pp | Fisher one-sided p < 0.20 |
+| S4 | agentic − retrieval | positive | ≥ 0pp | descriptive + Wilson 95% CI |
+| S5 | oracle-context − full-context | positive (expected) | descriptive | descriptive (SYSTEM_EVOLVED strict-abstain diagnostic) |
+
+Bench-Spec LOCK v1 multi-comparisons policy: secondary = descriptive, no correction required.
+
+---
+
+## 3. Sample design — **CHANGED (concurrency 2 → 1)**
+
+- **Cells:** five, run in a single invocation. Cell definitions unchanged from v4:
+  1. `no-context` — true zero-memory baseline.
+  2. `oracle-context` — PM-facing alias for harness `raw`.
+  3. `full-context` — oracle + SYSTEM_EVOLVED strict-abstain.
+  4. `retrieval` — conv-scope HybridSearch top-K=20.
+  5. `agentic` — softened SYSTEM_AGENTIC + bound search_memory + forced-fallback.
+- **N per cell:** 400 instances. **Unchanged.**
+- **Total evaluations:** 5 × 400 = **2000 judge-scored evaluations**. **Unchanged.**
+- **Instance selection seed:** `42`. **Unchanged.**
+- **Instance pool:** canonical LoCoMo dataset (see §4). **Unchanged.**
+- **Matched-pairs design:** same 400 instances flow through all cells. **Unchanged.**
+- **Concurrency (CHANGED):** **`--parallel-concurrency 1`**. Five cells run
+  sequentially (no intra-wrapper parallelism). Five serial batches of one
+  cell each. No inter-cell overlap.
+  - _Rationale:_ §1.3 Gate P+ empirical finding — Gemini per-model 25-RPM cap.
+    Concurrency=1 halves peak Gemini RPS (ceiling ≈ 12 Gemini/min at steady
+    rate), providing margin under the 20 RPM LiteLLM-side throttle (§5.2
+    addendum) + the 25 RPM Google-side cap.
+  - _Impact on runtime:_ ~2× wall-clock vs v4's concurrency=2 assumption
+    (see §14).
+
+---
+
+## 4. Dataset
+
+_Inherited verbatim from manifest v4 §4. No change._
+
+- **Source:** `benchmarks/data/locomo10.json` (snap-research LoCoMo).
+- **Upstream reference:** `https://raw.githubusercontent.com/snap-research/locomo/main/data/locomo10.json`.
+- **Paper reference:** Maharana et al., ACL-2024.
+- **Raw archive SHA-256:** `79fa87e90f04081343b8c8debecb80a9a6842b76a7aa537dc9fdf651ea698ff4`.
+- **Canonical build:** `benchmarks/data/locomo/locomo-1540.jsonl` (1531 instances after canonicalisation).
+- **Canonical dataset SHA-256:** `39e415e2f3a0fa1bd3cb1804a58d0b440b50d3070b2100698437e4ec402a5b24`.
+- **Selection:** 400 per cell via seed-42 shuffle + take-first-400.
+
+---
+
+## 5. Model stack
+
+### 5.1 Subject route table (Qwen 3.6-35B-A3B-Thinking)
+
+_Inherited verbatim from manifest v4 §5.1. No change._
+
+| Priority | alias | thinking | max_tokens |
+|----------|-------|----------|------------|
+| primary | `qwen3.6-35b-a3b-via-dashscope-direct` | `on` | 16000 |
+| fallback_1 | `qwen3.6-35b-a3b-via-openrouter` | `on` | 64000 |
+| fallback_2 | `NOT_AVAILABLE` | — | — |
+
+Pricing $0.20 / $0.80 per M in/out. `floating_alias` pinning. B3 addendum § 5.
+
+### 5.2 Judge ensemble — **CHANGED (rpm:20 addendum)**
+
+| Slot | alias | role | rate-limit (v5 addendum) |
+|------|-------|------|--------------------------|
+| primary_judge_1 | `claude-opus-4-7` | primary | none (Anthropic immutable, no observed rate-limit pressure at Stage 3 scale) |
+| primary_judge_2 | `gpt-5.4` | primary | none |
+| primary_judge_3 | `gemini-3.1-pro-preview` | primary | **`rpm: 20`** (v5 addendum via litellm-config.yaml §5.2 edit) |
+| tiebreak_reserve | `grok-4.20` | reserve (1/1/1 only) | none |
+
+- **rpm: 20 addendum** is the single permitted §11 exception in v5. Applied to
+  the `model_list` entry for `gemini-3.1-pro-preview` in `litellm-config.yaml`
+  (Step 2 of the v5 emission path). Chosen 20 < 25 Google-side cap with a 5
+  RPM margin for burst variance. Further throttle (rpm:15) reserved for PM
+  adjudication if §1.3c throttle-verification probe FAILs at rpm:20.
+- **Tie-break path:** 1/1/1 → Grok 4.20. 2/2 → PM escalation. Unchanged.
+- **Consistency constraint:** same physical judge models as v4 / Stage 2-Retry.
+  LiteLLM alias identifier unchanged (`gemini-3.1-pro-preview`); only the
+  in-file rate-limit metadata added. Physical upstream model binding
+  (`gemini/gemini-3.1-pro-preview` @ Google AI Studio) unchanged.
+- **κ monitoring:** Fleiss' κ thresholds inherited from Bench-Spec LOCK v1
+  (pass-no-flag ≥ 0.65; pass-with-flag [0.60, 0.65]; halt ≤ 0.60).
+
+### 5.3 Health-check predicate
+
+_Inherited verbatim from manifest v4 §5.3. No change._
+
+---
+
+## 6. Substrate (conv-scope retrieval)
+
+_Inherited verbatim from manifest v4 §6. No change._
+
+- `@waggle/core::HybridSearch` (RRF-fused FTS5 + vec0).
+- `gopId = conversation_id` scope filter at `search.ts:14`.
+- Top-K default 20; upper clamp 50.
+- `createOllamaEmbedder()` + `nomic-embed-text` (1024 dims, local, $0).
+- Ingest batch 200.
+
+### 6.1 Agentic-cell tool binding
+
+_Inherited verbatim from manifest v4 §6.1. No change._
+
+`makeSearchMemoryTool(substrate, 20, instance.conversation_id)`. `maxTurns=3`.
+180 s timeout. Forced-fallback on empty content + tool-use (0 firings at
+Gate C; load-bearing insurance).
+
+---
+
+## 7. SYSTEM_AGENTIC prompt — verbatim bytes locked
+
+_Inherited verbatim from manifest v4 §7. No change._
+
+**SHA-256:** `6facae6decc44a6404290514accb4f7cb364081b32d02847a20f8e871633e328` (1467 bytes, no trailing newline). Source: `benchmarks/harness/src/cells.ts` lines 75–102. Softened text from Stage 2-Retry Gate A (commit `373516c`). See manifest v4 §7 for full verbatim reproduction.
+
+---
+
+## 8. Stopping rules — **§7.4 updated for concurrency=1**
+
+| # | Rule | Source | Trigger | Action |
+|---|------|--------|---------|--------|
+| §7.1 | Budget hard halt | `runner.ts` | cumulative spend ≥ **$28.00** | halt, persist partial, exit ping |
+| §7.2 | Streak halt | `streak-tracker.ts` | 3 consecutive subject fetch failures | halt, persist partial |
+| §7.3 | Pre-cell health check fail | `health-check.ts` | 5xx / fetch-error on subject or judge probe | halt before cell |
+| **§7.4 (v5)** | **Runner lock** | `runner-lock.ts` | concurrent cross-process invocation detected | halt. Under v5 concurrency=1 there is no intra-wrapper parallelism — the §1.1 cross-process-only waiver still governs, now with no exemption needed because no intra-wrapper spawning occurs. |
+| §7.5 | Pre-registration deviation | this document | any change to §1–§9 during run | halt + PM raise |
+
+**No interim looks.** Halt only on the five conditions above.
+
+---
+
+## 9. Post-hoc exclusion policy: **NONE**
+
+_Inherited verbatim from manifest v4 §9. No change._
+
+All 2000 evals enter the denominator. `evaluator_loss` (judge-triple
+failure) counted in denominator, reported separately. No instance
+whitelist/blacklist. Selective exclusion forbidden ex-ante.
+
+---
+
+## 10. Deviation policy
+
+_Inherited verbatim from manifest v4 §10. No change._
+
+Any deviation from §1–§9 during run → (1) immediate halt, (2) PM raise,
+(3) re-pre-registration (manifest v6+) if accepted. Consistent with
+Bench-Spec LOCK v1 `preregistration.mid_run_amendment_policy: halt_restart_required`.
+
+---
+
+## 11. Code freeze — **updated with single permitted exception**
+
+The following code is **frozen at HEAD `373516c`** for the duration of
+Stage 3 N=400 under v5. No changes permitted between v5 anchor commit
+and Gate D exit **EXCEPT** the single §5.2 addendum edit to
+`litellm-config.yaml` noted in §0.5 Delta Log and §5.2 (`rpm: 20` on
+the `gemini-3.1-pro-preview` alias block).
+
+Frozen paths (inherited verbatim from v4 §11):
+
+- Cell semantics (`benchmarks/harness/src/cells.ts`).
+- Substrate (`benchmarks/harness/src/substrate.ts`, `@waggle/core::HybridSearch`, `@waggle/core::FrameStore`, `@waggle/core::SessionStore`).
+- SYSTEM_AGENTIC + SYSTEM_AGENTIC_FORCED_FALLBACK prompts.
+- Agent loop (`@waggle/agent::runAgentLoop`, `@waggle/agent::tools.ts`).
+- Judge ensemble + routing (`benchmarks/harness/src/judge-*.ts`, `config/models.json`, `litellm-config.yaml` judge aliases EXCEPT the single rpm:20 addendum).
+- Subject route table (`config/models.json` qwen aliases).
+- Test suite.
+
+Execution-only delta during N=400 run: new JSONL files emitted to
+`benchmarks/results/` AND the single pre-run `litellm-config.yaml`
+§5.2 addendum committed before the run. No other file modifications
+during or after run.
+
+---
+
+## 12. Scope boundaries
+
+_Inherited verbatim from manifest v4 §12. No change._
+
+### Can claim at Gate D:
+- Magnitude + significance of conv-scope retrieval memory-lift at Qwen-3.6-35b-a3b under HEAD 373516c.
+- Per-cell judge-accuracy with Wilson 95% CIs.
+- Monotonicity chain.
+- Conv-scope fair-comparison methodology.
+- Agentic discipline numbers.
+
+### Cannot claim at Gate D:
+- Direct comparability to Mem0 91.6% (different scope + memory-synthesis layer).
+- Multi-model generalization (Qwen-only).
+- Production performance.
+
+### Reserved for PM:
+- Public-claim phrasing + venue.
+- Matched-scope Mem0 co-run.
+- Publication timing.
+
+**CC-1 does NOT compose public SOTA claim.** Scope + data only.
+
+---
+
+## 13. PM gates
+
+_Inherited verbatim from manifest v4 §13. No change._
+
+### Gate P+ (v5 pre-run, pre-N=400 execution)
+- Trigger: commit of v5 files + litellm-config.yaml §5.2 addendum on feature/c3-v3-wrapper.
+- Halt: CC-1 stops; no N=400 kick without PM GO after §1.3c + §1.3e.
+- PM checks: v5 content matches all prior ratifications; §1–§10 locked unambiguously.
+
+### Gate D (post-run, pre-SOTA-claim)
+- Trigger: N=400 exit (clean or halted per §8).
+- Halt: CC-1 writes exit report at `PM-Waggle-OS/sessions/2026-04-24-task25-stage3-n400-complete.md`.
+- PM decides SOTA claim composition / publish gate / further scope.
+
+No self-advance at either gate.
+
+---
+
+## 14. Budget — envelope unchanged; wall-clock re-estimated
+
+- **Cap:** $30.00. **Unchanged.**
+- **Hard halt:** $28.00. **Unchanged.**
+- **Expected burn:** ~$23. **Unchanged.**
+- **Variance ceiling:** $28. **Unchanged.**
+- **Cost breakdown:** same (Qwen ~$2.50, judge triple ~$20, ollama $0, Grok ~$0.50).
+
+**Wall-clock re-estimate (v5-specific):**
+- Under concurrency=1 + rpm:20 Gemini throttle, effective cell throughput
+  is governed by the judge triple's slowest path. Gemini ≤ 20 RPM floor
+  means ≤ 20 full triples/min (Opus + GPT + Gemini all complete).
+- At 400 instances × 5 cells = 2000 evals / 20 RPM = **100 min minimum**
+  sustained-rate floor for the Gemini leg. Realistic wall-clock includes
+  subject call latency + Opus/GPT judge latency adding to the triple
+  critical path.
+- **Practical estimate:** 2–3 hours for the full N=400 run under v5.
+- **Upper bound:** if judge latency variance pushes instances past their
+  expected rate, halt §7.1 at $28 caps the downside independently of time.
+
+---
+
+## 15. Related artefacts
+
+- **Manifest v4 predecessor:** anchor commit `dedd69888e008fb1584bc249aff43b19f55a88e5` (short `dedd698`).
+- **§1.1 lock-semantics clarification (L-1 ratified):** anchor `67eb89914a49ec38049379bf952d5f62b82c188d` (short `67eb899`).
+- **§1.2 runner early-exit RCA (Task 2.6 tech-debt ratified):** anchor `274e9871b54599077a3d72de88d505550803a805` (short `274e987`).
+- **§1.3 Gate P+ probe FAIL:** anchor `66dcd5a1b18b9367662b04f1c9e1b66d855a9481` (short `66dcd5a`).
+- **§1.3b litellm-config scope audit (IN_SCOPE verdict):** anchor `69a14708f78a74d2cb7ef07faf2d949f6ffc3209` (short `69a14708`).
+- **Bench-Spec LOCK v1 parent:** `PM-Waggle-OS/decisions/2026-04-22-bench-spec-locked.manifest.yaml`.
+- **Stage 2-Retry Gate C exit:** `PM-Waggle-OS/sessions/2026-04-24-task25-stage2-retry-complete.md`.
+- **Stage 3 rekick brief (Option A / P4):** `PM-Waggle-OS/briefs/2026-04-24-cc-task25-stage3-rekick-option-a.md`.
+- **Stage 3 brief (original):** `PM-Waggle-OS/briefs/2026-04-24-cc-task25-stage3-n400-kickoff.md`.
+- **Rollback tag:** `checkpoint/pre-self-evolution-2026-04-14`.
+
+---
+
+_End of Manifest v5 pre-registration. This document is the anchor for all
+analysis choices at Stage 3 Gate D exit under the P4 (concurrency=1 + rpm:20)
+path._