oleg.m/waggle-os
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
master
waggle-os/benchmarks/preregistration/manifest-v5-preregistration.md
Oleg Maslov 0c3e2ead3b
Some checks failed
Installer Smoke / installer-smoke (push) Has been cancelled
Details
moving
2026-09-02 10:10:29 +02:00

18 KiB
Raw Permalink Blame History

Manifest v5 — Task 2.5 Stage 3 N=400 Pre-Registration (RPM-Throttled)

Manifest version: v5.0.0-preregistration Manifest type: stage_3_n400_preregistration_v5_rpm_throttled Preregistered date: 2026-04-24 Authority: PM (Marko Marković) — P4 path (manifest v5 + concurrency=1) ratified 2026-04-24 on §1.3b IN_SCOPE verdict following §1.3 Gate P+ probe FAIL. Inherits §1.1 lock-semantics waiver + §1.2 RCA ratifications. Branch: feature/c3-v3-wrapper Code freeze: HEAD 373516c2784807da8536dbc0c194c54f4e4cd4be (short 373516c) — frozen EXCEPT the single §5.2 addendum edit to litellm-config.yaml permitted by Step 2 of the v5 emission path (see §0.5 Delta Log + §11). Supersedes: Manifest v4 (anchor commit dedd698, obsoleted by this v5 emission after §1.3 probe FAIL revealed Google per-model 25-RPM preview cap on gemini-3.1-pro-preview). v4 pre-registration remains a predecessor audit artefact; v5 governs all Stage 3 re-kick forward. Inherits: Bench-Spec LOCK v1 (decisions/2026-04-22-bench-spec-locked.manifest.yaml). Machine-readable twin: manifest-v5-preregistration.yaml

0. Status

PRE-REGISTERED — PENDING PM RATIFICATION OF v5 EMISSION.

This document supersedes manifest v4 (anchor dedd698). v5's trigger and scope deltas are enumerated in §0.5 below. All other sections inherit v4 verbatim for tamper-evident standalone audit. Any change to v5's success criteria, statistical tests, sample design, or scope after v5's anchor commit invalidates the pre-registration and requires a new PM-ratified decision document (manifest v6+).

Anchor commit SHA: recorded in the git commit that adds these files. Manifest SHA-256 (MD + YAML bytes): computed at anchor-commit time via sha256sum benchmarks/preregistration/manifest-v5-preregistration.{md,yaml} — recorded in the commit message body for tamper-evident audit trail.

0.5. v5 Delta Log (NEW — v5-specific)

Trigger

§1.3 Gate P+ pre-flight probe (anchor 66dcd5a, 2026-04-24) empirical finding: 24 / 50 HTTP 429 on gemini-3.1-pro-preview via LiteLLM alias, at a steady 1.67 RPS submission rate. Google 429 body (call 25) cites:

Quota exceeded for metric: generativelanguage.googleapis.com/generate_requests_per_model, limit: 25, model: gemini-3.1-pro.

Root cause: Google PRODUCT POLICY — preview models have per-model sub-caps (here 25 RPM) that do NOT scale with account billing tier. Tier 2's 1000 RPM (Egzakta billing ID 01DBA5-921E58-9DAF46) applies account-wide across models, not per model.

§1.3b scope audit (anchor 69a14708, 2026-04-24): applying P2 (LiteLLM rpm: 20) inline to litellm-config.yaml:361-364 (gemini-3.1-pro-preview alias block) is IN_SCOPE of manifest v4 §11 frozen paths list under both strict YAML and narrow MD readings. P2 terminally blocked; fallback to P4 per PM decision tree.

Changes from v4

# Section v4 v5
§3 Sample concurrency --parallel-concurrency 2; three batches (cells 1+2, 3+4, 5) --parallel-concurrency 1; five sequential cell invocations (cells 1→2→3→4→5)
§5.2 Judge ensemble triple with no rate limits in LiteLLM config addendum: rpm: 20 on gemini-3.1-pro-preview alias (the ONLY §11 exception in v5)
§8 §7.4 Runner lock concurrent_runners: FORBIDDEN (cross-process) per §1.1 waiver; intra-wrapper parallel allowed concurrent_runners: SEQUENTIAL, parallel-concurrency=1; moot by sample-design change, but §1.1 cross-process waiver still applies at PID level
§11 Code freeze HEAD 373516c; no file modifications during run HEAD 373516c except the ONE permitted litellm-config.yaml §5.2 addendum (rpm: 20 on gemini-3.1-pro-preview alias). All other frozen paths from v4 unchanged.
§14 Budget $30 cap / $28 halt / ~$23 expected; wall-clock ~40-60 min (optimistic) $30 cap / $28 halt / ~$23 expected UNCHANGED; wall-clock re-estimated under concurrency=1 (see §14)

UNCHANGED from v4 (verbatim inheritance)

§1 primary hypothesis (Fisher one-sided p<0.10 on retrieval−no-context ≥ 5pp); §2 secondary endpoints (S1–S5); §4 dataset (LoCoMo 1531 instances, SHAs 79fa87e9… + 39e415e2…); §6 substrate (HybridSearch conv-scope top-K=20, nomic-embed-text); §7 SYSTEM_AGENTIC verbatim bytes (SHA-256 6facae6d…, 1467 bytes); §9 post-hoc exclusion policy NONE; §10 deviation policy (halt + restart-required); §12 scope boundaries (claim/not-claim + SOTA composition reserved for PM); §13 PM gates (Gate P + Gate D structure retained); §15 related artefacts (predecessor).

Parent chain

v4 predecessor pre-registration: anchor commit dedd698. v5 supersedes v4 for Stage 3 N=400 governance. v4's artefacts (probe FAIL at 66dcd5a, scope audit at 69a14708) inform v5 but do NOT re-pre-register.

LiteLLM alias naming reconciliation

LiteLLM alias naming reconciliation: Step 2 rpm: 20 edit applied to gemini-3.1-pro alias (litellm-config.yaml:361-364), which routes to upstream gemini/gemini-3.1-pro-preview. Brief §5.2 phrased this as "gemini-3.1-pro-preview alias"; terminology refers to upstream model name, not LiteLLM alias key. Both aliases share the same Google upstream bucket; see Fold-in 3.5b for sibling alias treatment.

1. Primary hypothesis (directional, confirmatory)

Inherited verbatim from manifest v4 §1. No change.

Memory-lift at conv-scope retrieval exceeds zero-memory baseline.

retrieval_judge_accuracy − no-context_judge_accuracy ≥ 5pp

evaluated at Fisher exact one-sided p-value < 0.10.

One-sided justification: theory-driven directional claim; ex-ante scaffolding from Gate B whole-corpus-leak dry-run (8/20 other-conv leak)

  • Gate C monotonicity chain (no-context 0.10 < retrieval 0.35 < agentic 0.40 < oracle 0.55 at N=20).

Failure mode: <2% probability at N=400 given Gate C's +25pp effect size (5× threshold). If primary fails despite coherent chain → PM adjudication on power-vs-signal question.

2. Secondary endpoints (ex-ante, non-blocking on primary)

Inherited verbatim from manifest v4 §2. No change.

# Endpoint Direction Threshold Test
S1 no-context ≤ retrieval positive ≥ 0pp Fisher one-sided p < 0.20
S2 retrieval ≤ agentic positive ≥ 0pp Fisher one-sided p < 0.20
S3 agentic ≤ oracle-context positive ≥ 0pp Fisher one-sided p < 0.20
S4 agentic − retrieval positive ≥ 0pp descriptive + Wilson 95% CI
S5 oracle-context − full-context positive (expected) descriptive descriptive (SYSTEM_EVOLVED strict-abstain diagnostic)

Bench-Spec LOCK v1 multi-comparisons policy: secondary = descriptive, no correction required.

3. Sample design — CHANGED (concurrency 2 → 1)

  • Cells: five, run in a single invocation. Cell definitions unchanged from v4:
    1. no-context — true zero-memory baseline.
    2. oracle-context — PM-facing alias for harness raw.
    3. full-context — oracle + SYSTEM_EVOLVED strict-abstain.
    4. retrieval — conv-scope HybridSearch top-K=20.
    5. agentic — softened SYSTEM_AGENTIC + bound search_memory + forced-fallback.
  • N per cell: 400 instances. Unchanged.
  • Total evaluations: 5 × 400 = 2000 judge-scored evaluations. Unchanged.
  • Instance selection seed: 42. Unchanged.
  • Instance pool: canonical LoCoMo dataset (see §4). Unchanged.
  • Matched-pairs design: same 400 instances flow through all cells. Unchanged.
  • Concurrency (CHANGED): --parallel-concurrency 1. Five cells run sequentially (no intra-wrapper parallelism). Five serial batches of one cell each. No inter-cell overlap.
    • Rationale: §1.3 Gate P+ empirical finding — Gemini per-model 25-RPM cap. Concurrency=1 halves peak Gemini RPS (ceiling ≈ 12 Gemini/min at steady rate), providing margin under the 20 RPM LiteLLM-side throttle (§5.2 addendum) + the 25 RPM Google-side cap.
    • Impact on runtime: ~2× wall-clock vs v4's concurrency=2 assumption (see §14).

4. Dataset

Inherited verbatim from manifest v4 §4. No change.

  • Source: benchmarks/data/locomo10.json (snap-research LoCoMo).
  • Upstream reference: https://raw.githubusercontent.com/snap-research/locomo/main/data/locomo10.json.
  • Paper reference: Maharana et al., ACL-2024.
  • Raw archive SHA-256: 79fa87e90f04081343b8c8debecb80a9a6842b76a7aa537dc9fdf651ea698ff4.
  • Canonical build: benchmarks/data/locomo/locomo-1540.jsonl (1531 instances after canonicalisation).
  • Canonical dataset SHA-256: 39e415e2f3a0fa1bd3cb1804a58d0b440b50d3070b2100698437e4ec402a5b24.
  • Selection: 400 per cell via seed-42 shuffle + take-first-400.

5. Model stack

5.1 Subject route table (Qwen 3.6-35B-A3B-Thinking)

Inherited verbatim from manifest v4 §5.1. No change.

Priority alias thinking max_tokens
primary qwen3.6-35b-a3b-via-dashscope-direct on 16000
fallback_1 qwen3.6-35b-a3b-via-openrouter on 64000
fallback_2 NOT_AVAILABLE — —

Pricing $0.20 / $0.80 per M in/out. floating_alias pinning. B3 addendum § 5.

5.2 Judge ensemble — CHANGED (rpm:20 addendum)

Slot alias role rate-limit (v5 addendum)
primary_judge_1 claude-opus-4-7 primary none (Anthropic immutable, no observed rate-limit pressure at Stage 3 scale)
primary_judge_2 gpt-5.4 primary none
primary_judge_3 gemini-3.1-pro-preview primary rpm: 20 (v5 addendum via litellm-config.yaml §5.2 edit)
tiebreak_reserve grok-4.20 reserve (1/1/1 only) none
  • rpm: 20 addendum is the single permitted §11 exception in v5. Applied to the model_list entry for gemini-3.1-pro-preview in litellm-config.yaml (Step 2 of the v5 emission path). Chosen 20 < 25 Google-side cap with a 5 RPM margin for burst variance. Further throttle (rpm:15) reserved for PM adjudication if §1.3c throttle-verification probe FAILs at rpm:20.
  • Tie-break path: 1/1/1 → Grok 4.20. 2/2 → PM escalation. Unchanged.
  • Consistency constraint: same physical judge models as v4 / Stage 2-Retry. LiteLLM alias identifier unchanged (gemini-3.1-pro-preview); only the in-file rate-limit metadata added. Physical upstream model binding (gemini/gemini-3.1-pro-preview @ Google AI Studio) unchanged.
  • κ monitoring: Fleiss' κ thresholds inherited from Bench-Spec LOCK v1 (pass-no-flag ≥ 0.65; pass-with-flag [0.60, 0.65]; halt ≤ 0.60).

5.3 Health-check predicate

Inherited verbatim from manifest v4 §5.3. No change.

6. Substrate (conv-scope retrieval)

Inherited verbatim from manifest v4 §6. No change.

  • @waggle/core::HybridSearch (RRF-fused FTS5 + vec0).
  • gopId = conversation_id scope filter at search.ts:14.
  • Top-K default 20; upper clamp 50.
  • createOllamaEmbedder() + nomic-embed-text (1024 dims, local, $0).
  • Ingest batch 200.

6.1 Agentic-cell tool binding

Inherited verbatim from manifest v4 §6.1. No change.

makeSearchMemoryTool(substrate, 20, instance.conversation_id). maxTurns=3. 180 s timeout. Forced-fallback on empty content + tool-use (0 firings at Gate C; load-bearing insurance).

7. SYSTEM_AGENTIC prompt — verbatim bytes locked

Inherited verbatim from manifest v4 §7. No change.

SHA-256: 6facae6decc44a6404290514accb4f7cb364081b32d02847a20f8e871633e328 (1467 bytes, no trailing newline). Source: benchmarks/harness/src/cells.ts lines 75–102. Softened text from Stage 2-Retry Gate A (commit 373516c). See manifest v4 §7 for full verbatim reproduction.

8. Stopping rules — §7.4 updated for concurrency=1

# Rule Source Trigger Action
§7.1 Budget hard halt runner.ts cumulative spend ≥ $28.00 halt, persist partial, exit ping
§7.2 Streak halt streak-tracker.ts 3 consecutive subject fetch failures halt, persist partial
§7.3 Pre-cell health check fail health-check.ts 5xx / fetch-error on subject or judge probe halt before cell
§7.4 (v5) Runner lock runner-lock.ts concurrent cross-process invocation detected halt. Under v5 concurrency=1 there is no intra-wrapper parallelism — the §1.1 cross-process-only waiver still governs, now with no exemption needed because no intra-wrapper spawning occurs.
§7.5 Pre-registration deviation this document any change to §1–§9 during run halt + PM raise

No interim looks. Halt only on the five conditions above.

9. Post-hoc exclusion policy: NONE

Inherited verbatim from manifest v4 §9. No change.

All 2000 evals enter the denominator. evaluator_loss (judge-triple failure) counted in denominator, reported separately. No instance whitelist/blacklist. Selective exclusion forbidden ex-ante.

10. Deviation policy

Inherited verbatim from manifest v4 §10. No change.

Any deviation from §1–§9 during run → (1) immediate halt, (2) PM raise, (3) re-pre-registration (manifest v6+) if accepted. Consistent with Bench-Spec LOCK v1 preregistration.mid_run_amendment_policy: halt_restart_required.

11. Code freeze — updated with single permitted exception

The following code is frozen at HEAD 373516c for the duration of Stage 3 N=400 under v5. No changes permitted between v5 anchor commit and Gate D exit EXCEPT the single §5.2 addendum edit to litellm-config.yaml noted in §0.5 Delta Log and §5.2 (rpm: 20 on the gemini-3.1-pro-preview alias block).

Frozen paths (inherited verbatim from v4 §11):

  • Cell semantics (benchmarks/harness/src/cells.ts).
  • Substrate (benchmarks/harness/src/substrate.ts, @waggle/core::HybridSearch, @waggle/core::FrameStore, @waggle/core::SessionStore).
  • SYSTEM_AGENTIC + SYSTEM_AGENTIC_FORCED_FALLBACK prompts.
  • Agent loop (@waggle/agent::runAgentLoop, @waggle/agent::tools.ts).
  • Judge ensemble + routing (benchmarks/harness/src/judge-*.ts, config/models.json, litellm-config.yaml judge aliases EXCEPT the single rpm:20 addendum).
  • Subject route table (config/models.json qwen aliases).
  • Test suite.

Execution-only delta during N=400 run: new JSONL files emitted to benchmarks/results/ AND the single pre-run litellm-config.yaml §5.2 addendum committed before the run. No other file modifications during or after run.

12. Scope boundaries

Inherited verbatim from manifest v4 §12. No change.

Can claim at Gate D:

  • Magnitude + significance of conv-scope retrieval memory-lift at Qwen-3.6-35b-a3b under HEAD 373516c.
  • Per-cell judge-accuracy with Wilson 95% CIs.
  • Monotonicity chain.
  • Conv-scope fair-comparison methodology.
  • Agentic discipline numbers.

Cannot claim at Gate D:

  • Direct comparability to Mem0 91.6% (different scope + memory-synthesis layer).
  • Multi-model generalization (Qwen-only).
  • Production performance.

Reserved for PM:

  • Public-claim phrasing + venue.
  • Matched-scope Mem0 co-run.
  • Publication timing.

CC-1 does NOT compose public SOTA claim. Scope + data only.

13. PM gates

Inherited verbatim from manifest v4 §13. No change.

Gate P+ (v5 pre-run, pre-N=400 execution)

  • Trigger: commit of v5 files + litellm-config.yaml §5.2 addendum on feature/c3-v3-wrapper.
  • Halt: CC-1 stops; no N=400 kick without PM GO after §1.3c + §1.3e.
  • PM checks: v5 content matches all prior ratifications; §1–§10 locked unambiguously.

Gate D (post-run, pre-SOTA-claim)

  • Trigger: N=400 exit (clean or halted per §8).
  • Halt: CC-1 writes exit report at PM-Waggle-OS/sessions/2026-04-24-task25-stage3-n400-complete.md.
  • PM decides SOTA claim composition / publish gate / further scope.

No self-advance at either gate.

14. Budget — envelope unchanged; wall-clock re-estimated

  • Cap: $30.00. Unchanged.
  • Hard halt: $28.00. Unchanged.
  • Expected burn: ~$23. Unchanged.
  • Variance ceiling: $28. Unchanged.
  • Cost breakdown: same (Qwen ~$2.50, judge triple ~$20, ollama $0, Grok ~$0.50).

Wall-clock re-estimate (v5-specific):

  • Under concurrency=1 + rpm:20 Gemini throttle, effective cell throughput is governed by the judge triple's slowest path. Gemini ≤ 20 RPM floor means ≤ 20 full triples/min (Opus + GPT + Gemini all complete).
  • At 400 instances × 5 cells = 2000 evals / 20 RPM = 100 min minimum sustained-rate floor for the Gemini leg. Realistic wall-clock includes subject call latency + Opus/GPT judge latency adding to the triple critical path.
  • Practical estimate: 2–3 hours for the full N=400 run under v5.
  • Upper bound: if judge latency variance pushes instances past their expected rate, halt §7.1 at $28 caps the downside independently of time.

15. Related artefacts

  • Manifest v4 predecessor: anchor commit dedd69888e008fb1584bc249aff43b19f55a88e5 (short dedd698).
  • §1.1 lock-semantics clarification (L-1 ratified): anchor 67eb89914a49ec38049379bf952d5f62b82c188d (short 67eb899).
  • §1.2 runner early-exit RCA (Task 2.6 tech-debt ratified): anchor 274e9871b54599077a3d72de88d505550803a805 (short 274e987).
  • §1.3 Gate P+ probe FAIL: anchor 66dcd5a1b18b9367662b04f1c9e1b66d855a9481 (short 66dcd5a).
  • §1.3b litellm-config scope audit (IN_SCOPE verdict): anchor 69a14708f78a74d2cb7ef07faf2d949f6ffc3209 (short 69a14708).
  • Bench-Spec LOCK v1 parent: PM-Waggle-OS/decisions/2026-04-22-bench-spec-locked.manifest.yaml.
  • Stage 2-Retry Gate C exit: PM-Waggle-OS/sessions/2026-04-24-task25-stage2-retry-complete.md.
  • Stage 3 rekick brief (Option A / P4): PM-Waggle-OS/briefs/2026-04-24-cc-task25-stage3-rekick-option-a.md.
  • Stage 3 brief (original): PM-Waggle-OS/briefs/2026-04-24-cc-task25-stage3-n400-kickoff.md.
  • Rollback tag: checkpoint/pre-self-evolution-2026-04-14.

End of Manifest v5 pre-registration. This document is the anchor for all analysis choices at Stage 3 Gate D exit under the P4 (concurrency=1 + rpm:20) path.

Reference in New Issue View Git Blame Copy Permalink