This commit is contained in:
Oleg Maslov
356
benchmarks/preregistration/manifest-v5-preregistration.md
Normal file
356
benchmarks/preregistration/manifest-v5-preregistration.md
Normal file
@@ -0,0 +1,356 @@
|
||||
# Manifest v5 — Task 2.5 Stage 3 N=400 Pre-Registration (RPM-Throttled)
|
||||
|
||||
**Manifest version:** v5.0.0-preregistration
|
||||
**Manifest type:** `stage_3_n400_preregistration_v5_rpm_throttled`
|
||||
**Preregistered date:** 2026-04-24
|
||||
**Authority:** PM (Marko Marković) — P4 path (manifest v5 + concurrency=1) ratified 2026-04-24 on §1.3b IN_SCOPE verdict following §1.3 Gate P+ probe FAIL. Inherits §1.1 lock-semantics waiver + §1.2 RCA ratifications.
|
||||
**Branch:** `feature/c3-v3-wrapper`
|
||||
**Code freeze:** HEAD `373516c2784807da8536dbc0c194c54f4e4cd4be` (short `373516c`) — frozen EXCEPT the single §5.2 addendum edit to `litellm-config.yaml` permitted by Step 2 of the v5 emission path (see §0.5 Delta Log + §11).
|
||||
**Supersedes:** Manifest v4 (anchor commit `dedd698`, obsoleted by this v5 emission after §1.3 probe FAIL revealed Google per-model 25-RPM preview cap on `gemini-3.1-pro-preview`). v4 pre-registration remains a predecessor audit artefact; v5 governs all Stage 3 re-kick forward.
|
||||
**Inherits:** Bench-Spec LOCK v1 (`decisions/2026-04-22-bench-spec-locked.manifest.yaml`).
|
||||
**Machine-readable twin:** [`manifest-v5-preregistration.yaml`](manifest-v5-preregistration.yaml)
|
||||
|
||||
---
|
||||
|
||||
## 0. Status
|
||||
|
||||
**PRE-REGISTERED — PENDING PM RATIFICATION OF v5 EMISSION.**
|
||||
|
||||
This document supersedes manifest v4 (anchor `dedd698`). v5's trigger and scope deltas are enumerated in §0.5 below. All other sections inherit v4 verbatim for tamper-evident standalone audit. Any change to v5's success criteria, statistical tests, sample design, or scope after v5's anchor commit invalidates the pre-registration and requires a new PM-ratified decision document (manifest v6+).
|
||||
|
||||
**Anchor commit SHA:** recorded in the git commit that adds these files.
|
||||
**Manifest SHA-256 (MD + YAML bytes):** computed at anchor-commit time via
|
||||
`sha256sum benchmarks/preregistration/manifest-v5-preregistration.{md,yaml}` — recorded
|
||||
in the commit message body for tamper-evident audit trail.
|
||||
|
||||
---
|
||||
|
||||
## 0.5. v5 Delta Log (NEW — v5-specific)
|
||||
|
||||
### Trigger
|
||||
|
||||
**§1.3 Gate P+ pre-flight probe (anchor `66dcd5a`, 2026-04-24) empirical finding:** 24 / 50 HTTP 429 on `gemini-3.1-pro-preview` via LiteLLM alias, at a steady 1.67 RPS submission rate. Google 429 body (call 25) cites:
|
||||
|
||||
> `Quota exceeded for metric: generativelanguage.googleapis.com/generate_requests_per_model, limit: 25, model: gemini-3.1-pro`.
|
||||
|
||||
Root cause: Google PRODUCT POLICY — preview models have per-model sub-caps (here **25 RPM**) that do NOT scale with account billing tier. Tier 2's 1000 RPM (Egzakta billing ID 01DBA5-921E58-9DAF46) applies account-wide across models, not per model.
|
||||
|
||||
**§1.3b scope audit (anchor `69a14708`, 2026-04-24):** applying P2 (LiteLLM `rpm: 20`) inline to `litellm-config.yaml:361-364` (gemini-3.1-pro-preview alias block) is IN_SCOPE of manifest v4 §11 frozen paths list under both strict YAML and narrow MD readings. P2 terminally blocked; fallback to P4 per PM decision tree.
|
||||
|
||||
### Changes from v4
|
||||
|
||||
| # | Section | v4 | v5 |
|
||||
|---|---------|-----|-----|
|
||||
| §3 | Sample concurrency | `--parallel-concurrency 2`; three batches (cells 1+2, 3+4, 5) | **`--parallel-concurrency 1`; five sequential cell invocations (cells 1→2→3→4→5)** |
|
||||
| §5.2 | Judge ensemble | triple with no rate limits in LiteLLM config | **addendum: `rpm: 20` on `gemini-3.1-pro-preview` alias** (the ONLY §11 exception in v5) |
|
||||
| §8 §7.4 | Runner lock | `concurrent_runners: FORBIDDEN (cross-process)` per §1.1 waiver; intra-wrapper parallel allowed | **`concurrent_runners: SEQUENTIAL, parallel-concurrency=1`**; moot by sample-design change, but §1.1 cross-process waiver still applies at PID level |
|
||||
| §11 | Code freeze | HEAD 373516c; no file modifications during run | **HEAD 373516c except the ONE permitted `litellm-config.yaml` §5.2 addendum** (`rpm: 20` on gemini-3.1-pro-preview alias). All other frozen paths from v4 unchanged. |
|
||||
| §14 | Budget | $30 cap / $28 halt / ~$23 expected; wall-clock ~40-60 min (optimistic) | $30 cap / $28 halt / ~$23 expected UNCHANGED; wall-clock re-estimated under concurrency=1 (see §14) |
|
||||
|
||||
### UNCHANGED from v4 (verbatim inheritance)
|
||||
|
||||
**§1** primary hypothesis (Fisher one-sided p<0.10 on retrieval−no-context ≥ 5pp); **§2** secondary endpoints (S1–S5); **§4** dataset (LoCoMo 1531 instances, SHAs `79fa87e9…` + `39e415e2…`); **§6** substrate (HybridSearch conv-scope top-K=20, nomic-embed-text); **§7** SYSTEM_AGENTIC verbatim bytes (SHA-256 `6facae6d…`, 1467 bytes); **§9** post-hoc exclusion policy NONE; **§10** deviation policy (halt + restart-required); **§12** scope boundaries (claim/not-claim + SOTA composition reserved for PM); **§13** PM gates (Gate P + Gate D structure retained); **§15** related artefacts (predecessor).
|
||||
|
||||
### Parent chain
|
||||
|
||||
v4 predecessor pre-registration: anchor commit `dedd698`. v5 supersedes v4 for Stage 3 N=400 governance. v4's artefacts (probe FAIL at `66dcd5a`, scope audit at `69a14708`) inform v5 but do NOT re-pre-register.
|
||||
|
||||
### LiteLLM alias naming reconciliation
|
||||
|
||||
LiteLLM alias naming reconciliation: Step 2 `rpm: 20` edit applied to
|
||||
`gemini-3.1-pro` alias (`litellm-config.yaml:361-364`), which routes to
|
||||
upstream `gemini/gemini-3.1-pro-preview`. Brief §5.2 phrased this as
|
||||
"`gemini-3.1-pro-preview` alias"; terminology refers to upstream model
|
||||
name, not LiteLLM alias key. Both aliases share the same Google upstream
|
||||
bucket; see Fold-in 3.5b for sibling alias treatment.
|
||||
|
||||
---
|
||||
|
||||
## 1. Primary hypothesis (directional, confirmatory)
|
||||
|
||||
_Inherited verbatim from manifest v4 §1. No change._
|
||||
|
||||
> **Memory-lift at conv-scope retrieval exceeds zero-memory baseline.**
|
||||
>
|
||||
> `retrieval_judge_accuracy − no-context_judge_accuracy ≥ 5pp`
|
||||
>
|
||||
> evaluated at **Fisher exact one-sided** p-value **< 0.10**.
|
||||
|
||||
**One-sided justification:** theory-driven directional claim; ex-ante
|
||||
scaffolding from Gate B whole-corpus-leak dry-run (8/20 other-conv leak)
|
||||
+ Gate C monotonicity chain (no-context 0.10 < retrieval 0.35 < agentic
|
||||
0.40 < oracle 0.55 at N=20).
|
||||
|
||||
**Failure mode:** <2% probability at N=400 given Gate C's +25pp effect
|
||||
size (5× threshold). If primary fails despite coherent chain → PM
|
||||
adjudication on power-vs-signal question.
|
||||
|
||||
---
|
||||
|
||||
## 2. Secondary endpoints (ex-ante, non-blocking on primary)
|
||||
|
||||
_Inherited verbatim from manifest v4 §2. No change._
|
||||
|
||||
| # | Endpoint | Direction | Threshold | Test |
|
||||
|---|----------|-----------|-----------|------|
|
||||
| S1 | no-context ≤ retrieval | positive | ≥ 0pp | Fisher one-sided p < 0.20 |
|
||||
| S2 | retrieval ≤ agentic | positive | ≥ 0pp | Fisher one-sided p < 0.20 |
|
||||
| S3 | agentic ≤ oracle-context | positive | ≥ 0pp | Fisher one-sided p < 0.20 |
|
||||
| S4 | agentic − retrieval | positive | ≥ 0pp | descriptive + Wilson 95% CI |
|
||||
| S5 | oracle-context − full-context | positive (expected) | descriptive | descriptive (SYSTEM_EVOLVED strict-abstain diagnostic) |
|
||||
|
||||
Bench-Spec LOCK v1 multi-comparisons policy: secondary = descriptive, no correction required.
|
||||
|
||||
---
|
||||
|
||||
## 3. Sample design — **CHANGED (concurrency 2 → 1)**
|
||||
|
||||
- **Cells:** five, run in a single invocation. Cell definitions unchanged from v4:
|
||||
1. `no-context` — true zero-memory baseline.
|
||||
2. `oracle-context` — PM-facing alias for harness `raw`.
|
||||
3. `full-context` — oracle + SYSTEM_EVOLVED strict-abstain.
|
||||
4. `retrieval` — conv-scope HybridSearch top-K=20.
|
||||
5. `agentic` — softened SYSTEM_AGENTIC + bound search_memory + forced-fallback.
|
||||
- **N per cell:** 400 instances. **Unchanged.**
|
||||
- **Total evaluations:** 5 × 400 = **2000 judge-scored evaluations**. **Unchanged.**
|
||||
- **Instance selection seed:** `42`. **Unchanged.**
|
||||
- **Instance pool:** canonical LoCoMo dataset (see §4). **Unchanged.**
|
||||
- **Matched-pairs design:** same 400 instances flow through all cells. **Unchanged.**
|
||||
- **Concurrency (CHANGED):** **`--parallel-concurrency 1`**. Five cells run
|
||||
sequentially (no intra-wrapper parallelism). Five serial batches of one
|
||||
cell each. No inter-cell overlap.
|
||||
- _Rationale:_ §1.3 Gate P+ empirical finding — Gemini per-model 25-RPM cap.
|
||||
Concurrency=1 halves peak Gemini RPS (ceiling ≈ 12 Gemini/min at steady
|
||||
rate), providing margin under the 20 RPM LiteLLM-side throttle (§5.2
|
||||
addendum) + the 25 RPM Google-side cap.
|
||||
- _Impact on runtime:_ ~2× wall-clock vs v4's concurrency=2 assumption
|
||||
(see §14).
|
||||
|
||||
---
|
||||
|
||||
## 4. Dataset
|
||||
|
||||
_Inherited verbatim from manifest v4 §4. No change._
|
||||
|
||||
- **Source:** `benchmarks/data/locomo10.json` (snap-research LoCoMo).
|
||||
- **Upstream reference:** `https://raw.githubusercontent.com/snap-research/locomo/main/data/locomo10.json`.
|
||||
- **Paper reference:** Maharana et al., ACL-2024.
|
||||
- **Raw archive SHA-256:** `79fa87e90f04081343b8c8debecb80a9a6842b76a7aa537dc9fdf651ea698ff4`.
|
||||
- **Canonical build:** `benchmarks/data/locomo/locomo-1540.jsonl` (1531 instances after canonicalisation).
|
||||
- **Canonical dataset SHA-256:** `39e415e2f3a0fa1bd3cb1804a58d0b440b50d3070b2100698437e4ec402a5b24`.
|
||||
- **Selection:** 400 per cell via seed-42 shuffle + take-first-400.
|
||||
|
||||
---
|
||||
|
||||
## 5. Model stack
|
||||
|
||||
### 5.1 Subject route table (Qwen 3.6-35B-A3B-Thinking)
|
||||
|
||||
_Inherited verbatim from manifest v4 §5.1. No change._
|
||||
|
||||
| Priority | alias | thinking | max_tokens |
|
||||
|----------|-------|----------|------------|
|
||||
| primary | `qwen3.6-35b-a3b-via-dashscope-direct` | `on` | 16000 |
|
||||
| fallback_1 | `qwen3.6-35b-a3b-via-openrouter` | `on` | 64000 |
|
||||
| fallback_2 | `NOT_AVAILABLE` | — | — |
|
||||
|
||||
Pricing $0.20 / $0.80 per M in/out. `floating_alias` pinning. B3 addendum § 5.
|
||||
|
||||
### 5.2 Judge ensemble — **CHANGED (rpm:20 addendum)**
|
||||
|
||||
| Slot | alias | role | rate-limit (v5 addendum) |
|
||||
|------|-------|------|--------------------------|
|
||||
| primary_judge_1 | `claude-opus-4-7` | primary | none (Anthropic immutable, no observed rate-limit pressure at Stage 3 scale) |
|
||||
| primary_judge_2 | `gpt-5.4` | primary | none |
|
||||
| primary_judge_3 | `gemini-3.1-pro-preview` | primary | **`rpm: 20`** (v5 addendum via litellm-config.yaml §5.2 edit) |
|
||||
| tiebreak_reserve | `grok-4.20` | reserve (1/1/1 only) | none |
|
||||
|
||||
- **rpm: 20 addendum** is the single permitted §11 exception in v5. Applied to
|
||||
the `model_list` entry for `gemini-3.1-pro-preview` in `litellm-config.yaml`
|
||||
(Step 2 of the v5 emission path). Chosen 20 < 25 Google-side cap with a 5
|
||||
RPM margin for burst variance. Further throttle (rpm:15) reserved for PM
|
||||
adjudication if §1.3c throttle-verification probe FAILs at rpm:20.
|
||||
- **Tie-break path:** 1/1/1 → Grok 4.20. 2/2 → PM escalation. Unchanged.
|
||||
- **Consistency constraint:** same physical judge models as v4 / Stage 2-Retry.
|
||||
LiteLLM alias identifier unchanged (`gemini-3.1-pro-preview`); only the
|
||||
in-file rate-limit metadata added. Physical upstream model binding
|
||||
(`gemini/gemini-3.1-pro-preview` @ Google AI Studio) unchanged.
|
||||
- **κ monitoring:** Fleiss' κ thresholds inherited from Bench-Spec LOCK v1
|
||||
(pass-no-flag ≥ 0.65; pass-with-flag [0.60, 0.65]; halt ≤ 0.60).
|
||||
|
||||
### 5.3 Health-check predicate
|
||||
|
||||
_Inherited verbatim from manifest v4 §5.3. No change._
|
||||
|
||||
---
|
||||
|
||||
## 6. Substrate (conv-scope retrieval)
|
||||
|
||||
_Inherited verbatim from manifest v4 §6. No change._
|
||||
|
||||
- `@waggle/core::HybridSearch` (RRF-fused FTS5 + vec0).
|
||||
- `gopId = conversation_id` scope filter at `search.ts:14`.
|
||||
- Top-K default 20; upper clamp 50.
|
||||
- `createOllamaEmbedder()` + `nomic-embed-text` (1024 dims, local, $0).
|
||||
- Ingest batch 200.
|
||||
|
||||
### 6.1 Agentic-cell tool binding
|
||||
|
||||
_Inherited verbatim from manifest v4 §6.1. No change._
|
||||
|
||||
`makeSearchMemoryTool(substrate, 20, instance.conversation_id)`. `maxTurns=3`.
|
||||
180 s timeout. Forced-fallback on empty content + tool-use (0 firings at
|
||||
Gate C; load-bearing insurance).
|
||||
|
||||
---
|
||||
|
||||
## 7. SYSTEM_AGENTIC prompt — verbatim bytes locked
|
||||
|
||||
_Inherited verbatim from manifest v4 §7. No change._
|
||||
|
||||
**SHA-256:** `6facae6decc44a6404290514accb4f7cb364081b32d02847a20f8e871633e328` (1467 bytes, no trailing newline). Source: `benchmarks/harness/src/cells.ts` lines 75–102. Softened text from Stage 2-Retry Gate A (commit `373516c`). See manifest v4 §7 for full verbatim reproduction.
|
||||
|
||||
---
|
||||
|
||||
## 8. Stopping rules — **§7.4 updated for concurrency=1**
|
||||
|
||||
| # | Rule | Source | Trigger | Action |
|
||||
|---|------|--------|---------|--------|
|
||||
| §7.1 | Budget hard halt | `runner.ts` | cumulative spend ≥ **$28.00** | halt, persist partial, exit ping |
|
||||
| §7.2 | Streak halt | `streak-tracker.ts` | 3 consecutive subject fetch failures | halt, persist partial |
|
||||
| §7.3 | Pre-cell health check fail | `health-check.ts` | 5xx / fetch-error on subject or judge probe | halt before cell |
|
||||
| **§7.4 (v5)** | **Runner lock** | `runner-lock.ts` | concurrent cross-process invocation detected | halt. Under v5 concurrency=1 there is no intra-wrapper parallelism — the §1.1 cross-process-only waiver still governs, now with no exemption needed because no intra-wrapper spawning occurs. |
|
||||
| §7.5 | Pre-registration deviation | this document | any change to §1–§9 during run | halt + PM raise |
|
||||
|
||||
**No interim looks.** Halt only on the five conditions above.
|
||||
|
||||
---
|
||||
|
||||
## 9. Post-hoc exclusion policy: **NONE**
|
||||
|
||||
_Inherited verbatim from manifest v4 §9. No change._
|
||||
|
||||
All 2000 evals enter the denominator. `evaluator_loss` (judge-triple
|
||||
failure) counted in denominator, reported separately. No instance
|
||||
whitelist/blacklist. Selective exclusion forbidden ex-ante.
|
||||
|
||||
---
|
||||
|
||||
## 10. Deviation policy
|
||||
|
||||
_Inherited verbatim from manifest v4 §10. No change._
|
||||
|
||||
Any deviation from §1–§9 during run → (1) immediate halt, (2) PM raise,
|
||||
(3) re-pre-registration (manifest v6+) if accepted. Consistent with
|
||||
Bench-Spec LOCK v1 `preregistration.mid_run_amendment_policy: halt_restart_required`.
|
||||
|
||||
---
|
||||
|
||||
## 11. Code freeze — **updated with single permitted exception**
|
||||
|
||||
The following code is **frozen at HEAD `373516c`** for the duration of
|
||||
Stage 3 N=400 under v5. No changes permitted between v5 anchor commit
|
||||
and Gate D exit **EXCEPT** the single §5.2 addendum edit to
|
||||
`litellm-config.yaml` noted in §0.5 Delta Log and §5.2 (`rpm: 20` on
|
||||
the `gemini-3.1-pro-preview` alias block).
|
||||
|
||||
Frozen paths (inherited verbatim from v4 §11):
|
||||
|
||||
- Cell semantics (`benchmarks/harness/src/cells.ts`).
|
||||
- Substrate (`benchmarks/harness/src/substrate.ts`, `@waggle/core::HybridSearch`, `@waggle/core::FrameStore`, `@waggle/core::SessionStore`).
|
||||
- SYSTEM_AGENTIC + SYSTEM_AGENTIC_FORCED_FALLBACK prompts.
|
||||
- Agent loop (`@waggle/agent::runAgentLoop`, `@waggle/agent::tools.ts`).
|
||||
- Judge ensemble + routing (`benchmarks/harness/src/judge-*.ts`, `config/models.json`, `litellm-config.yaml` judge aliases EXCEPT the single rpm:20 addendum).
|
||||
- Subject route table (`config/models.json` qwen aliases).
|
||||
- Test suite.
|
||||
|
||||
Execution-only delta during N=400 run: new JSONL files emitted to
|
||||
`benchmarks/results/` AND the single pre-run `litellm-config.yaml`
|
||||
§5.2 addendum committed before the run. No other file modifications
|
||||
during or after run.
|
||||
|
||||
---
|
||||
|
||||
## 12. Scope boundaries
|
||||
|
||||
_Inherited verbatim from manifest v4 §12. No change._
|
||||
|
||||
### Can claim at Gate D:
|
||||
- Magnitude + significance of conv-scope retrieval memory-lift at Qwen-3.6-35b-a3b under HEAD 373516c.
|
||||
- Per-cell judge-accuracy with Wilson 95% CIs.
|
||||
- Monotonicity chain.
|
||||
- Conv-scope fair-comparison methodology.
|
||||
- Agentic discipline numbers.
|
||||
|
||||
### Cannot claim at Gate D:
|
||||
- Direct comparability to Mem0 91.6% (different scope + memory-synthesis layer).
|
||||
- Multi-model generalization (Qwen-only).
|
||||
- Production performance.
|
||||
|
||||
### Reserved for PM:
|
||||
- Public-claim phrasing + venue.
|
||||
- Matched-scope Mem0 co-run.
|
||||
- Publication timing.
|
||||
|
||||
**CC-1 does NOT compose public SOTA claim.** Scope + data only.
|
||||
|
||||
---
|
||||
|
||||
## 13. PM gates
|
||||
|
||||
_Inherited verbatim from manifest v4 §13. No change._
|
||||
|
||||
### Gate P+ (v5 pre-run, pre-N=400 execution)
|
||||
- Trigger: commit of v5 files + litellm-config.yaml §5.2 addendum on feature/c3-v3-wrapper.
|
||||
- Halt: CC-1 stops; no N=400 kick without PM GO after §1.3c + §1.3e.
|
||||
- PM checks: v5 content matches all prior ratifications; §1–§10 locked unambiguously.
|
||||
|
||||
### Gate D (post-run, pre-SOTA-claim)
|
||||
- Trigger: N=400 exit (clean or halted per §8).
|
||||
- Halt: CC-1 writes exit report at `PM-Waggle-OS/sessions/2026-04-24-task25-stage3-n400-complete.md`.
|
||||
- PM decides SOTA claim composition / publish gate / further scope.
|
||||
|
||||
No self-advance at either gate.
|
||||
|
||||
---
|
||||
|
||||
## 14. Budget — envelope unchanged; wall-clock re-estimated
|
||||
|
||||
- **Cap:** $30.00. **Unchanged.**
|
||||
- **Hard halt:** $28.00. **Unchanged.**
|
||||
- **Expected burn:** ~$23. **Unchanged.**
|
||||
- **Variance ceiling:** $28. **Unchanged.**
|
||||
- **Cost breakdown:** same (Qwen ~$2.50, judge triple ~$20, ollama $0, Grok ~$0.50).
|
||||
|
||||
**Wall-clock re-estimate (v5-specific):**
|
||||
- Under concurrency=1 + rpm:20 Gemini throttle, effective cell throughput
|
||||
is governed by the judge triple's slowest path. Gemini ≤ 20 RPM floor
|
||||
means ≤ 20 full triples/min (Opus + GPT + Gemini all complete).
|
||||
- At 400 instances × 5 cells = 2000 evals / 20 RPM = **100 min minimum**
|
||||
sustained-rate floor for the Gemini leg. Realistic wall-clock includes
|
||||
subject call latency + Opus/GPT judge latency adding to the triple
|
||||
critical path.
|
||||
- **Practical estimate:** 2–3 hours for the full N=400 run under v5.
|
||||
- **Upper bound:** if judge latency variance pushes instances past their
|
||||
expected rate, halt §7.1 at $28 caps the downside independently of time.
|
||||
|
||||
---
|
||||
|
||||
## 15. Related artefacts
|
||||
|
||||
- **Manifest v4 predecessor:** anchor commit `dedd69888e008fb1584bc249aff43b19f55a88e5` (short `dedd698`).
|
||||
- **§1.1 lock-semantics clarification (L-1 ratified):** anchor `67eb89914a49ec38049379bf952d5f62b82c188d` (short `67eb899`).
|
||||
- **§1.2 runner early-exit RCA (Task 2.6 tech-debt ratified):** anchor `274e9871b54599077a3d72de88d505550803a805` (short `274e987`).
|
||||
- **§1.3 Gate P+ probe FAIL:** anchor `66dcd5a1b18b9367662b04f1c9e1b66d855a9481` (short `66dcd5a`).
|
||||
- **§1.3b litellm-config scope audit (IN_SCOPE verdict):** anchor `69a14708f78a74d2cb7ef07faf2d949f6ffc3209` (short `69a14708`).
|
||||
- **Bench-Spec LOCK v1 parent:** `PM-Waggle-OS/decisions/2026-04-22-bench-spec-locked.manifest.yaml`.
|
||||
- **Stage 2-Retry Gate C exit:** `PM-Waggle-OS/sessions/2026-04-24-task25-stage2-retry-complete.md`.
|
||||
- **Stage 3 rekick brief (Option A / P4):** `PM-Waggle-OS/briefs/2026-04-24-cc-task25-stage3-rekick-option-a.md`.
|
||||
- **Stage 3 brief (original):** `PM-Waggle-OS/briefs/2026-04-24-cc-task25-stage3-n400-kickoff.md`.
|
||||
- **Rollback tag:** `checkpoint/pre-self-evolution-2026-04-14`.
|
||||
|
||||
---
|
||||
|
||||
_End of Manifest v5 pre-registration. This document is the anchor for all
|
||||
analysis choices at Stage 3 Gate D exit under the P4 (concurrency=1 + rpm:20)
|
||||
path._
|
||||
498
benchmarks/preregistration/manifest-v5-preregistration.yaml
Normal file
498
benchmarks/preregistration/manifest-v5-preregistration.yaml
Normal file
@@ -0,0 +1,498 @@
|
||||
# Manifest v5 — Task 2.5 Stage 3 N=400 Pre-Registration (RPM-Throttled)
|
||||
# Canonical markdown surface: manifest-v5-preregistration.md
|
||||
# Supersedes manifest v4 (anchor dedd698) — v5 governs all Stage 3 re-kick.
|
||||
# SHA-256 of this YAML's bytes recorded in v5 anchor commit message.
|
||||
|
||||
manifest_version: v5.0.0-preregistration
|
||||
manifest_type: stage_3_n400_preregistration_v5_rpm_throttled
|
||||
locked_date: 2026-04-24
|
||||
authority: PM (Marko Markovic) — P4 path (manifest v5 + concurrency=1) ratified 2026-04-24 on §1.3b IN_SCOPE verdict; inherits §1.1 + §1.2 ratifications
|
||||
sprint: 12
|
||||
task: 2.5
|
||||
stage: 3
|
||||
branch: feature/c3-v3-wrapper
|
||||
code_freeze_head: 373516c2784807da8536dbc0c194c54f4e4cd4be
|
||||
code_freeze_head_short: 373516c
|
||||
supersedes: manifest_v4_2026_04_24_anchor_dedd698
|
||||
inherits_from: bench_spec_lock_v1_2026_04_22
|
||||
|
||||
# ── v5 Delta Log ────────────────────────────────────────────────────────────
|
||||
|
||||
v5_delta_log:
|
||||
trigger:
|
||||
event: stage_3_gate_p_plus_probe_FAIL
|
||||
anchor_commit: 66dcd5a1b18b9367662b04f1c9e1b66d855a9481
|
||||
date: 2026-04-24
|
||||
observation: "24 / 50 HTTP 429 on gemini-3.1-pro-preview at 1.67 RPS steady"
|
||||
root_cause: "Google PRODUCT POLICY — per-model preview cap 25 RPM on generativelanguage.googleapis.com/generate_requests_per_model, independent of account billing tier"
|
||||
google_429_body_excerpt: "Quota exceeded for metric: generativelanguage.googleapis.com/generate_requests_per_model, limit: 25, model: gemini-3.1-pro"
|
||||
scope_audit:
|
||||
anchor_commit: 69a14708f78a74d2cb7ef07faf2d949f6ffc3209
|
||||
memo: benchmarks/results/manifest-v4-litellm-config-scope-audit.md
|
||||
verdict: IN_SCOPE
|
||||
rationale: "MD narrow (`litellm-config.yaml judge aliases`) + YAML strict (flat `litellm-config.yaml` in frozen_paths) both yield IN_SCOPE for any rpm:20 edit"
|
||||
p2_status: terminally_blocked
|
||||
fallback_path: P4_manifest_v5_concurrency_1
|
||||
changes_from_v4:
|
||||
sample_concurrency:
|
||||
v4: 2
|
||||
v5: 1
|
||||
rationale: "halves peak Gemini RPS; pairs with LiteLLM rpm:20 throttle for safety margin under 25 RPM Google cap"
|
||||
judge_ensemble_section_5_2:
|
||||
v4: "no rate-limit metadata on any judge alias"
|
||||
v5: "rpm: 20 addendum on gemini-3.1-pro-preview alias block in litellm-config.yaml"
|
||||
rationale: "20 < 25 Google cap with 5 RPM margin for burst variance"
|
||||
permitted_exception: true
|
||||
exception_scope: single_edit_single_file
|
||||
exception_target_path: litellm-config.yaml
|
||||
exception_target_line_range_approx: "361-364 (gemini-3.1-pro-preview alias block)"
|
||||
stopping_rule_7_4:
|
||||
v4: "concurrent_runners: FORBIDDEN (cross-process); §1.1 waiver permits intra-wrapper parallel"
|
||||
v5: "concurrent_runners: SEQUENTIAL (parallel-concurrency=1); §1.1 cross-process-only waiver still governs but moot under sequential design"
|
||||
code_freeze_section_11:
|
||||
v4: "HEAD 373516c; no file modifications during run"
|
||||
v5: "HEAD 373516c; the SINGLE permitted pre-run modification = litellm-config.yaml §5.2 addendum (rpm:20 on gemini-3.1-pro-preview). All other frozen paths unchanged."
|
||||
budget_wall_clock_estimate:
|
||||
v4: "40-60 min (optimistic, pre-empirical)"
|
||||
v5: "2-3 hours under concurrency=1 + rpm:20 Gemini throttle"
|
||||
cli_invocation:
|
||||
v4_flag: "--parallel-concurrency 2"
|
||||
v5_flag: "--parallel-concurrency 1"
|
||||
v5_manifest_flag: "--manifest benchmarks/preregistration/manifest-v5-preregistration.yaml"
|
||||
unchanged_from_v4:
|
||||
sections:
|
||||
- "§1 primary hypothesis (Fisher one-sided p<0.10 on retrieval − no-context ≥ 5pp)"
|
||||
- "§2 secondary endpoints S1-S5"
|
||||
- "§4 dataset (LoCoMo 1531 instances, raw SHA 79fa87e9..., canonical SHA 39e415e2...)"
|
||||
- "§6 substrate (HybridSearch conv-scope top-K=20, nomic-embed-text)"
|
||||
- "§7 SYSTEM_AGENTIC verbatim bytes (SHA-256 6facae6d..., 1467 bytes)"
|
||||
- "§9 post-hoc exclusion policy NONE"
|
||||
- "§10 deviation policy (halt + restart-required)"
|
||||
- "§12 scope boundaries + SOTA composition reserved for PM"
|
||||
- "§13 PM gates structure (Gate P / Gate D)"
|
||||
- "budget envelope §14 dollar amounts ($30/$28/~$23)"
|
||||
parent_chain:
|
||||
v4_predecessor: dedd69888e008fb1584bc249aff43b19f55a88e5
|
||||
section_1_1_lock_waiver: 67eb89914a49ec38049379bf952d5f62b82c188d
|
||||
section_1_2_rca: 274e9871b54599077a3d72de88d505550803a805
|
||||
section_1_3_probe_fail: 66dcd5a1b18b9367662b04f1c9e1b66d855a9481
|
||||
section_1_3b_scope_audit: 69a14708f78a74d2cb7ef07faf2d949f6ffc3209
|
||||
|
||||
# ── Field 7 slots (preregistration.ts PreregistrationManifestPayload) ───────
|
||||
|
||||
manifest_path: benchmarks/preregistration/manifest-v5-preregistration.yaml
|
||||
manifest_locked_at: 2026-04-24T00:00:00Z
|
||||
|
||||
dataset:
|
||||
name: locomo
|
||||
source_url: https://raw.githubusercontent.com/snap-research/locomo/main/data/locomo10.json
|
||||
raw_archive_path: benchmarks/data/locomo10.json
|
||||
raw_archive_sha256: 79fa87e90f04081343b8c8debecb80a9a6842b76a7aa537dc9fdf651ea698ff4
|
||||
raw_archive_bytes: 2805274
|
||||
canonical_path: benchmarks/data/locomo/locomo-1540.jsonl
|
||||
canonical_sha256: 39e415e2f3a0fa1bd3cb1804a58d0b440b50d3070b2100698437e4ec402a5b24
|
||||
canonical_instance_count: 1531
|
||||
paper_total_claim: 1540
|
||||
paper_reference: "Maharana et al., ACL-2024 — Evaluating Very Long-Term Conversational Memory of LLM Agents"
|
||||
category_distribution:
|
||||
single_hop: 841
|
||||
multi_hop: 281
|
||||
temporal: 320
|
||||
open_ended: 89
|
||||
|
||||
# ── Primary hypothesis (unchanged from v4) ──────────────────────────────────
|
||||
|
||||
primary_hypothesis:
|
||||
name: memory_lift_retrieval_vs_no_context
|
||||
direction: one_sided_positive
|
||||
statement: "retrieval_judge_accuracy − no-context_judge_accuracy ≥ 5pp"
|
||||
test: fisher_exact_one_sided
|
||||
alpha_threshold: 0.10
|
||||
effect_size_threshold_pp: 5
|
||||
justification_ex_ante:
|
||||
- gate_b_dry_run_conv_scope_20_of_20_vs_whole_corpus_8_of_20_leak_2026_04_24
|
||||
- gate_c_monotonicity_no_context_0_10_lt_retrieval_0_35_lt_agentic_0_40_lt_oracle_0_55
|
||||
|
||||
# ── Secondary endpoints (unchanged from v4) ─────────────────────────────────
|
||||
|
||||
secondary_endpoints:
|
||||
S1_monotonicity_no_context_leq_retrieval:
|
||||
direction: one_sided_positive
|
||||
threshold_pp: 0
|
||||
test: fisher_exact_one_sided
|
||||
alpha: 0.20
|
||||
S2_monotonicity_retrieval_leq_agentic:
|
||||
direction: one_sided_positive
|
||||
threshold_pp: 0
|
||||
test: fisher_exact_one_sided
|
||||
alpha: 0.20
|
||||
S3_monotonicity_agentic_leq_oracle_context:
|
||||
direction: one_sided_positive
|
||||
threshold_pp: 0
|
||||
test: fisher_exact_one_sided
|
||||
alpha: 0.20
|
||||
S4_agentic_lift_over_retrieval:
|
||||
direction: descriptive
|
||||
threshold_pp: 0
|
||||
report: [point_estimate, wilson_95_ci]
|
||||
S5_abstain_penalty_oracle_minus_full_context:
|
||||
direction: descriptive_expected_positive
|
||||
report: [point_estimate]
|
||||
|
||||
# ── Sample design (CHANGED: concurrency 2 → 1) ──────────────────────────────
|
||||
|
||||
sample:
|
||||
cells:
|
||||
- no-context
|
||||
- oracle-context
|
||||
- full-context
|
||||
- retrieval
|
||||
- agentic
|
||||
n_per_cell: 400
|
||||
total_evaluations: 2000
|
||||
instance_selection_seed: 42
|
||||
instance_selection_method: "shuffle-then-take-first-N, deterministic given seed"
|
||||
matched_pairs: true
|
||||
concurrency: 1
|
||||
concurrency_rationale: "§1.3 Gate P+ empirical finding — Gemini per-model 25 RPM cap. Concurrency=1 halves peak Gemini RPS; pairs with §5.2 addendum rpm:20 for 5 RPM margin."
|
||||
|
||||
# ── Cells semantics (unchanged from v4, frozen at HEAD 373516c) ─────────────
|
||||
|
||||
cells_semantics:
|
||||
no_context:
|
||||
system_prompt: SYSTEM_BASELINE
|
||||
user_prompt: "Question: {question}"
|
||||
memory_injection: none
|
||||
added_at: stage_2_retry_1_1_2026_04_24
|
||||
oracle_context:
|
||||
system_prompt: SYSTEM_BASELINE
|
||||
user_prompt: "Context: {instance.context}\\n\\nQuestion: {instance.question}"
|
||||
memory_injection: oracle_fed_by_locomo
|
||||
harness_alias: raw
|
||||
full_context:
|
||||
system_prompt: SYSTEM_EVOLVED
|
||||
memory_injection: oracle_fed_plus_evolved_abstain
|
||||
retrieval:
|
||||
system_prompt: SYSTEM_BASELINE
|
||||
substrate: waggle_core_hybrid_search
|
||||
scope: conversation_scoped_via_gopId
|
||||
top_k_default: 20
|
||||
top_k_upper_clamp: 50
|
||||
agentic:
|
||||
system_prompt: SYSTEM_AGENTIC_softened_stage2_retry
|
||||
system_prompt_sha256: 6facae6decc44a6404290514accb4f7cb364081b32d02847a20f8e871633e328
|
||||
system_prompt_bytes: 1467
|
||||
tool_allowlist:
|
||||
- search_memory
|
||||
tool_binding: "search_memory bound to instance.conversation_id; non-overridable"
|
||||
max_turns: 3
|
||||
timeout_ms: 180000
|
||||
forced_answer_fallback:
|
||||
enabled: true
|
||||
system_prompt: SYSTEM_AGENTIC_FORCED_FALLBACK
|
||||
gate_c_firing_rate: 0
|
||||
|
||||
# ── Model stack (CHANGED: §5.2 addendum rpm:20 on Gemini alias) ─────────────
|
||||
|
||||
subject_model: qwen3.6-35b-a3b-via-dashscope-direct
|
||||
subject_fallback_1: qwen3.6-35b-a3b-via-openrouter
|
||||
subject_fallback_2: NOT_AVAILABLE
|
||||
|
||||
subject_route_table:
|
||||
primary:
|
||||
alias: qwen3.6-35b-a3b-via-dashscope-direct
|
||||
litellm_model: qwen3.6-35b-a3b-via-dashscope-direct
|
||||
upstream_route: "LiteLLM local alias -> openai/qwen3.6-35b-a3b @ https://dashscope-intl.aliyuncs.com/compatible-mode/v1"
|
||||
provider: alibaba
|
||||
thinking: on
|
||||
max_tokens: 16000
|
||||
price_per_million_input_usd: 0.20
|
||||
price_per_million_output_usd: 0.80
|
||||
context_window: 262144
|
||||
pinning_surface: floating_alias
|
||||
fallback_1:
|
||||
alias: qwen3.6-35b-a3b-via-openrouter
|
||||
litellm_model: qwen3.6-35b-a3b-via-openrouter
|
||||
upstream_route: "LiteLLM -> OpenRouter bridge (openrouter/qwen/qwen3.5-35b-a3b)"
|
||||
thinking: on
|
||||
max_tokens: 64000
|
||||
pinning_surface: floating_alias
|
||||
trigger_condition: fetch_error_on_primary
|
||||
fallback_2:
|
||||
alias: NOT_AVAILABLE
|
||||
|
||||
judge_ensemble:
|
||||
primary:
|
||||
- judge_role: primary
|
||||
slot: primary_judge_1
|
||||
model_id: claude-opus-4-7
|
||||
provider: anthropic
|
||||
litellm_model: claude-opus-4-7
|
||||
pinning_surface: anthropic_immutable
|
||||
rate_limit_v5: null
|
||||
price_per_million_input_usd: 15.00
|
||||
price_per_million_output_usd: 75.00
|
||||
- judge_role: primary
|
||||
slot: primary_judge_2
|
||||
model_id: gpt-5.4
|
||||
provider: openai_via_openrouter
|
||||
litellm_model: gpt-5.4
|
||||
pinning_surface: floating_alias
|
||||
rate_limit_v5: null
|
||||
price_per_million_input_usd: 10.00
|
||||
price_per_million_output_usd: 30.00
|
||||
- judge_role: primary
|
||||
slot: primary_judge_3
|
||||
model_id: gemini-3.1-pro-preview
|
||||
provider: google
|
||||
litellm_model: gemini-3.1-pro-preview
|
||||
pinning_surface: floating_alias
|
||||
rate_limit_v5:
|
||||
rpm: 20
|
||||
rationale: "v5 §5.2 addendum under 25 RPM Google per-model cap (§1.3 probe root cause); 5 RPM margin for burst variance"
|
||||
delivery_mechanism: "inline in litellm-config.yaml model_list[gemini-3.1-pro-preview] block (the SINGLE permitted §11 exception)"
|
||||
fail_escalation:
|
||||
- rpm_15_if_probe_v2_FAIL_at_20
|
||||
- openrouter_route_swap_P7_if_throttle_insufficient
|
||||
price_per_million_input_usd: 3.50
|
||||
price_per_million_output_usd: 10.50
|
||||
tiebreak:
|
||||
judge_role: reserve
|
||||
model_id: grok-4.20
|
||||
provider: xai_via_openrouter
|
||||
litellm_model: openrouter/x-ai/grok-4.20
|
||||
rate_limit_v5: null
|
||||
trigger: three_way_split_1_1_1
|
||||
defensive_2_2_path: pm-escalation
|
||||
consistency_constraint: same_physical_judge_models_as_stage_1_stage_1_5_stage_2_stage_2_retry_v4
|
||||
vote_policy: majority_with_grok_reserve_on_1_1_1_split
|
||||
|
||||
judge_primary:
|
||||
id: claude-opus-4-7
|
||||
judge_secondary:
|
||||
id: gpt-5.4
|
||||
judge_tie_breaker:
|
||||
id: gemini-3.1-pro-preview
|
||||
|
||||
# ── Substrate (unchanged from v4) ───────────────────────────────────────────
|
||||
|
||||
substrate:
|
||||
implementation: "@waggle/core::HybridSearch (RRF-fused FTS5 + vec0)"
|
||||
scope_filter:
|
||||
parameter: gopId
|
||||
source_location: packages/core/src/mind/search.ts:14
|
||||
field_name: SearchOptions.gopId
|
||||
benchmark_binding: instance.conversation_id
|
||||
top_k_default: 20
|
||||
top_k_upper_clamp: 50
|
||||
embedder:
|
||||
factory: createOllamaEmbedder
|
||||
base_url: http://localhost:11434
|
||||
model: nomic-embed-text
|
||||
dims: 1024
|
||||
cost: zero_local_inference
|
||||
ingest_batch_size: 200
|
||||
|
||||
# ── κ, CI, failure taxonomy (all unchanged from v4) ─────────────────────────
|
||||
|
||||
kappa_monitoring:
|
||||
baseline_reference: sprint_10_task_2_2_kappa_0_7458
|
||||
compute: fleiss_kappa_on_pre_tiebreak_vote_matrix
|
||||
thresholds:
|
||||
pass_no_flag_kappa_min: 0.65
|
||||
pass_with_flag_kappa_range: [0.60, 0.65]
|
||||
halt_kappa_max: 0.60
|
||||
halt_drop_from_baseline_max_pp: 10
|
||||
|
||||
confidence_intervals:
|
||||
primary:
|
||||
method: wilson_score_95
|
||||
secondary:
|
||||
method: cluster_bootstrap_95
|
||||
iterations: 10000
|
||||
seed: 42
|
||||
cluster_unit: conversation_id
|
||||
|
||||
failure_taxonomy:
|
||||
version: v1
|
||||
categories:
|
||||
- {code: F1, name: contradicts_ground_truth}
|
||||
- {code: F2, name: partial_answer}
|
||||
- {code: F3, name: off_topic}
|
||||
- {code: F4, name: refusal}
|
||||
- {code: F5, name: tool_use_error}
|
||||
- {code: F6, name: format_violation}
|
||||
|
||||
# ── Stopping rules (§7.4 updated under concurrency=1) ───────────────────────
|
||||
|
||||
stopping_rules:
|
||||
budget_hard_halt_usd: 28.00
|
||||
budget_cap_usd: 30.00
|
||||
streak_halt: "3 consecutive subject fetch failures -> halt (streak-tracker.ts)"
|
||||
pre_cell_health_check: "GET /health/liveliness + POST /v1/chat/completions ping per model -> halt on any 5xx/fetch-error"
|
||||
runner_lock: "concurrent_runners: SEQUENTIAL (parallel-concurrency=1); §1.1 cross-process waiver still governs — moot under sequential design"
|
||||
deviation_from_preregistration: "any change to §1-§9 during run -> immediate halt + PM raise"
|
||||
no_interim_looks: true
|
||||
mid_run_amendment_policy: halt_restart_required
|
||||
|
||||
# ── Post-hoc exclusion: NONE (unchanged from v4) ────────────────────────────
|
||||
|
||||
post_hoc_exclusion:
|
||||
policy: none
|
||||
evaluator_loss_handling:
|
||||
included_in_denominator: true
|
||||
reported_separately: true
|
||||
denominator_formula: "correct + incorrect + evaluator_loss"
|
||||
|
||||
# ── Budget (envelope unchanged; wall-clock re-estimated) ────────────────────
|
||||
|
||||
budget:
|
||||
cap_usd: 30.00
|
||||
hard_halt_usd: 28.00
|
||||
expected_burn_usd: 23.00
|
||||
variance_ceiling_usd: 28.00
|
||||
breakdown_expected:
|
||||
subject_qwen_dashscope_direct_usd: 2.50
|
||||
judge_triple_opus_gpt5_gemini_usd: 20.00
|
||||
embedding_ollama_local_usd: 0.00
|
||||
tie_break_grok_reserve_usd: 0.50
|
||||
wall_clock_estimate:
|
||||
v4_optimistic_min: 40
|
||||
v4_optimistic_max: 60
|
||||
v5_realistic_hours_min: 2
|
||||
v5_realistic_hours_max: 3
|
||||
v5_floor_minutes_derived_from_rpm_20_and_2000_evals: 100
|
||||
|
||||
# ── Target sample + CLI invocation (v5-specific) ────────────────────────────
|
||||
|
||||
target_N: 400
|
||||
target_cells:
|
||||
- no-context
|
||||
- oracle-context
|
||||
- full-context
|
||||
- retrieval
|
||||
- agentic
|
||||
target_total_evaluations: 2000
|
||||
target_concurrency: 1
|
||||
|
||||
cli_invocation_template: >
|
||||
npx tsx scripts/run-mini-locomo.ts
|
||||
--manifest benchmarks/preregistration/manifest-v5-preregistration.yaml
|
||||
--subject qwen3.6-35b-a3b-via-dashscope-direct
|
||||
--subject-fallback-1 qwen3.6-35b-a3b-via-openrouter
|
||||
--judge-ensemble claude-opus-4-7,gpt-5.4,gemini-3.1-pro
|
||||
--v3-cells --N 400 --parallel-concurrency 1 --seed 42
|
||||
|
||||
# ── Code freeze — SINGLE permitted exception ────────────────────────────────
|
||||
|
||||
code_freeze:
|
||||
head: 373516c2784807da8536dbc0c194c54f4e4cd4be
|
||||
branch: feature/c3-v3-wrapper
|
||||
frozen_paths:
|
||||
- benchmarks/harness/src/cells.ts
|
||||
- benchmarks/harness/src/substrate.ts
|
||||
- benchmarks/harness/src/judge-client.ts
|
||||
- benchmarks/harness/src/judge-runner.ts
|
||||
- benchmarks/harness/src/health-check.ts
|
||||
- benchmarks/harness/src/streak-tracker.ts
|
||||
- benchmarks/harness/src/runner-lock.ts
|
||||
- benchmarks/harness/src/runner.ts
|
||||
- benchmarks/harness/config/models.json
|
||||
- packages/agent/src/agent-loop.ts
|
||||
- packages/agent/src/tools.ts
|
||||
- packages/core/src/mind/search.ts
|
||||
- packages/core/src/mind/frames.ts
|
||||
- packages/core/src/mind/sessions.ts
|
||||
- packages/core/src/mind/db.ts
|
||||
- litellm-config.yaml
|
||||
permitted_delta_during_run:
|
||||
- "new JSONL files emitted to benchmarks/results/ by the N=400 run"
|
||||
permitted_pre_run_delta_v5_single_exception:
|
||||
path: litellm-config.yaml
|
||||
scope: "add rpm: 20 to gemini-3.1-pro-preview alias block (model_list entry); no other fields modified"
|
||||
committed_separately_before_run: true
|
||||
justification: "manifest v5 §5.2 addendum; only permitted exception to §11 freeze; governed by v5 delta log"
|
||||
|
||||
# ── Deviation policy (unchanged from v4) ────────────────────────────────────
|
||||
|
||||
deviation_policy:
|
||||
on_detection:
|
||||
- immediate_halt
|
||||
- pm_raise
|
||||
- re_preregister_new_manifest_v6_if_accepted
|
||||
|
||||
# ── PM gates (unchanged from v4 structure) ──────────────────────────────────
|
||||
|
||||
pm_gates:
|
||||
gate_p_plus_v5_pre_run:
|
||||
trigger: "anchor commit of v5 md + yaml + litellm-config.yaml §5.2 addendum on feature/c3-v3-wrapper"
|
||||
pre_kick_checks_required:
|
||||
- section_1_1_lock_semantics_ratified
|
||||
- section_1_2_rca_ratified
|
||||
- section_1_3_probe_v1_FAIL_adjudicated
|
||||
- section_1_3b_scope_audit_ratified
|
||||
- section_1_3c_throttle_probe_v2_PASS
|
||||
- section_1_3e_rpd_feasibility_FEASIBLE
|
||||
action: "CC-1 halts; awaits GATE-D-REKICK-GO after §1.3c + §1.3e"
|
||||
gate_d_post_run:
|
||||
trigger: "N=400 run exit (clean or halted per stopping_rules)"
|
||||
action: "CC-1 writes Gate D exit report at PM-Waggle-OS/sessions/2026-04-24-task25-stage3-n400-complete.md; halts"
|
||||
outcomes:
|
||||
- compose_sota_claim_authority_pm
|
||||
- publish_gate
|
||||
- further_scope
|
||||
cc1_self_advance: forbidden_at_both_gates
|
||||
|
||||
# ── Scope boundaries (unchanged from v4) ────────────────────────────────────
|
||||
|
||||
scope_boundaries:
|
||||
claimable_at_gate_d:
|
||||
- memory_lift_magnitude_and_significance_conv_scope_qwen_harness_head_373516c
|
||||
- per_cell_judge_accuracy_wilson_95
|
||||
- monotonicity_chain_observation_5_cell
|
||||
- conv_scope_fair_comparison_methodology
|
||||
- agentic_discipline_search_rate_turns_unknown_fallback
|
||||
not_claimable_at_gate_d:
|
||||
- direct_comparability_to_mem0_91_6_different_scope_and_memory_layer
|
||||
- multi_model_generalization_stage_3_is_qwen_only
|
||||
- production_waggle_orchestrator_performance
|
||||
reserved_for_pm_at_gate_d:
|
||||
- public_claim_phrasing_venue
|
||||
- matched_scope_mem0_co_run_stage_4
|
||||
- publication_timing
|
||||
cc1_does_not_compose_public_sota_claim: true
|
||||
|
||||
# ── Related artefacts ───────────────────────────────────────────────────────
|
||||
|
||||
related:
|
||||
v4_predecessor: dedd69888e008fb1584bc249aff43b19f55a88e5
|
||||
section_1_1_lock_waiver: 67eb89914a49ec38049379bf952d5f62b82c188d
|
||||
section_1_2_rca: 274e9871b54599077a3d72de88d505550803a805
|
||||
section_1_3_probe_fail: 66dcd5a1b18b9367662b04f1c9e1b66d855a9481
|
||||
section_1_3b_scope_audit: 69a14708f78a74d2cb7ef07faf2d949f6ffc3209
|
||||
bench_spec_lock_v1_parent: PM-Waggle-OS/decisions/2026-04-22-bench-spec-locked.manifest.yaml
|
||||
stage_2_retry_gate_c_exit: PM-Waggle-OS/sessions/2026-04-24-task25-stage2-retry-complete.md
|
||||
stage_3_rekick_brief_option_a: PM-Waggle-OS/briefs/2026-04-24-cc-task25-stage3-rekick-option-a.md
|
||||
rollback_tag: checkpoint/pre-self-evolution-2026-04-14
|
||||
canonical_md_surface: benchmarks/preregistration/manifest-v5-preregistration.md
|
||||
|
||||
# ── Validation gates ────────────────────────────────────────────────────────
|
||||
|
||||
validation_gates:
|
||||
before_n400_kickoff_v5:
|
||||
- v5_anchor_commit_sha_recorded
|
||||
- v5_md_sha256_recorded_in_commit_message
|
||||
- v5_yaml_sha256_recorded_in_commit_message
|
||||
- litellm_config_rpm_20_edit_committed_separately
|
||||
- section_1_3c_throttle_probe_v2_PASS
|
||||
- section_1_3e_rpd_feasibility_FEASIBLE
|
||||
- pm_gate_d_rekick_go_received
|
||||
- kickoff_mechanism_clean_foreground_non_harness_process_tree
|
||||
at_gate_d_exit_v5:
|
||||
- all_2000_evals_accounted_in_denominators
|
||||
- evaluator_loss_reported_separately
|
||||
- primary_fisher_one_sided_computed
|
||||
- secondary_endpoints_reported
|
||||
- budget_reconciled
|
||||
- deviation_count_reported
|
||||
- code_freeze_reverified_head_373516c_plus_single_litellm_addendum
|
||||
478
benchmarks/preregistration/manifest-v6-preregistration.md
Normal file
478
benchmarks/preregistration/manifest-v6-preregistration.md
Normal file
@@ -0,0 +1,478 @@
|
||||
# Manifest v6 — Task 2.5 Stage 3 N=400 Pre-Registration (Judge Ensemble Swap)
|
||||
|
||||
**Manifest version:** v6.0.0-preregistration
|
||||
**Manifest type:** `stage_3_n400_preregistration_v6_ensemble_swap`
|
||||
**Preregistered date:** 2026-04-24
|
||||
**Authority:** PM (Marko Marković) — §2.0 + §2.1 Phase 1 authorization 2026-04-24 on closure of the full §1.3f → §1.3h-C judge swap validation sequence. Inherits §1.1 lock-semantics waiver + §1.2 RCA + §1.3 throttle chain ratifications from v5.
|
||||
**Branch:** `feature/c3-v3-wrapper`
|
||||
**Code freeze:** HEAD `373516c2784807da8536dbc0c194c54f4e4cd4be` (short `373516c`) — frozen EXCEPT the v5 §5.2 Gemini `rpm:20` addendum (retained as audit artefact) and the v6 `litellm-config.yaml` supersession amendment committed separately as Phase 1 Commit 2 (adds `minimax-m27-via-openrouter` + `kimi-k26-direct` aliases).
|
||||
**Supersedes:** Manifest v5 (anchor commit `fc16925`). v5 remains audit-immutable predecessor. v6 governs all Stage 3 N=400 re-kick from this commit forward.
|
||||
**Inherits:** Bench-Spec LOCK v1 (`decisions/2026-04-22-bench-spec-locked.manifest.yaml`).
|
||||
**Machine-readable twin:** [`manifest-v6-preregistration.yaml`](manifest-v6-preregistration.yaml)
|
||||
|
||||
---
|
||||
|
||||
## 0. Status
|
||||
|
||||
**PRE-REGISTERED — PHASE 1 KICK (κ RE-CAL + CONFIG AMENDMENT). PENDING PM RATIFICATION FOR PHASE 2 (N=400 EXECUTION).**
|
||||
|
||||
This document supersedes manifest v5 (anchor `fc16925`). v6's trigger and scope deltas are enumerated in §0.5 below. All other sections inherit v5 verbatim for tamper-evident standalone audit. Any change to v6's success criteria, statistical tests, sample design, or scope after v6's anchor commit invalidates the pre-registration and requires a new PM-ratified decision document (manifest v7+).
|
||||
|
||||
**Anchor commit SHA:** recorded in the git commit that adds these files.
|
||||
**Manifest SHA-256 (MD + YAML bytes):** computed at anchor-commit time via
|
||||
`sha256sum benchmarks/preregistration/manifest-v6-preregistration.{md,yaml}` — recorded in the commit message body for tamper-evident audit trail.
|
||||
|
||||
---
|
||||
|
||||
## 0.5. v6 Delta Log (NEW — v6-specific)
|
||||
|
||||
### Trigger
|
||||
|
||||
**§1.3f-§1.3h-C judge swap validation sequence (2026-04-24):**
|
||||
|
||||
Starting from v5's §1.3 Gate P+ probe FAIL + §1.3c throttle probe PASS + §1.3e RPD feasibility check INFEASIBLE@250 / FEASIBLE@2500, Stage 3 N=400 remained blocked by Google quota ceiling for `gemini-3.1-pro-preview`. PM approved two parallel paths: (A) Google quota relief ticket (unresolved; indefinite wait), (B) judge swap to a non-Google flagship reasoning model.
|
||||
|
||||
**§1.3f (anchor `8ad0567`, 2026-04-24):** Vertex AI Batch Prediction eligibility probe for `gemini-3.1-pro-preview`. Outcome: **INFEASIBLE**. Vertex v1beta does not list this preview model in the batch-eligible catalog; no publisher/model endpoint accepts batch ingestion. Branch A closed.
|
||||
|
||||
**§1.3g (anchor `8a2f0e6`, 2026-04-24):** 4-candidate judge swap κ probe (Kimi K2.6 + MiniMax M2.7 + DeepSeek V4 Pro + Zhipu GLM-5.1) on 20 stratified instances (first-4-per-cell from the 100-row v5 κ calibration set at `benchmarks/results/locomo-mini-n20-retry-2026-04-24T00-02-12Z.jsonl`). Outcome: **MULTI_PASS** with methodological caveat — κ=1.0 across all 4 on the unanimous-biased subset (0/20 Opus-GPT splits vs full-set 7% split rate). Operational ranking (Zhipu > DeepSeek > MiniMax > Kimi on speed × parse × direct) was heuristic only, not empirical κ discrimination.
|
||||
|
||||
**§1.3h (anchor `ae0d312`, 2026-04-24):** PM-adjudicated stratified discriminating re-probe on the 7 available Opus≠GPT split cases (PM-amended min 7 under §1.3H-POOL-SHORTAGE OPTION 1). Executed 28 calls (7 × 4 candidates) with MiniMax direct-first routing test. Outcome: **INCONCLUSIVE_BUT_OPERATIONAL_SIGNAL** — split-only κ structurally degenerate (all 7 splits Opus=correct / GPT=incorrect → reference column has no variance). Informative signal = correctness on oriented splits (agreement with verified-correct Opus reference):
|
||||
- MiniMax: 6/7 = **86%** (best)
|
||||
- Kimi: 4/5 = 80%
|
||||
- DeepSeek: 2/5 = 40% (mis-calibrated)
|
||||
- Zhipu: 0/6 = **0% — GPT-echo, DISQUALIFIED** (violates ensemble independence assumption)
|
||||
|
||||
MiniMax direct routing failed both `api.minimaxi.com` and `api.minimax.chat` v2 endpoints (MINIMAX_GROUP_ID did not unblock); OpenRouter fallback 7/7 parse.
|
||||
|
||||
**§1.3h-C (anchor `005a19a`, 2026-04-24):** DeepSeek `max_tokens` 1024→2048 bump verification on same 7-split sample. Outcome: **truncation_fixable_but_correctness_regressed** — parse 5/7 → 7/7 (truncation confirmed as root cause of NULLs), but correctness 40% → 14% (longer reasoning budget made DeepSeek more GPT-strict, moving further from verified-correct Opus reference). DeepSeek DISQUALIFIED on correctness grounds regardless of parse fix.
|
||||
|
||||
### Final ensemble selection ratified 2026-04-24
|
||||
|
||||
| Role | Model | Selection rationale | Routing |
|
||||
|------|-------|----------------------|---------|
|
||||
| primary_judge_1 | Claude Opus 4.7 | inherited from v5 (unchanged) | anthropic direct |
|
||||
| primary_judge_2 | GPT-5.4 | inherited from v5 (unchanged) | openai direct |
|
||||
| **primary_judge_3** | **MiniMax M2.7** | **86% correct on splits (best empirical fit), 100% parse via OR** | **openrouter (direct failed)** |
|
||||
| **backup_judge** | **Kimi K2.6** | **80% correct on splits, per-instance failover on primary_judge_3 failure** | **moonshot direct** |
|
||||
|
||||
### Disqualified candidates (audit trail)
|
||||
|
||||
| Model | DQ reason | Evidence anchor |
|
||||
|-------|-----------|------------------|
|
||||
| Gemini 3.1 Pro Preview | Google per-model 25 RPM cap + Vertex batch INFEASIBLE | §1.3 `66dcd5a` + §1.3e `1d3851d` + §1.3f `8ad0567` |
|
||||
| Zhipu GLM-5.1 | 100% GPT-echo on splits (p_opus=0%, p_gpt=100%) — violates ensemble independence | §1.3h `ae0d312` |
|
||||
| DeepSeek V4-Pro | 14% correctness on splits at mt=2048 (regressed from 40% at mt=1024); GPT-alignment escalates with reasoning depth | §1.3h-C `005a19a` |
|
||||
|
||||
### Changes from v5
|
||||
|
||||
| # | Section | v5 | v6 |
|
||||
|---|---------|-----|-----|
|
||||
| §5.2 | Judge ensemble | Opus + GPT + Gemini-3.1-Pro-Preview primary; Grok-4.20 reserve (1/1/1 only) | **Opus + GPT + MiniMax-M2.7 primary; Kimi-K2.6 backup (per-instance failover)** |
|
||||
| §5.2 | Tie-break policy | majority + Grok-4.20 on 1/1/1 split | **primary 3-judge majority; backup activates per-instance on MiniMax failure; three-way 1/1/1 → PM escalation (no reserve judge in v6)** |
|
||||
| §5.2 | Rate-limit metadata | `rpm: 20` on `gemini-3.1-pro-preview` (v5 addendum) | **No active rpm:20 in judge path (Gemini alias retained but unused); MiniMax OR rpm + Kimi Moonshot rpm TBD at §1.3c-v6 probe time** |
|
||||
| §11 | Code freeze `litellm-config.yaml` | Frozen except v5 §5.2 Gemini rpm:20 addendum | **v5 freeze superseded; v6 amendment adds `minimax-m27-via-openrouter` + `kimi-k26-direct` aliases + retains all v5 entries (Gemini alias with rpm:20 kept as orphan audit artefact). Post-amendment state pinned by v6 §11.** |
|
||||
| §14 | Budget envelope | $30 cap / $28 halt / ~$23 expected | **$60 cap / $55 halt / ~$50 expected (Phase 1 κ re-cal ~$25 + Phase 2 N=400 ~$25; no preview-model premium)** |
|
||||
| §0.5 | Delta log | v4→v5 trigger from §1.3 probe FAIL + §1.3b IN_SCOPE + naming reconciliation | **v5→v6 trigger from §1.3f → §1.3h-C sequence closure; MiniMax primary + Kimi backup selection rationale; Zhipu/DeepSeek DQ; κ re-cal methodology** |
|
||||
| §13 | PM gates | Gate P+ (v5 pre-run) + Gate D (post-run) | **Gate P++ (v6 Phase 1: κ re-cal + config amendment) + Gate P+++ (v6 Phase 2 kick = PM-RATIFY-V6-KAPPA) + Gate D (post-run unchanged)** |
|
||||
|
||||
### UNCHANGED from v5 (verbatim inheritance)
|
||||
|
||||
- **§1** primary hypothesis (Fisher one-sided p<0.10 on retrieval − no-context ≥ 5pp)
|
||||
- **§2** secondary endpoints (S1–S5)
|
||||
- **§3** sample design (concurrency=1; five cells sequential; N=400 per cell; seed=42)
|
||||
- **§4** dataset (LoCoMo 1531 instances, raw SHA `79fa87e9...`, canonical SHA `39e415e2...`)
|
||||
- **§5.1** subject route table (Qwen 3.6-35B-A3B-Thinking, DashScope-intl direct primary, OR fallback)
|
||||
- **§5.3** health-check predicate
|
||||
- **§6** substrate (HybridSearch conv-scope top-K=20, nomic-embed-text)
|
||||
- **§7** SYSTEM_AGENTIC verbatim bytes (SHA-256 `6facae6d...`, 1467 bytes)
|
||||
- **§8** stopping rules (budget + streak + pre-cell health + §1.1 waiver + deviation)
|
||||
- **§9** post-hoc exclusion policy NONE
|
||||
- **§10** deviation policy (halt + restart-required)
|
||||
- **§12** scope boundaries (claim/not-claim + SOTA composition reserved for PM)
|
||||
- **§15** related artefacts (predecessor chain extended to include v6 ancestry)
|
||||
|
||||
### Parent chain (extended)
|
||||
|
||||
| Phase | Anchor | Note |
|
||||
|-------|--------|------|
|
||||
| v4 | `dedd698` | obsolete predecessor pre-reg |
|
||||
| §1.1 lock waiver | `67eb899` | ratified |
|
||||
| §1.2 RCA | `274e987` | ratified |
|
||||
| §1.3 probe FAIL | `66dcd5a` | preview 25 RPM discovery |
|
||||
| §1.3b scope audit | `69a14708` | IN_SCOPE verdict |
|
||||
| v5 emission | `fc16925` | manifest-v5 anchor (throttle config) |
|
||||
| §5.2 rpm:20 edit | `ad324cc` | v5 §11 exception |
|
||||
| §1.3c throttle probe PASS | `3a146ef` | empirical verification |
|
||||
| Fold-in 3.5b sibling mirror | `d0ab680` | defensive rpm:20 on sibling alias |
|
||||
| §1.3e RPD feasibility | `1d3851d` | INFEASIBLE@250, FEASIBLE@2500 |
|
||||
| §1.3f Vertex Batch | `8ad0567` | INFEASIBLE → Branch A closed |
|
||||
| §1.3g Judge swap MULTI_PASS | `8a2f0e6` | 4-candidate κ=1.0 (unanimous-biased) |
|
||||
| §1.3h Stratified re-probe | `ae0d312` | INCONCLUSIVE_BUT_OPERATIONAL_SIGNAL (bias exposed) |
|
||||
| §1.3h-C DeepSeek mt bump | `005a19a` | truncation_fixable_but_correctness_regressed |
|
||||
| **v6 emission** | **THIS COMMIT** | **manifest-v6 anchor** |
|
||||
|
||||
---
|
||||
|
||||
## 1. Primary hypothesis (directional, confirmatory)
|
||||
|
||||
_Inherited verbatim from manifest v5 §1 (which inherited verbatim from v4 §1). No change._
|
||||
|
||||
> **Memory-lift at conv-scope retrieval exceeds zero-memory baseline.**
|
||||
>
|
||||
> `retrieval_judge_accuracy − no-context_judge_accuracy ≥ 5pp`
|
||||
>
|
||||
> evaluated at **Fisher exact one-sided** p-value **< 0.10**.
|
||||
|
||||
**One-sided justification:** theory-driven directional claim; ex-ante scaffolding from Gate B whole-corpus-leak dry-run (8/20 other-conv leak) + Gate C monotonicity chain (no-context 0.10 < retrieval 0.35 < agentic 0.40 < oracle 0.55 at N=20).
|
||||
|
||||
**Failure mode:** <2% probability at N=400 given Gate C's +25pp effect size (5× threshold). If primary fails despite coherent chain → PM adjudication on power-vs-signal question.
|
||||
|
||||
---
|
||||
|
||||
## 2. Secondary endpoints (ex-ante, non-blocking on primary)
|
||||
|
||||
_Inherited verbatim from manifest v5 §2. No change._
|
||||
|
||||
| # | Endpoint | Direction | Threshold | Test |
|
||||
|---|----------|-----------|-----------|------|
|
||||
| S1 | no-context ≤ retrieval | positive | ≥ 0pp | Fisher one-sided p < 0.20 |
|
||||
| S2 | retrieval ≤ agentic | positive | ≥ 0pp | Fisher one-sided p < 0.20 |
|
||||
| S3 | agentic ≤ oracle-context | positive | ≥ 0pp | Fisher one-sided p < 0.20 |
|
||||
| S4 | agentic − retrieval | positive | ≥ 0pp | descriptive + Wilson 95% CI |
|
||||
| S5 | oracle-context − full-context | positive (expected) | descriptive | descriptive (SYSTEM_EVOLVED strict-abstain diagnostic) |
|
||||
|
||||
Bench-Spec LOCK v1 multi-comparisons policy: secondary = descriptive, no correction required.
|
||||
|
||||
---
|
||||
|
||||
## 3. Sample design
|
||||
|
||||
_Inherited verbatim from manifest v5 §3 (concurrency=1 retained). No change from v5._
|
||||
|
||||
- **Cells:** five, run in a single invocation. Definitions unchanged.
|
||||
1. `no-context` — true zero-memory baseline.
|
||||
2. `oracle-context` — PM-facing alias for harness `raw`.
|
||||
3. `full-context` — oracle + SYSTEM_EVOLVED strict-abstain.
|
||||
4. `retrieval` — conv-scope HybridSearch top-K=20.
|
||||
5. `agentic` — softened SYSTEM_AGENTIC + bound search_memory + forced-fallback.
|
||||
- **N per cell:** 400.
|
||||
- **Total evaluations:** 2000.
|
||||
- **Instance selection seed:** `42`.
|
||||
- **Matched-pairs design:** same 400 instances flow through all cells.
|
||||
- **Concurrency:** `--parallel-concurrency 1`. Five cells sequential.
|
||||
|
||||
---
|
||||
|
||||
## 4. Dataset
|
||||
|
||||
_Inherited verbatim from manifest v5 §4. No change._
|
||||
|
||||
- **Source:** `benchmarks/data/locomo10.json` (snap-research LoCoMo).
|
||||
- **Raw archive SHA-256:** `79fa87e90f04081343b8c8debecb80a9a6842b76a7aa537dc9fdf651ea698ff4`.
|
||||
- **Canonical build:** `benchmarks/data/locomo/locomo-1540.jsonl` (1531 instances).
|
||||
- **Canonical dataset SHA-256:** `39e415e2f3a0fa1bd3cb1804a58d0b440b50d3070b2100698437e4ec402a5b24`.
|
||||
- **Selection:** 400 per cell via seed-42 shuffle + take-first-400.
|
||||
|
||||
---
|
||||
|
||||
## 5. Model stack
|
||||
|
||||
### 5.1 Subject route table (Qwen 3.6-35B-A3B-Thinking)
|
||||
|
||||
_Inherited verbatim from manifest v5 §5.1. No change._
|
||||
|
||||
| Priority | alias | thinking | max_tokens |
|
||||
|----------|-------|----------|------------|
|
||||
| primary | `qwen3.6-35b-a3b-via-dashscope-direct` | `on` | 16000 |
|
||||
| fallback_1 | `qwen3.6-35b-a3b-via-openrouter` | `on` | 64000 |
|
||||
| fallback_2 | `NOT_AVAILABLE` | — | — |
|
||||
|
||||
Pricing $0.20 / $0.80 per M in/out. `floating_alias` pinning. B3 addendum § 5.
|
||||
|
||||
### 5.2 Judge ensemble — **CHANGED (ensemble swap + backup policy)**
|
||||
|
||||
| Slot | alias (LiteLLM) | role | routing | rate-limit (v6) |
|
||||
|------|------------------|------|---------|-------------------|
|
||||
| primary_judge_1 | `claude-opus-4-7` | primary | anthropic direct | none (Anthropic immutable) |
|
||||
| primary_judge_2 | `gpt-5.4` | primary | openai direct | none |
|
||||
| **primary_judge_3** | **`minimax-m27-via-openrouter`** | **primary** | **openrouter (direct failed per §1.3h)** | **TBD at §1.3c-v6 probe time (OR tier-dependent; default 60 RPM if unspecified)** |
|
||||
| **backup_judge** | **`kimi-k26-direct`** | **backup (per-instance failover)** | **moonshot direct api.moonshot.ai/v1** | **TBD at §1.3c-v6 probe time (Moonshot tier-dependent)** |
|
||||
| ~~tiebreak_reserve~~ | ~~`grok-4.20`~~ | — (RETIRED in v6) | — | — |
|
||||
|
||||
**Backup activation policy (new in v6):**
|
||||
- Primary judges (Opus + GPT + MiniMax) execute majority vote per instance.
|
||||
- If MiniMax primary fails (API error / parse failure / 60s timeout / non-200 HTTP), Kimi K2.6 backup is activated **for that single instance only** (per-instance failover).
|
||||
- If both MiniMax and Kimi fail for a single instance → `judge_ensemble_fail` marker; instance excluded from final analysis per post-hoc exclusion policy §9 (counted as `evaluator_loss` in denominator).
|
||||
- Three-way 1/1/1 split on primary trio → PM escalation (no reserve judge in v6; Grok-4.20 retired from tie-break role).
|
||||
- 2/2 defensive tie → PM escalation (unchanged from v5 policy).
|
||||
|
||||
**Consistency constraint:** One judge call per instance per primary judge; backup called only on primary_judge_3 failure. No prompt-level batching. Identical prompt template per `failure-mode-judge.ts:245-258` verbatim. Temperature=0.0. Matched max_tokens per model (MiniMax/Kimi: 4096 per §1.3h findings; Opus/GPT per v5).
|
||||
|
||||
**κ monitoring (κ re-cal phase, §5.4):** three pairwise Cohen's κ + conservative trio min. Thresholds from Bench-Spec LOCK v1 (pass ≥ 0.65; borderline 0.60-0.65; halt ≤ 0.60) retained. v6 κ re-cal success criterion ≥ 0.70 substantial agreement (tighter than operational halt threshold).
|
||||
|
||||
#### 5.2.1 Failover behavior on MiniMax unavailability (clarification — added 2026-04-24 post-Phase-2 pre-flight, under v6 authority; canonical anchor `60d061e` preserved)
|
||||
|
||||
The pre-registered backup activation ("Kimi K2.6 per-instance failover")
|
||||
is RETRACTED based on §1.3g-h-C Kimi reliability findings (parse rate
|
||||
67-71% on challenging samples, p50 32s latency, p95 exceeds 60s timeout
|
||||
threshold). Kimi retirement from v6 ensemble is a clarification, not
|
||||
substantive methodology change: ensemble membership (Opus+GPT+MiniMax trio),
|
||||
primary hypothesis test, and κ baseline remain unchanged.
|
||||
|
||||
Quorum policy on MiniMax failure (API error, parse fail, timeout >60s
|
||||
after standard 3-retry judge-runner policy):
|
||||
- If Opus and GPT agree → majority verdict = their consensus (2-of-2 quorum)
|
||||
- If Opus and GPT disagree → evaluator_loss marker, instance excluded
|
||||
from primary hypothesis analysis
|
||||
- Expected MiniMax failure rate <1% per Phase 1 empirical evidence
|
||||
(100/100 parse, 0 routing errors)
|
||||
- Expected evaluator_loss rate projected <1% of N=400
|
||||
|
||||
This clarification **supersedes** the pre-retraction "Backup activation policy" paragraph above (Kimi per-instance failover + both-fail `judge_ensemble_fail`). The retained text is kept in place as audit-visible "pre-amendment intent" for tamper-evident trail; operational Phase 2 runs per §5.2.1 quorum policy.
|
||||
|
||||
#### 5.2.2 Kimi alias retention in litellm-config.
|
||||
|
||||
Kimi alias (`kimi-k26-direct`) retained in `litellm-config.yaml` as orphan
|
||||
declaration (not invoked by runner). Removal would require additional
|
||||
config amendment commit; retention preserves audit trail of v6 Phase 1
|
||||
intent and is zero-cost operationally.
|
||||
|
||||
### 5.3 Health-check predicate
|
||||
|
||||
_Inherited from manifest v5 §5.3 (health-check.ts frozen), amended for new aliases._
|
||||
|
||||
Pre-cell health check must verify liveness on all v6 active aliases:
|
||||
- `claude-opus-4-7` via `/v1/chat/completions` ping
|
||||
- `gpt-5.4` via ping
|
||||
- `minimax-m27-via-openrouter` via ping
|
||||
- `kimi-k26-direct` via ping
|
||||
- Subject aliases per v5
|
||||
|
||||
No code change to `health-check.ts` itself (retained as §11 frozen path); new aliases consumed via LiteLLM config lookup.
|
||||
|
||||
### 5.4 κ re-calibration methodology (NEW in v6 — Phase 1 gate)
|
||||
|
||||
Conducted at Phase 1 before any Stage 3 N=400 execution. Gates Phase 2 authorization.
|
||||
|
||||
**Sample:** full 100-instance κ calibration set from v5 at
|
||||
`benchmarks/results/locomo-mini-n20-retry-2026-04-24T00-02-12Z.jsonl`
|
||||
(same authoritative source used for §1.3h split analysis). Identical sample
|
||||
→ new trio κ is directly comparable to v5's original κ=0.7458 three-way
|
||||
baseline.
|
||||
|
||||
**Judge verdicts reused:**
|
||||
- `claude-opus-4-7`: 100 existing verdicts from `judge_ensemble` field. Zero new calls.
|
||||
- `gpt-5.4`: 100 existing verdicts from `judge_ensemble` field. Zero new calls.
|
||||
|
||||
**Judge verdicts new (Phase 1 execution):**
|
||||
- `minimax-m27-via-openrouter`: 100 new calls, verbatim prompt from `failure-mode-judge.ts:245-258`, temperature=0.0, max_tokens=4096.
|
||||
|
||||
**Total new API calls at Phase 1: 100 (MiniMax only).**
|
||||
|
||||
**Computation (three pairwise Cohen's κ):**
|
||||
- κ(Opus, GPT): should match v5's historical baseline (~0.74-0.82 range)
|
||||
- κ(Opus, MiniMax): new measurement
|
||||
- κ(GPT, MiniMax): new measurement
|
||||
|
||||
**Conservative trio κ = min(three pairwise κ values).**
|
||||
|
||||
**Also reported:**
|
||||
- Raw agreement % per pair
|
||||
- Confusion matrix per pair
|
||||
- Per-cell breakdown (no-context / oracle-context / full-context / retrieval / agentic)
|
||||
|
||||
**Success criteria (v6 Phase 1 κ re-cal gate):**
|
||||
- `κ_conservative_trio ≥ 0.70` → **PASS**, halt with `PM-RATIFY-V6-KAPPA` for Phase 2 authorization
|
||||
- `0.60 ≤ κ_conservative_trio < 0.70` → **BORDERLINE**, halt with PM adjudication request
|
||||
- `κ_conservative_trio < 0.60` → **FAIL**, halt with swap-path-re-evaluation request (may require Kimi promoted to primary or backup-judge strategy rework)
|
||||
|
||||
**Operational hedge:** during 100-call execution, log parse rate (target ≥95/100), latency p50 (target ≤25s) + p95, OpenRouter routing errors. If parse rate <90/100, halt before κ compute and raise PM flag.
|
||||
|
||||
---
|
||||
|
||||
## 6. Substrate (conv-scope retrieval)
|
||||
|
||||
_Inherited verbatim from manifest v5 §6. No change._
|
||||
|
||||
- `@waggle/core::HybridSearch` (RRF-fused FTS5 + vec0).
|
||||
- `gopId = conversation_id` scope filter at `search.ts:14`.
|
||||
- Top-K default 20; upper clamp 50.
|
||||
- `createOllamaEmbedder()` + `nomic-embed-text` (1024 dims, local, $0).
|
||||
- Ingest batch 200.
|
||||
|
||||
### 6.1 Agentic-cell tool binding
|
||||
|
||||
_Inherited verbatim from manifest v5 §6.1. No change._
|
||||
|
||||
`makeSearchMemoryTool(substrate, 20, instance.conversation_id)`. `maxTurns=3`. 180 s timeout. Forced-fallback on empty content + tool-use (0 firings at Gate C; load-bearing insurance).
|
||||
|
||||
---
|
||||
|
||||
## 7. SYSTEM_AGENTIC prompt — verbatim bytes locked
|
||||
|
||||
_Inherited verbatim from manifest v5 §7. No change._
|
||||
|
||||
**SHA-256:** `6facae6decc44a6404290514accb4f7cb364081b32d02847a20f8e871633e328` (1467 bytes, no trailing newline). Source: `benchmarks/harness/src/cells.ts` lines 75–102. Softened text from Stage 2-Retry Gate A (commit `373516c`).
|
||||
|
||||
---
|
||||
|
||||
## 8. Stopping rules
|
||||
|
||||
_Inherited verbatim from manifest v5 §8 (v5 §7.4 update under concurrency=1). No change._
|
||||
|
||||
| # | Rule | Source | Trigger | Action |
|
||||
|---|------|--------|---------|--------|
|
||||
| §7.1 | Budget hard halt | `runner.ts` | cumulative spend ≥ **$55.00** (v6 budget) | halt, persist partial, exit ping |
|
||||
| §7.2 | Streak halt | `streak-tracker.ts` | 3 consecutive subject fetch failures | halt, persist partial |
|
||||
| §7.3 | Pre-cell health check fail | `health-check.ts` | 5xx / fetch-error on subject or any judge probe | halt before cell |
|
||||
| §7.4 | Runner lock | `runner-lock.ts` | concurrent cross-process invocation detected | halt (§1.1 waiver unchanged) |
|
||||
| §7.5 | Pre-registration deviation | this document | any change to §1–§9 during run | halt + PM raise |
|
||||
|
||||
Note: v6 budget hard halt at $55 (was $28 in v5) reflects expanded envelope for κ re-cal + N=400 combined. See §14.
|
||||
|
||||
**No interim looks.** Halt only on the five conditions above.
|
||||
|
||||
---
|
||||
|
||||
## 9. Post-hoc exclusion policy: **NONE**
|
||||
|
||||
_Inherited verbatim from manifest v5 §9. No change. `judge_ensemble_fail` (from v6 §5.2 backup-failover failure) counts in denominator as `evaluator_loss`._
|
||||
|
||||
All 2000 evals enter the denominator. `evaluator_loss` (judge-triple failure, including MiniMax+Kimi both-failed failover) counted in denominator, reported separately. No instance whitelist/blacklist. Selective exclusion forbidden ex-ante.
|
||||
|
||||
---
|
||||
|
||||
## 10. Deviation policy
|
||||
|
||||
_Inherited verbatim from manifest v5 §10. No change._
|
||||
|
||||
Any deviation from §1–§9 during run → (1) immediate halt, (2) PM raise, (3) re-pre-registration (manifest v7+) if accepted.
|
||||
|
||||
---
|
||||
|
||||
## 11. Code freeze — **updated via v6 supersession of v5 §11**
|
||||
|
||||
The following code is **frozen at HEAD `373516c`** for the duration of Stage 3 N=400 under v6. v6 emits the single permitted amendment to `litellm-config.yaml` as Phase 1 Commit 2 (under v6 authority — explicit supersession of v5 §11 freeze per PM authorization 2026-04-24).
|
||||
|
||||
**v6 post-amendment state pinned:** `litellm-config.yaml` at Phase 1 Commit 2's tree state. The amendment adds `minimax-m27-via-openrouter` + `kimi-k26-direct` aliases. All v5 entries retained (including the Gemini `gemini-3.1-pro` alias with `rpm:20` — retained as orphan audit artefact; not routed in v6 judge ensemble).
|
||||
|
||||
Frozen paths (inherited from v5 §11, unchanged EXCEPT `litellm-config.yaml`):
|
||||
|
||||
- Cell semantics (`benchmarks/harness/src/cells.ts`).
|
||||
- Substrate (`benchmarks/harness/src/substrate.ts`, `@waggle/core::HybridSearch`, `@waggle/core::FrameStore`, `@waggle/core::SessionStore`).
|
||||
- SYSTEM_AGENTIC + SYSTEM_AGENTIC_FORCED_FALLBACK prompts.
|
||||
- Agent loop (`@waggle/agent::runAgentLoop`, `@waggle/agent::tools.ts`).
|
||||
- Judge ensemble + routing (`benchmarks/harness/src/judge-*.ts`, `benchmarks/harness/src/failure-mode-judge.ts`, `config/models.json`).
|
||||
- Runner + health-check (`benchmarks/harness/src/runner.ts`, `benchmarks/harness/src/health-check.ts`, `benchmarks/harness/src/runner-lock.ts`, `benchmarks/harness/src/streak-tracker.ts`).
|
||||
- Subject route table entries within `config/models.json`.
|
||||
- Test suite.
|
||||
- `litellm-config.yaml` pinned at **v6 Phase 1 Commit 2's tree state** (supersedes v5's pre-amendment pin).
|
||||
|
||||
Execution-only delta during N=400 run: new JSONL files emitted to `benchmarks/results/` (κ re-cal output goes to `benchmarks/calibration/v6-kappa-recal/`). No code file modifications during or after run.
|
||||
|
||||
---
|
||||
|
||||
## 12. Scope boundaries
|
||||
|
||||
_Inherited verbatim from manifest v5 §12. No change._
|
||||
|
||||
### Can claim at Gate D:
|
||||
- Magnitude + significance of conv-scope retrieval memory-lift at Qwen-3.6-35b-a3b under HEAD 373516c.
|
||||
- Per-cell judge-accuracy with Wilson 95% CIs.
|
||||
- Monotonicity chain.
|
||||
- Conv-scope fair-comparison methodology.
|
||||
- Agentic discipline numbers.
|
||||
|
||||
### Cannot claim at Gate D:
|
||||
- Direct comparability to Mem0 91.6% (different scope + memory-synthesis layer).
|
||||
- Multi-model generalization (Qwen-only).
|
||||
- Production performance.
|
||||
|
||||
### Reserved for PM:
|
||||
- Public-claim phrasing + venue.
|
||||
- Matched-scope Mem0 co-run.
|
||||
- Publication timing.
|
||||
|
||||
**CC-1 does NOT compose public SOTA claim.** Scope + data only.
|
||||
|
||||
---
|
||||
|
||||
## 13. PM gates — **Gate P++ + Gate P+++ new; Gate D unchanged**
|
||||
|
||||
### Gate P++ (v6 Phase 1: κ re-cal + config amendment)
|
||||
|
||||
- Trigger: Phase 1 completion = v6 emission commit + `litellm-config.yaml` amendment commit + κ re-cal analysis commit on `feature/c3-v3-wrapper`.
|
||||
- Halt: CC-1 stops; no Phase 2 N=400 kick without PM-RATIFY-V6-KAPPA.
|
||||
- PM checks: v6 content matches brief §1–§5; κ_conservative_trio ≥ 0.70; MiniMax parse + latency + routing operational metrics acceptable.
|
||||
|
||||
### Gate P+++ (v6 Phase 2 kick = post-κ ratification)
|
||||
|
||||
- Trigger: PM-RATIFY-V6-KAPPA received after Phase 1 ratification.
|
||||
- Action: CC-1 kicks N=400 execution via v5's `cli_invocation_template` patched for v6 aliases (`--judge-ensemble claude-opus-4-7,gpt-5.4,minimax-m27-via-openrouter --backup-judge kimi-k26-direct`).
|
||||
|
||||
### Gate D (post-run, pre-SOTA-claim)
|
||||
|
||||
- Trigger: N=400 run exit (clean or halted per §8).
|
||||
- Action: CC-1 writes exit report at `PM-Waggle-OS/sessions/2026-04-24-task25-stage3-n400-complete.md`.
|
||||
- PM decides SOTA claim composition / publish gate / further scope.
|
||||
|
||||
No self-advance at any gate.
|
||||
|
||||
---
|
||||
|
||||
## 14. Budget — **envelope expanded for Phase 1 + Phase 2**
|
||||
|
||||
- **v6 total cap:** $60.00 (v5: $30.00)
|
||||
- **v6 total hard halt:** $55.00 (v5: $28.00)
|
||||
- **v6 expected total burn:** ~$50.00 (v5: ~$23.00)
|
||||
- Phase 1 κ re-cal: ~$25 (100 MiniMax calls via OR @ $0.30 prompt + $1.20 completion per M; ~250K prompt tokens + ~50K completion tokens estimated → well under cap)
|
||||
- Phase 2 N=400: ~$25 (subject + 3 primary judges × 2000 evals; OR MiniMax pricing vs v5's Gemini preview premium delta)
|
||||
|
||||
- **Phase 1 cap:** $30 (brief §7)
|
||||
- **Phase 1 halt:** $35
|
||||
|
||||
- **Phase 2 cap:** $30 (separate envelope; authorized by PM-RATIFY-V6-KAPPA + subsequent brief)
|
||||
|
||||
**Cost breakdown (expected, per phase):**
|
||||
- Subject (Qwen DashScope-intl): ~$2.50 (Phase 2 only)
|
||||
- Judge triple Opus+GPT+MiniMax: ~$22 (Phase 2)
|
||||
- MiniMax κ re-cal: ~$2 (Phase 1)
|
||||
- Kimi backup activations (per-instance failover, expected <5% trigger rate): ~$1 (Phase 2, variable)
|
||||
- Ollama embedding local: $0
|
||||
|
||||
**Wall-clock estimate (Phase 2 N=400 unchanged from v5's 2-3 hour estimate);** Phase 1 κ re-cal ≤90 min per brief §7.
|
||||
|
||||
---
|
||||
|
||||
## 15. Related artefacts
|
||||
|
||||
### v6 ancestry
|
||||
- **Manifest v5 predecessor:** anchor commit `fc16925` (audit-immutable).
|
||||
- **§5.2 rpm:20 edit:** anchor `ad324cc` (v5 §11 exception, retained in v6 config).
|
||||
- **§1.3c throttle probe PASS:** anchor `3a146ef`.
|
||||
- **Fold-in 3.5b sibling mirror:** anchor `d0ab680`.
|
||||
- **§1.3e RPD feasibility:** anchor `1d3851d`.
|
||||
- **§1.3f Vertex Batch INFEASIBLE:** anchor `8ad0567`.
|
||||
- **§1.3g Judge swap MULTI_PASS:** anchor `8a2f0e6`.
|
||||
- **§1.3h Stratified re-probe:** anchor `ae0d312`.
|
||||
- **§1.3h-C DeepSeek mt bump:** anchor `005a19a`.
|
||||
|
||||
### Inherited predecessors (unchanged)
|
||||
- **Manifest v4:** anchor `dedd698` (obsolete).
|
||||
- **§1.1 lock-semantics waiver:** anchor `67eb899`.
|
||||
- **§1.2 runner RCA:** anchor `274e987`.
|
||||
- **§1.3 Gate P+ probe FAIL:** anchor `66dcd5a`.
|
||||
- **§1.3b scope audit:** anchor `69a14708`.
|
||||
- **Bench-Spec LOCK v1 parent:** `PM-Waggle-OS/decisions/2026-04-22-bench-spec-locked.manifest.yaml`.
|
||||
- **Stage 2-Retry Gate C exit:** `PM-Waggle-OS/sessions/2026-04-24-task25-stage2-retry-complete.md`.
|
||||
- **Rollback tag:** `checkpoint/pre-self-evolution-2026-04-14`.
|
||||
|
||||
### v6-specific (this pre-registration)
|
||||
- **v6 Phase 1 Commit 1 (manifest emission):** THIS COMMIT.
|
||||
- **v6 Phase 1 Commit 2 (config amendment):** recorded at Commit 2 time.
|
||||
- **v6 Phase 1 Commit 3 (κ re-cal artefacts):** recorded at Commit 3 time.
|
||||
- **v6 brief:** `PM-Waggle-OS/briefs/2026-04-24-cc1-manifest-v6-phase1-kappa-recal-brief.md`.
|
||||
|
||||
---
|
||||
|
||||
_End of Manifest v6 pre-registration. This document is the anchor for all analysis choices at Stage 3 Gate D exit under the judge-ensemble-swap path. v5 remains audit-immutable predecessor._
|
||||
687
benchmarks/preregistration/manifest-v6-preregistration.yaml
Normal file
687
benchmarks/preregistration/manifest-v6-preregistration.yaml
Normal file
@@ -0,0 +1,687 @@
|
||||
# Manifest v6 — Task 2.5 Stage 3 N=400 Pre-Registration (Judge Ensemble Swap)
|
||||
# Canonical markdown surface: manifest-v6-preregistration.md
|
||||
# Supersedes manifest v5 (anchor fc16925) — v6 governs all Stage 3 re-kick forward.
|
||||
# SHA-256 of this YAML's bytes recorded in v6 anchor commit message.
|
||||
|
||||
manifest_version: v6.0.0-preregistration
|
||||
manifest_type: stage_3_n400_preregistration_v6_ensemble_swap
|
||||
locked_date: 2026-04-24
|
||||
authority: PM (Marko Markovic) — §2.0 + §2.1 Phase 1 authorization 2026-04-24 on closure of full §1.3f-§1.3h-C judge swap validation sequence; inherits §1.1 + §1.2 + §1.3 chain ratifications
|
||||
sprint: 12
|
||||
task: 2.5
|
||||
stage: 3
|
||||
phase: 1
|
||||
branch: feature/c3-v3-wrapper
|
||||
code_freeze_head: 373516c2784807da8536dbc0c194c54f4e4cd4be
|
||||
code_freeze_head_short: 373516c
|
||||
supersedes: manifest_v5_2026_04_24_anchor_fc16925
|
||||
inherits_from: bench_spec_lock_v1_2026_04_22
|
||||
|
||||
# ── v6 Delta Log ────────────────────────────────────────────────────────────
|
||||
|
||||
v6_delta_log:
|
||||
trigger:
|
||||
event: judge_swap_validation_sequence_closure
|
||||
sub_gates:
|
||||
- id: "1.3f"
|
||||
anchor_commit: 8ad056736f36b4ae
|
||||
verdict: INFEASIBLE
|
||||
observation: "Vertex AI v1beta batch-prediction catalog does not list gemini-3.1-pro-preview; Branch A closed"
|
||||
- id: "1.3g"
|
||||
anchor_commit: 8a2f0e61621f9684
|
||||
verdict: MULTI_PASS_with_methodological_caveat
|
||||
observation: "kappa=1.0 across 4 Chinese flagship candidates on 20 unanimous-biased instances (0/20 splits vs full-set 7% split rate); operational ranking (Zhipu > DeepSeek > MiniMax > Kimi) heuristic only"
|
||||
- id: "1.3h"
|
||||
anchor_commit: ae0d312b4088693e
|
||||
verdict: INCONCLUSIVE_BUT_OPERATIONAL_SIGNAL
|
||||
observation: "7-split stratified re-probe; split-only kappa structurally degenerate (all 7 splits Opus=correct/GPT=incorrect); correctness on oriented splits: MiniMax 86%, Kimi 80%, DeepSeek 40%, Zhipu 0% (GPT-echo)"
|
||||
- id: "1.3h-C"
|
||||
anchor_commit: 005a19a8c6c4146c
|
||||
verdict: truncation_fixable_but_correctness_regressed
|
||||
observation: "DeepSeek max_tokens 1024->2048: parse 5/7 -> 7/7; correctness 40% -> 14% (GPT-alignment escalates with reasoning depth); DeepSeek DQ"
|
||||
root_cause_recap: "Google product policy 25 RPM per-model preview cap + Vertex batch ineligibility for preview models blocks Branch A; empirical judge swap validation sequence produces Chinese flagship roster"
|
||||
|
||||
final_ensemble_selection:
|
||||
primary_judge_1:
|
||||
model: claude-opus-4-7
|
||||
rationale: "inherited from v5 (unchanged)"
|
||||
primary_judge_2:
|
||||
model: gpt-5.4
|
||||
rationale: "inherited from v5 (unchanged)"
|
||||
primary_judge_3:
|
||||
model: minimax-m27-via-openrouter
|
||||
upstream_identifier: openrouter/minimax/minimax-m2.7
|
||||
rationale: "86% correctness on oriented splits (best empirical fit); 100% parse via OR; direct routing failed both api.minimaxi.com + api.minimax.chat v2 despite MINIMAX_GROUP_ID addition (§1.3h)"
|
||||
selection_anchor: ae0d312b4088693e
|
||||
backup_judge:
|
||||
model: kimi-k26-direct
|
||||
upstream_identifier: moonshot/kimi-k2.6 (via api.moonshot.ai/v1 OpenAI-compatible endpoint)
|
||||
rationale: "80% correctness on oriented splits; per-instance failover on primary_judge_3 failure"
|
||||
selection_anchor: ae0d312b4088693e
|
||||
activation_policy: per_instance_failover_on_primary_judge_3_failure
|
||||
|
||||
disqualified_candidates:
|
||||
- model: gemini-3.1-pro-preview
|
||||
dq_reason: "Google per-model 25 RPM preview cap + Vertex batch INFEASIBLE"
|
||||
evidence_anchors:
|
||||
- 66dcd5a1b18b9367
|
||||
- 1d3851d...
|
||||
- 8ad056736f36b4ae
|
||||
- model: zhipu_glm-5.1
|
||||
dq_reason: "100% GPT-echo on oriented splits (p_opus=0%, p_gpt=100%); violates ensemble independence assumption"
|
||||
evidence_anchor: ae0d312b4088693e
|
||||
- model: deepseek_v4-pro
|
||||
dq_reason: "14% correctness on oriented splits at max_tokens=2048 (regressed from 40% at max_tokens=1024); GPT-alignment escalates with reasoning depth"
|
||||
evidence_anchor: 005a19a8c6c4146c
|
||||
- model: grok-4.20
|
||||
dq_reason: "retired from tie-break reserve role in v6; no operational issue, but v6 backup-failover policy replaces reserve-judge mechanism"
|
||||
evidence: "v6 §5.2 policy change (not empirical DQ)"
|
||||
|
||||
changes_from_v5:
|
||||
judge_ensemble_section_5_2:
|
||||
v5: "Opus + GPT + Gemini-3.1-Pro-Preview primary; Grok-4.20 reserve (1/1/1 only)"
|
||||
v6: "Opus + GPT + MiniMax-M2.7 primary; Kimi-K2.6 backup (per-instance failover); Grok retired"
|
||||
tiebreak_policy:
|
||||
v5: "majority_with_grok_reserve_on_1_1_1_split"
|
||||
v6: "primary_3_judge_majority_with_kimi_per_instance_failover_on_minimax_failure; three-way 1/1/1 -> PM escalation (no reserve judge)"
|
||||
rate_limit_metadata:
|
||||
v5: "rpm:20 on gemini-3.1-pro-preview (v5 §5.2 addendum)"
|
||||
v6: "no active rpm:20 in judge path (Gemini alias retained in litellm-config as orphan audit artefact, not routed in v6); MiniMax OR rpm + Kimi Moonshot rpm TBD at §1.3c-v6 probe time"
|
||||
code_freeze_section_11:
|
||||
v5: "HEAD 373516c; single permitted pre-run modification = litellm-config.yaml Gemini rpm:20 addendum"
|
||||
v6: "v5 §11 freeze on litellm-config.yaml superseded by v6 authority; Phase 1 Commit 2 amendment adds minimax-m27-via-openrouter + kimi-k26-direct aliases + retains all v5 entries; v6 §11 pins post-amendment state"
|
||||
budget_envelope:
|
||||
v5: "$30 cap / $28 halt / ~$23 expected"
|
||||
v6: "$60 cap / $55 halt / ~$50 expected (Phase 1 kappa re-cal ~$25 + Phase 2 N=400 ~$25; no preview-model premium)"
|
||||
pm_gates:
|
||||
v5: "Gate P+ (pre-run) + Gate D (post-run)"
|
||||
v6: "Gate P++ (Phase 1 kappa re-cal + config amendment) + Gate P+++ (Phase 2 kick on PM-RATIFY-V6-KAPPA) + Gate D (post-run unchanged)"
|
||||
|
||||
unchanged_from_v5:
|
||||
sections:
|
||||
- "§1 primary hypothesis (Fisher one-sided p<0.10 on retrieval - no-context >= 5pp)"
|
||||
- "§2 secondary endpoints S1-S5"
|
||||
- "§3 sample design (concurrency=1, 5 cells sequential, N=400, seed=42)"
|
||||
- "§4 dataset (LoCoMo 1531 instances, raw SHA 79fa87e9..., canonical SHA 39e415e2...)"
|
||||
- "§5.1 subject route table (Qwen 3.6-35B-A3B-Thinking, DashScope-intl direct primary)"
|
||||
- "§5.3 health-check predicate"
|
||||
- "§6 substrate (HybridSearch conv-scope top-K=20, nomic-embed-text)"
|
||||
- "§7 SYSTEM_AGENTIC verbatim bytes (SHA-256 6facae6d..., 1467 bytes)"
|
||||
- "§8 stopping rules (budget + streak + pre-cell health + deviation)"
|
||||
- "§9 post-hoc exclusion policy NONE"
|
||||
- "§10 deviation policy (halt + restart-required)"
|
||||
- "§12 scope boundaries + SOTA composition reserved for PM"
|
||||
- "Gate D structure (post-run, pre-SOTA-claim)"
|
||||
|
||||
parent_chain:
|
||||
v4: dedd69888e008fb158
|
||||
section_1_1_lock_waiver: 67eb89914a49ec38
|
||||
section_1_2_rca: 274e9871b54599077a3d
|
||||
section_1_3_probe_fail: 66dcd5a1b18b9367
|
||||
section_1_3b_scope_audit: 69a14708f78a74d2
|
||||
v5_emission: fc169250c3c27cd3
|
||||
section_5_2_rpm20_edit: ad324ccf...
|
||||
section_1_3c_throttle_probe_pass: 3a146efc...
|
||||
fold_in_3_5b_sibling_mirror: d0ab680...
|
||||
section_1_3e_rpd_feasibility: 1d3851d...
|
||||
section_1_3f_vertex_batch: 8ad056736f36b4ae
|
||||
section_1_3g_judge_swap_multi_pass: 8a2f0e61621f9684
|
||||
section_1_3h_stratified_reprobe: ae0d312b4088693e
|
||||
section_1_3h_c_deepseek_mt_bump: 005a19a8c6c4146c
|
||||
|
||||
# ── Manifest path + lock timestamp ──────────────────────────────────────────
|
||||
|
||||
manifest_path: benchmarks/preregistration/manifest-v6-preregistration.yaml
|
||||
manifest_locked_at: 2026-04-24T00:00:00Z
|
||||
|
||||
# ── Dataset (unchanged from v5) ─────────────────────────────────────────────
|
||||
|
||||
dataset:
|
||||
name: locomo
|
||||
source_url: https://raw.githubusercontent.com/snap-research/locomo/main/data/locomo10.json
|
||||
raw_archive_path: benchmarks/data/locomo10.json
|
||||
raw_archive_sha256: 79fa87e90f04081343b8c8debecb80a9a6842b76a7aa537dc9fdf651ea698ff4
|
||||
raw_archive_bytes: 2805274
|
||||
canonical_path: benchmarks/data/locomo/locomo-1540.jsonl
|
||||
canonical_sha256: 39e415e2f3a0fa1bd3cb1804a58d0b440b50d3070b2100698437e4ec402a5b24
|
||||
canonical_instance_count: 1531
|
||||
paper_total_claim: 1540
|
||||
paper_reference: "Maharana et al., ACL-2024 — Evaluating Very Long-Term Conversational Memory of LLM Agents"
|
||||
category_distribution:
|
||||
single_hop: 841
|
||||
multi_hop: 281
|
||||
temporal: 320
|
||||
open_ended: 89
|
||||
|
||||
# ── Primary hypothesis (unchanged from v5) ──────────────────────────────────
|
||||
|
||||
primary_hypothesis:
|
||||
name: memory_lift_retrieval_vs_no_context
|
||||
direction: one_sided_positive
|
||||
statement: "retrieval_judge_accuracy - no-context_judge_accuracy >= 5pp"
|
||||
test: fisher_exact_one_sided
|
||||
alpha_threshold: 0.10
|
||||
effect_size_threshold_pp: 5
|
||||
justification_ex_ante:
|
||||
- gate_b_dry_run_conv_scope_20_of_20_vs_whole_corpus_8_of_20_leak_2026_04_24
|
||||
- gate_c_monotonicity_no_context_0_10_lt_retrieval_0_35_lt_agentic_0_40_lt_oracle_0_55
|
||||
|
||||
# ── Secondary endpoints (unchanged from v5) ─────────────────────────────────
|
||||
|
||||
secondary_endpoints:
|
||||
S1_monotonicity_no_context_leq_retrieval:
|
||||
direction: one_sided_positive
|
||||
threshold_pp: 0
|
||||
test: fisher_exact_one_sided
|
||||
alpha: 0.20
|
||||
S2_monotonicity_retrieval_leq_agentic:
|
||||
direction: one_sided_positive
|
||||
threshold_pp: 0
|
||||
test: fisher_exact_one_sided
|
||||
alpha: 0.20
|
||||
S3_monotonicity_agentic_leq_oracle_context:
|
||||
direction: one_sided_positive
|
||||
threshold_pp: 0
|
||||
test: fisher_exact_one_sided
|
||||
alpha: 0.20
|
||||
S4_agentic_lift_over_retrieval:
|
||||
direction: descriptive
|
||||
threshold_pp: 0
|
||||
report: [point_estimate, wilson_95_ci]
|
||||
S5_abstain_penalty_oracle_minus_full_context:
|
||||
direction: descriptive_expected_positive
|
||||
report: [point_estimate]
|
||||
|
||||
# ── Sample design (unchanged from v5: concurrency=1) ────────────────────────
|
||||
|
||||
sample:
|
||||
cells:
|
||||
- no-context
|
||||
- oracle-context
|
||||
- full-context
|
||||
- retrieval
|
||||
- agentic
|
||||
n_per_cell: 400
|
||||
total_evaluations: 2000
|
||||
instance_selection_seed: 42
|
||||
instance_selection_method: "shuffle-then-take-first-N, deterministic given seed"
|
||||
matched_pairs: true
|
||||
concurrency: 1
|
||||
concurrency_rationale: "inherited from v5 §3 (§1.3 Gate P+ empirical finding on 25 RPM Gemini preview cap); concurrency=1 retained in v6 even though Gemini retired, for stable empirical comparability with v5 planning"
|
||||
|
||||
# ── Cells semantics (unchanged from v5, frozen at HEAD 373516c) ─────────────
|
||||
|
||||
cells_semantics:
|
||||
no_context:
|
||||
system_prompt: SYSTEM_BASELINE
|
||||
user_prompt: "Question: {question}"
|
||||
memory_injection: none
|
||||
added_at: stage_2_retry_1_1_2026_04_24
|
||||
oracle_context:
|
||||
system_prompt: SYSTEM_BASELINE
|
||||
user_prompt: "Context: {instance.context}\\n\\nQuestion: {instance.question}"
|
||||
memory_injection: oracle_fed_by_locomo
|
||||
harness_alias: raw
|
||||
full_context:
|
||||
system_prompt: SYSTEM_EVOLVED
|
||||
memory_injection: oracle_fed_plus_evolved_abstain
|
||||
retrieval:
|
||||
system_prompt: SYSTEM_BASELINE
|
||||
substrate: waggle_core_hybrid_search
|
||||
scope: conversation_scoped_via_gopId
|
||||
top_k_default: 20
|
||||
top_k_upper_clamp: 50
|
||||
agentic:
|
||||
system_prompt: SYSTEM_AGENTIC_softened_stage2_retry
|
||||
system_prompt_sha256: 6facae6decc44a6404290514accb4f7cb364081b32d02847a20f8e871633e328
|
||||
system_prompt_bytes: 1467
|
||||
tool_allowlist:
|
||||
- search_memory
|
||||
tool_binding: "search_memory bound to instance.conversation_id; non-overridable"
|
||||
max_turns: 3
|
||||
timeout_ms: 180000
|
||||
forced_answer_fallback:
|
||||
enabled: true
|
||||
system_prompt: SYSTEM_AGENTIC_FORCED_FALLBACK
|
||||
gate_c_firing_rate: 0
|
||||
|
||||
# ── Model stack (CHANGED: §5.2 ensemble swap + backup policy) ───────────────
|
||||
|
||||
subject_model: qwen3.6-35b-a3b-via-dashscope-direct
|
||||
subject_fallback_1: qwen3.6-35b-a3b-via-openrouter
|
||||
subject_fallback_2: NOT_AVAILABLE
|
||||
|
||||
subject_route_table:
|
||||
primary:
|
||||
alias: qwen3.6-35b-a3b-via-dashscope-direct
|
||||
litellm_model: qwen3.6-35b-a3b-via-dashscope-direct
|
||||
upstream_route: "LiteLLM local alias -> openai/qwen3.6-35b-a3b @ https://dashscope-intl.aliyuncs.com/compatible-mode/v1"
|
||||
provider: alibaba
|
||||
thinking: on
|
||||
max_tokens: 16000
|
||||
price_per_million_input_usd: 0.20
|
||||
price_per_million_output_usd: 0.80
|
||||
context_window: 262144
|
||||
pinning_surface: floating_alias
|
||||
fallback_1:
|
||||
alias: qwen3.6-35b-a3b-via-openrouter
|
||||
litellm_model: qwen3.6-35b-a3b-via-openrouter
|
||||
upstream_route: "LiteLLM -> OpenRouter bridge (openrouter/qwen/qwen3.5-35b-a3b)"
|
||||
thinking: on
|
||||
max_tokens: 64000
|
||||
pinning_surface: floating_alias
|
||||
trigger_condition: fetch_error_on_primary
|
||||
fallback_2:
|
||||
alias: NOT_AVAILABLE
|
||||
|
||||
judge_ensemble:
|
||||
primary:
|
||||
- judge_role: primary
|
||||
slot: primary_judge_1
|
||||
model_id: claude-opus-4-7
|
||||
provider: anthropic
|
||||
litellm_model: claude-opus-4-7
|
||||
pinning_surface: anthropic_immutable
|
||||
rate_limit_v6: null
|
||||
price_per_million_input_usd: 15.00
|
||||
price_per_million_output_usd: 75.00
|
||||
unchanged_from_v5: true
|
||||
- judge_role: primary
|
||||
slot: primary_judge_2
|
||||
model_id: gpt-5.4
|
||||
provider: openai_via_openrouter
|
||||
litellm_model: gpt-5.4
|
||||
pinning_surface: floating_alias
|
||||
rate_limit_v6: null
|
||||
price_per_million_input_usd: 10.00
|
||||
price_per_million_output_usd: 30.00
|
||||
unchanged_from_v5: true
|
||||
- judge_role: primary
|
||||
slot: primary_judge_3_v6_swap
|
||||
model_id: minimax-m2.7
|
||||
provider: openrouter_bridge
|
||||
litellm_model: minimax-m27-via-openrouter
|
||||
upstream_identifier: openrouter/minimax/minimax-m2.7
|
||||
pinning_surface: floating_alias
|
||||
rate_limit_v6:
|
||||
rpm: null
|
||||
rationale: "TBD at §1.3c-v6 probe time (OR tier-dependent; default 60 RPM if unspecified by OR)"
|
||||
delivery_mechanism: "LiteLLM litellm_params.rpm on the v6 alias entry in litellm-config.yaml (Phase 1 Commit 2)"
|
||||
price_per_million_input_usd: 0.30
|
||||
price_per_million_output_usd: 1.20
|
||||
context_window: 196608
|
||||
v6_selection_rationale: "86% correctness on oriented splits (§1.3h); 100% parse via OR; direct routing failed despite MINIMAX_GROUP_ID addition"
|
||||
v6_selection_anchor: ae0d312b4088693e
|
||||
unchanged_from_v5: false
|
||||
backup:
|
||||
judge_role: backup
|
||||
slot: backup_judge_v6
|
||||
model_id: kimi-k2.6
|
||||
provider: moonshot_direct
|
||||
litellm_model: kimi-k26-direct
|
||||
upstream_identifier: "moonshot/kimi-k2.6 via api.moonshot.ai/v1 OpenAI-compatible endpoint"
|
||||
pinning_surface: floating_alias
|
||||
rate_limit_v6:
|
||||
rpm: null
|
||||
rationale: "TBD at §1.3c-v6 probe time (Moonshot tier-dependent)"
|
||||
price_per_million_input_usd: 0.60
|
||||
price_per_million_output_usd: 2.40
|
||||
activation_policy: per_instance_failover_on_primary_judge_3_failure
|
||||
activation_triggers:
|
||||
- api_error_non_200_status
|
||||
- parse_failure_verdict_none
|
||||
- timeout_60s
|
||||
both_fail_behavior: "judge_ensemble_fail marker; instance counted as evaluator_loss in denominator per §9"
|
||||
v6_selection_rationale: "80% correctness on oriented splits (§1.3h); direct Moonshot route stable; per-instance failover rather than reserve tie-break judge"
|
||||
v6_selection_anchor: ae0d312b4088693e
|
||||
retired_in_v6:
|
||||
model_id: grok-4.20
|
||||
v5_role: tiebreak_reserve_on_1_1_1
|
||||
v6_status: retired
|
||||
v6_replacement_policy: "three-way 1/1/1 split on primary trio -> PM escalation (no reserve judge in v6; backup-failover policy replaces reserve mechanism — further superseded by §5.2.1 2-of-2 quorum clarification 2026-04-24)"
|
||||
section_5_2_1_clarification_2026_04_24:
|
||||
scope: "v6 §5.2 amendment under v6 authority (canonical anchor 60d061e preserved); NOT v7 re-pre-registration; NOT §10 deviation"
|
||||
pm_adjudication_anchor: pm_adjudicate_v6_phase2_blockers_option_b_accept
|
||||
change_summary: "retract Kimi backup; adopt 2-of-2 quorum on MiniMax failure; evaluator_loss on Opus/GPT split"
|
||||
rationale:
|
||||
- "MiniMax Phase 1 empirical reliability: 100/100 parse + 0 routing errors (expected <1% N=400 failure rate)"
|
||||
- "Kimi cold probe (Phase 2 pre-flight fa7464b): 2/3 parse (67%); §1.3g-h-C consistent 67-71% on challenging samples + p95 >60s timeout"
|
||||
- "Kimi-as-backup = insurance that fails when needed"
|
||||
backup_judge_retraction:
|
||||
retracted_policy: per_instance_failover_on_primary_judge_3_failure
|
||||
retracted_model: kimi-k26-direct
|
||||
retracted_model_status_in_litellm_config: retained_as_orphan_not_invoked_by_runner
|
||||
new_quorum_policy_on_minimax_failure:
|
||||
opus_gpt_agree: majority_verdict_equals_their_consensus_2_of_2_quorum
|
||||
opus_gpt_disagree: evaluator_loss_reason_minimax_failed_opus_gpt_split_exclude_from_h1
|
||||
expected_failure_rate_n400: lt_0_01
|
||||
expected_evaluator_loss_rate_n400: lt_0_01
|
||||
expected_phase2_execution_semantics:
|
||||
primary_judges_parallel: [opus, gpt, minimax]
|
||||
minimax_failure_triggers: [api_error, parse_fail, timeout_gt_60s_after_3_retry]
|
||||
minimax_failure_fallback: 2_of_2_opus_gpt_quorum
|
||||
kimi_involvement: none_retired
|
||||
evaluator_loss_only_when: opus_gpt_disagree_and_minimax_failed
|
||||
audit_chain:
|
||||
parent_commit: fa7464b
|
||||
canonical_v6_anchor: 60d061e
|
||||
phase_2_kick_gate: PM_RATIFY_V6_5_2_CLARIFICATION
|
||||
consistency_constraint:
|
||||
same_physical_judge_models_subset_as_v5_plus_minimax: true
|
||||
single_call_per_judge_per_instance: true
|
||||
no_prompt_batching: true
|
||||
identical_prompt_template: "failure-mode-judge.ts:245-258 verbatim"
|
||||
temperature: 0.0
|
||||
max_tokens_per_judge:
|
||||
claude_opus_4_7: 1024
|
||||
gpt_5_4: 1024
|
||||
minimax_m27: 4096
|
||||
kimi_k26: 4096
|
||||
vote_policy:
|
||||
primary_trio_majority: true
|
||||
backup_activates_on_primary_judge_3_failure: true
|
||||
one_one_one_split: pm_escalation_no_reserve
|
||||
two_two_defensive_tie: pm_escalation
|
||||
|
||||
judge_primary:
|
||||
id: claude-opus-4-7
|
||||
judge_secondary:
|
||||
id: gpt-5.4
|
||||
judge_tertiary_v6:
|
||||
id: minimax-m27-via-openrouter
|
||||
judge_backup_v6:
|
||||
id: kimi-k26-direct
|
||||
|
||||
# ── κ re-calibration methodology (NEW in v6 §5.4) ───────────────────────────
|
||||
|
||||
kappa_recalibration_v6:
|
||||
phase: 1
|
||||
required_before_phase_2_n400: true
|
||||
gate_id: gate_p_plus_plus
|
||||
sample:
|
||||
source: benchmarks/results/locomo-mini-n20-retry-2026-04-24T00-02-12Z.jsonl
|
||||
n_instances: 100
|
||||
rationale: "same authoritative source as v5 kappa baseline + §1.3h split analysis; new trio kappa directly comparable to v5 historical kappa=0.7458"
|
||||
verdicts_reused:
|
||||
claude_opus_4_7: "100 from judge_ensemble field (zero new calls)"
|
||||
gpt_5_4: "100 from judge_ensemble field (zero new calls)"
|
||||
verdicts_new:
|
||||
minimax_m27_via_openrouter: "100 new calls, verbatim prompt failure-mode-judge.ts:245-258, temperature=0.0, max_tokens=4096"
|
||||
total_new_api_calls_phase_1: 100
|
||||
computation:
|
||||
pairwise_cohen_kappa:
|
||||
- "kappa(opus, gpt) — should match v5 baseline ~0.74-0.82"
|
||||
- "kappa(opus, minimax) — new"
|
||||
- "kappa(gpt, minimax) — new"
|
||||
conservative_trio_kappa: "min of three pairwise"
|
||||
also_reported:
|
||||
- raw_agreement_pct_per_pair
|
||||
- confusion_matrix_per_pair
|
||||
- per_cell_breakdown
|
||||
success_criteria:
|
||||
pass_trio_kappa_gte: 0.70
|
||||
borderline_trio_kappa_range: [0.60, 0.70]
|
||||
fail_trio_kappa_lt: 0.60
|
||||
pass_action: halt_with_PM_RATIFY_V6_KAPPA_request_for_phase_2_authorization
|
||||
borderline_action: halt_with_PM_adjudication_request
|
||||
fail_action: halt_with_swap_path_re_evaluation_request
|
||||
operational_hedge:
|
||||
parse_rate_target_per_100: 95
|
||||
parse_rate_halt_threshold_per_100: 90
|
||||
latency_p50_target_seconds: 25
|
||||
openrouter_routing_error_rate_flag_threshold: 0.05
|
||||
|
||||
# ── Substrate (unchanged from v5) ───────────────────────────────────────────
|
||||
|
||||
substrate:
|
||||
implementation: "@waggle/core::HybridSearch (RRF-fused FTS5 + vec0)"
|
||||
scope_filter:
|
||||
parameter: gopId
|
||||
source_location: packages/core/src/mind/search.ts:14
|
||||
field_name: SearchOptions.gopId
|
||||
benchmark_binding: instance.conversation_id
|
||||
top_k_default: 20
|
||||
top_k_upper_clamp: 50
|
||||
embedder:
|
||||
factory: createOllamaEmbedder
|
||||
base_url: http://localhost:11434
|
||||
model: nomic-embed-text
|
||||
dims: 1024
|
||||
cost: zero_local_inference
|
||||
ingest_batch_size: 200
|
||||
|
||||
# ── κ monitoring runtime (during Phase 2 N=400) ─────────────────────────────
|
||||
|
||||
kappa_monitoring_runtime:
|
||||
baseline_reference: sprint_10_task_2_2_kappa_0_7458_plus_v6_phase_1_kappa_value
|
||||
compute: fleiss_kappa_on_pre_tiebreak_vote_matrix
|
||||
thresholds:
|
||||
pass_no_flag_kappa_min: 0.65
|
||||
pass_with_flag_kappa_range: [0.60, 0.65]
|
||||
halt_kappa_max: 0.60
|
||||
halt_drop_from_baseline_max_pp: 10
|
||||
|
||||
# ── Confidence intervals + failure taxonomy (unchanged from v5) ─────────────
|
||||
|
||||
confidence_intervals:
|
||||
primary:
|
||||
method: wilson_score_95
|
||||
secondary:
|
||||
method: cluster_bootstrap_95
|
||||
iterations: 10000
|
||||
seed: 42
|
||||
cluster_unit: conversation_id
|
||||
|
||||
failure_taxonomy:
|
||||
version: v1
|
||||
categories:
|
||||
- {code: F1, name: contradicts_ground_truth}
|
||||
- {code: F2, name: partial_answer}
|
||||
- {code: F3, name: off_topic}
|
||||
- {code: F4, name: refusal}
|
||||
- {code: F5, name: tool_use_error}
|
||||
- {code: F6, name: format_violation}
|
||||
|
||||
# ── Stopping rules (§7.1 budget updated to v6 envelope) ─────────────────────
|
||||
|
||||
stopping_rules:
|
||||
budget_hard_halt_usd: 55.00
|
||||
budget_cap_usd: 60.00
|
||||
streak_halt: "3 consecutive subject fetch failures -> halt (streak-tracker.ts)"
|
||||
pre_cell_health_check: "GET /health/liveliness + POST /v1/chat/completions ping per model including v6 new aliases -> halt on any 5xx/fetch-error"
|
||||
runner_lock: "concurrent_runners: SEQUENTIAL (parallel-concurrency=1); §1.1 cross-process waiver still governs"
|
||||
deviation_from_preregistration: "any change to §1-§9 during run -> immediate halt + PM raise"
|
||||
no_interim_looks: true
|
||||
mid_run_amendment_policy: halt_restart_required
|
||||
|
||||
# ── Post-hoc exclusion: NONE (unchanged from v5; judge_ensemble_fail in denominator) ──
|
||||
|
||||
post_hoc_exclusion:
|
||||
policy: none
|
||||
evaluator_loss_handling:
|
||||
included_in_denominator: true
|
||||
reported_separately: true
|
||||
denominator_formula: "correct + incorrect + evaluator_loss"
|
||||
evaluator_loss_sources:
|
||||
- "judge parse failure on all active primaries after retries"
|
||||
- "v6 NEW: judge_ensemble_fail when primary_judge_3 (MiniMax) and backup_judge (Kimi) both fail on same instance"
|
||||
|
||||
# ── Budget (v6 expanded envelope) ───────────────────────────────────────────
|
||||
|
||||
budget:
|
||||
v6_total_cap_usd: 60.00
|
||||
v6_total_hard_halt_usd: 55.00
|
||||
v6_expected_total_burn_usd: 50.00
|
||||
variance_ceiling_usd: 55.00
|
||||
phase_1_cap_usd: 30.00
|
||||
phase_1_halt_usd: 35.00
|
||||
phase_2_cap_usd: 30.00
|
||||
breakdown_expected:
|
||||
phase_1_minimax_kappa_recal_usd: 2.50
|
||||
phase_2_subject_qwen_dashscope_direct_usd: 2.50
|
||||
phase_2_judge_triple_opus_gpt5_minimax_usd: 22.00
|
||||
phase_2_kimi_backup_activations_variable_usd: 1.00
|
||||
embedding_ollama_local_usd: 0.00
|
||||
wall_clock_estimate:
|
||||
phase_1_kappa_recal_minutes: 90
|
||||
phase_2_n400_hours_min: 2
|
||||
phase_2_n400_hours_max: 3
|
||||
|
||||
# ── Target sample + CLI invocation (v6-specific) ────────────────────────────
|
||||
|
||||
target_N: 400
|
||||
target_cells:
|
||||
- no-context
|
||||
- oracle-context
|
||||
- full-context
|
||||
- retrieval
|
||||
- agentic
|
||||
target_total_evaluations: 2000
|
||||
target_concurrency: 1
|
||||
|
||||
cli_invocation_template: >
|
||||
npx tsx scripts/run-mini-locomo.ts
|
||||
--manifest benchmarks/preregistration/manifest-v6-preregistration.yaml
|
||||
--subject qwen3.6-35b-a3b-via-dashscope-direct
|
||||
--subject-fallback-1 qwen3.6-35b-a3b-via-openrouter
|
||||
--judge-ensemble claude-opus-4-7,gpt-5.4,minimax-m27-via-openrouter
|
||||
--backup-judge kimi-k26-direct
|
||||
--v3-cells --N 400 --parallel-concurrency 1 --seed 42
|
||||
|
||||
# ── Code freeze — v6 supersession of v5 §11 ─────────────────────────────────
|
||||
|
||||
code_freeze:
|
||||
head: 373516c2784807da8536dbc0c194c54f4e4cd4be
|
||||
branch: feature/c3-v3-wrapper
|
||||
v5_section_11_superseded_by: v6_phase_1_commit_2_under_pm_authorization
|
||||
frozen_paths:
|
||||
- benchmarks/harness/src/cells.ts
|
||||
- benchmarks/harness/src/substrate.ts
|
||||
- benchmarks/harness/src/judge-client.ts
|
||||
- benchmarks/harness/src/judge-runner.ts
|
||||
- benchmarks/harness/src/failure-mode-judge.ts
|
||||
- benchmarks/harness/src/health-check.ts
|
||||
- benchmarks/harness/src/streak-tracker.ts
|
||||
- benchmarks/harness/src/runner-lock.ts
|
||||
- benchmarks/harness/src/runner.ts
|
||||
- benchmarks/harness/config/models.json
|
||||
- packages/agent/src/agent-loop.ts
|
||||
- packages/agent/src/tools.ts
|
||||
- packages/core/src/mind/search.ts
|
||||
- packages/core/src/mind/frames.ts
|
||||
- packages/core/src/mind/sessions.ts
|
||||
- packages/core/src/mind/db.ts
|
||||
permitted_delta_during_run:
|
||||
- "new JSONL files emitted to benchmarks/results/ by the N=400 run"
|
||||
- "new artefacts under benchmarks/calibration/v6-kappa-recal/ at Phase 1"
|
||||
permitted_pre_run_delta_v6_single_authorized_amendment:
|
||||
path: litellm-config.yaml
|
||||
scope: "add minimax-m27-via-openrouter + kimi-k26-direct aliases; retain all v5 entries (Gemini with rpm:20 kept as orphan audit artefact); no modifications to other aliases"
|
||||
committed_separately_before_kappa_recal: true
|
||||
justification: "manifest v6 §5.2 ensemble swap + §11 supersession under PM authorization 2026-04-24 post-§1.3h-C closure"
|
||||
v6_phase_1_commit_2_pins_post_amendment_state: true
|
||||
|
||||
# ── Deviation policy (unchanged from v5) ────────────────────────────────────
|
||||
|
||||
deviation_policy:
|
||||
on_detection:
|
||||
- immediate_halt
|
||||
- pm_raise
|
||||
- re_preregister_new_manifest_v7_if_accepted
|
||||
|
||||
# ── PM gates (v6: Gate P++ + Gate P+++ new; Gate D unchanged) ──────────────
|
||||
|
||||
pm_gates:
|
||||
gate_p_plus_plus_v6_phase_1:
|
||||
trigger: "v6 emission commit + litellm-config.yaml amendment commit + kappa re-cal analysis commit on feature/c3-v3-wrapper"
|
||||
pre_kick_checks_required:
|
||||
- v6_anchor_commit_sha_recorded
|
||||
- v6_md_sha256_recorded_in_commit_message
|
||||
- v6_yaml_sha256_recorded_in_commit_message
|
||||
- litellm_config_amendment_committed_separately
|
||||
- kappa_conservative_trio_gte_0_70
|
||||
- minimax_parse_rate_gte_95_per_100
|
||||
- minimax_routing_error_rate_lt_0_05
|
||||
action: "CC-1 halts; awaits PM-RATIFY-V6-KAPPA for Phase 2 authorization"
|
||||
success_verdict_bands:
|
||||
PASS: kappa_trio_gte_0_70
|
||||
BORDERLINE: kappa_trio_0_60_to_0_70
|
||||
FAIL: kappa_trio_lt_0_60
|
||||
gate_p_plus_plus_plus_v6_phase_2_kick:
|
||||
trigger: "PM-RATIFY-V6-KAPPA received"
|
||||
action: "CC-1 kicks N=400 execution via v6 CLI invocation template"
|
||||
prerequisites:
|
||||
- pm_ratify_v6_kappa_received
|
||||
gate_d_post_run:
|
||||
trigger: "N=400 run exit (clean or halted per stopping_rules)"
|
||||
action: "CC-1 writes Gate D exit report at PM-Waggle-OS/sessions/2026-04-24-task25-stage3-n400-complete.md; halts"
|
||||
outcomes:
|
||||
- compose_sota_claim_authority_pm
|
||||
- publish_gate
|
||||
- further_scope
|
||||
cc1_self_advance: forbidden_at_all_gates
|
||||
|
||||
# ── Scope boundaries (unchanged from v5) ────────────────────────────────────
|
||||
|
||||
scope_boundaries:
|
||||
claimable_at_gate_d:
|
||||
- memory_lift_magnitude_and_significance_conv_scope_qwen_harness_head_373516c
|
||||
- per_cell_judge_accuracy_wilson_95
|
||||
- monotonicity_chain_observation_5_cell
|
||||
- conv_scope_fair_comparison_methodology
|
||||
- agentic_discipline_search_rate_turns_unknown_fallback
|
||||
not_claimable_at_gate_d:
|
||||
- direct_comparability_to_mem0_91_6_different_scope_and_memory_layer
|
||||
- multi_model_generalization_stage_3_is_qwen_only
|
||||
- production_waggle_orchestrator_performance
|
||||
reserved_for_pm_at_gate_d:
|
||||
- public_claim_phrasing_venue
|
||||
- matched_scope_mem0_co_run_stage_4
|
||||
- publication_timing
|
||||
cc1_does_not_compose_public_sota_claim: true
|
||||
|
||||
# ── Related artefacts ───────────────────────────────────────────────────────
|
||||
|
||||
related:
|
||||
v5_predecessor: fc169250c3c27cd3
|
||||
v4_ancestor: dedd69888e008fb158
|
||||
section_1_1_lock_waiver: 67eb89914a49ec38
|
||||
section_1_2_rca: 274e9871b54599077a3d
|
||||
section_1_3_probe_fail: 66dcd5a1b18b9367
|
||||
section_1_3b_scope_audit: 69a14708f78a74d2
|
||||
section_5_2_v5_rpm_20_edit: ad324ccf...
|
||||
section_1_3c_throttle_probe_pass: 3a146efc...
|
||||
fold_in_3_5b_sibling_mirror: d0ab680...
|
||||
section_1_3e_rpd_feasibility: 1d3851d...
|
||||
section_1_3f_vertex_batch_infeasible: 8ad056736f36b4ae
|
||||
section_1_3g_judge_swap_multi_pass: 8a2f0e61621f9684
|
||||
section_1_3h_stratified_reprobe: ae0d312b4088693e
|
||||
section_1_3h_c_deepseek_mt_bump: 005a19a8c6c4146c
|
||||
bench_spec_lock_v1_parent: PM-Waggle-OS/decisions/2026-04-22-bench-spec-locked.manifest.yaml
|
||||
stage_2_retry_gate_c_exit: PM-Waggle-OS/sessions/2026-04-24-task25-stage2-retry-complete.md
|
||||
v6_brief: PM-Waggle-OS/briefs/2026-04-24-cc1-manifest-v6-phase1-kappa-recal-brief.md
|
||||
rollback_tag: checkpoint/pre-self-evolution-2026-04-14
|
||||
canonical_md_surface: benchmarks/preregistration/manifest-v6-preregistration.md
|
||||
|
||||
# ── Validation gates ────────────────────────────────────────────────────────
|
||||
|
||||
validation_gates:
|
||||
before_phase_2_n400_kickoff_v6:
|
||||
- v6_anchor_commit_sha_recorded
|
||||
- v6_md_sha256_recorded_in_commit_message
|
||||
- v6_yaml_sha256_recorded_in_commit_message
|
||||
- litellm_config_v6_amendment_committed_separately
|
||||
- kappa_recalibration_phase_1_PASS_verdict
|
||||
- kappa_conservative_trio_value_gte_0_70
|
||||
- minimax_parse_rate_gte_95_per_100
|
||||
- pm_ratify_v6_kappa_received
|
||||
- kickoff_mechanism_clean_foreground_non_harness_process_tree
|
||||
at_gate_d_exit_v6:
|
||||
- all_2000_evals_accounted_in_denominators
|
||||
- evaluator_loss_reported_separately
|
||||
- judge_ensemble_fail_count_reported_separately
|
||||
- primary_fisher_one_sided_computed
|
||||
- secondary_endpoints_reported
|
||||
- budget_reconciled
|
||||
- deviation_count_reported
|
||||
- code_freeze_reverified_head_373516c_plus_v6_litellm_amendment
|
||||
1606
benchmarks/preregistration/manifest-v7-gepa-faza1.yaml
Normal file
1606
benchmarks/preregistration/manifest-v7-gepa-faza1.yaml
Normal file
File diff suppressed because it is too large
Load Diff
458
benchmarks/preregistration/manifest-v8-gaia2-preregistration.md
Normal file
458
benchmarks/preregistration/manifest-v8-gaia2-preregistration.md
Normal file
@@ -0,0 +1,458 @@
|
||||
# Manifest v8 — GAIA 2 Ambiguity+Adaptability Ablation Pre-Registration
|
||||
|
||||
**Manifest version:** v8.0.0-preregistration
|
||||
**Manifest type:** `gaia2_ambiguity_adaptability_hive_mind_ablation`
|
||||
**Preregistered date:** 2026-06-04
|
||||
**Authority:** PM (Marko Marković) — v8 emission under full PM authority. Extends waggle-os benchmark programme to the ARE/GAIA 2 evaluation environment. Does not supersede v6/v7 (LoCoMo / GEPA lineage continues independently). v8 governs all GAIA 2 work from this pre-registration forward.
|
||||
**Branch:** `feature/gaia2-hive-mind-ablation` (to be created at code-freeze)
|
||||
**Machine-readable twin:** [`manifest-v8-gaia2-preregistration.yaml`](manifest-v8-gaia2-preregistration.yaml)
|
||||
|
||||
---
|
||||
|
||||
## 0. Status
|
||||
|
||||
**PRE-REGISTERED — PENDING PM RATIFICATION FOR PHASE 1 (ENVIRONMENT UNBLOCK + ADAPTER BUILD).**
|
||||
|
||||
Any change to §1–§9 of this document after the anchor commit invalidates the pre-registration and requires a new PM-ratified manifest (v9+). The anchor commit SHA and SHA-256 of both MD and YAML files are recorded in the commit message body.
|
||||
|
||||
**Anchor commit SHA:** recorded in the git commit that adds these files.
|
||||
**Manifest SHA-256 (MD + YAML bytes):** `sha256sum benchmarks/preregistration/manifest-v8-gaia2-preregistration.{md,yaml}` — recorded in the commit message body.
|
||||
|
||||
---
|
||||
|
||||
## 0.1. Motivation and lineage
|
||||
|
||||
This pre-registration extends the waggle-os benchmark programme from its LoCoMo memory-recall focus (v2–v7) to the **GAIA 2 ARE** (Agent Runtime Environments) evaluation environment. The scientific motivation is a direct continuation of the hive-mind LoCoMo finding:
|
||||
|
||||
> **Substrate ≫ subject model**: Opus 4.7 and Qwen3.6-35B converge to 73.1% / 73.4% on identical retrieval substrate (hive-mind LoCoMo v5, N=320).
|
||||
|
||||
The LoCoMo benchmark tests single-turn factoid recall from conversational memory. The central open question is whether the **I/P/B frame architecture** (Intra, Predicted, Bidirectional frames in `@waggle/core::FrameStore`) provides incremental lift on tasks that require:
|
||||
|
||||
1. **Detecting and tracking conflicting instructions** across turns (GAIA 2 Ambiguity split)
|
||||
2. **Replanning in response to environment changes** that contradict earlier agent decisions (GAIA 2 Adaptability split)
|
||||
|
||||
These two splits were selected because they map directly onto the semantic purpose of I/P/B frames — specifically:
|
||||
- **P-frames (Predicted):** record agent hypotheses and planned actions before execution
|
||||
- **B-frames (Bidirectional):** record corrections and resolved contradictions
|
||||
- The Ambiguity split requires an agent to detect that two instructions conflict before acting
|
||||
- The Adaptability split requires an agent to update an earlier committed plan when new information arrives
|
||||
|
||||
The ARE SIGALRM blocker (`module 'signal' has no attribute 'SIGALRM'` — smoke run `smoke-c2-2026-04-30`) must be resolved as a **Phase 1 prerequisite** before any benchmark execution.
|
||||
|
||||
---
|
||||
|
||||
## 0.2. Relationship to prior manifests
|
||||
|
||||
| Manifest | Dataset | Claim | Status |
|
||||
|---|---|---|---|
|
||||
| v2–v5 | LoCoMo | Memory lift proof (retrieval cell) | Completed |
|
||||
| v6 | LoCoMo | Judge ensemble swap (MiniMax) | Completed |
|
||||
| v7 | LoCoMo (GEPA) | Prompt-shape evolution | Active |
|
||||
| **v8 (this)** | **GAIA 2 ARE** | **I/P/B frame lift on Ambiguity + Adaptability** | **Pre-registered** |
|
||||
|
||||
v8 does NOT inherit v6/v7 code freeze or judge ensemble choices. v8 is a new benchmark programme on a new evaluation framework. All sections are v8-native.
|
||||
|
||||
---
|
||||
|
||||
## 1. Primary hypothesis (directional, confirmatory)
|
||||
|
||||
> **I/P/B-augmented hive-mind substrate improves agent task success on GAIA 2 Ambiguity and Adaptability splits vs. a substrate-free baseline, by a margin of ≥ 10 percentage points.**
|
||||
>
|
||||
> `score(hive_mind_ipb) − score(no_memory_baseline) ≥ 10pp`
|
||||
>
|
||||
> evaluated at **Fisher exact one-sided** p-value **< 0.10** on the pooled Ambiguity + Adaptability scenario set.
|
||||
|
||||
**One-sided justification:** theory-driven directional claim. The I/P/B frame architecture was explicitly designed to track conflicting and corrective information. Both selected splits require this capability. No prior GAIA 2 / hive-mind result exists; direction is grounded in hive-mind LoCoMo monotonicity chain (no-memory 0.0% → retrieval 35% → agentic 40% → oracle 55% at N=20, Stage 2 Gate C) and the semantic alignment of I/P/B frames with Ambiguity + Adaptability task structure.
|
||||
|
||||
**Threshold rationale:** 10pp chosen over the LoCoMo-standard 5pp because:
|
||||
- GAIA 2 write-action scoring is harder (argument-level exact + soft checks vs. string match)
|
||||
- I/P/B frames are a structural advantage, not a marginal one, on contradiction-requiring tasks
|
||||
- A weaker effect at <10pp would be scientifically interesting but would not support a strong architectural claim
|
||||
- Power: 10pp effect size at N=100 per cell gives ~70% power at α=0.10 (Wilson-based estimate); adequate for a pilot
|
||||
|
||||
**Failure mode:** If primary fails (Δ < 10pp or p ≥ 0.10):
|
||||
- Report full cell distribution
|
||||
- PM adjudication on whether to run a larger N or revise the architectural integration
|
||||
- Do NOT claim negative result without replication; pilot N=100 may underpower marginal effects
|
||||
|
||||
---
|
||||
|
||||
## 2. Secondary endpoints (ex-ante, non-blocking on primary)
|
||||
|
||||
| # | Endpoint | Direction | Threshold | Test |
|
||||
|---|---|---|---|---|
|
||||
| S1 | `no_memory_baseline ≤ hive_mind_retrieval_only` | positive | ≥ 0pp | Fisher one-sided p < 0.20 |
|
||||
| S2 | `hive_mind_retrieval_only ≤ hive_mind_ipb` | positive | ≥ 0pp | Fisher one-sided p < 0.20 |
|
||||
| S3 | Ambiguity split: `hive_mind_ipb − no_memory_baseline` | descriptive | ≥ 0pp | point estimate + Wilson 95% CI |
|
||||
| S4 | Adaptability split: `hive_mind_ipb − no_memory_baseline` | descriptive | ≥ 0pp | point estimate + Wilson 95% CI |
|
||||
| S5 | Temporal split: `hive_mind_ipb − no_memory_baseline` | descriptive (null expected) | n/a | point estimate (negative control) |
|
||||
| S6 | Cost-efficiency: `hive_mind_ipb` cost per solved scenario vs. no-memory | descriptive | n/a | median + IQR in USD |
|
||||
| S7 | `subject_model_strong` (Opus 4.x) vs `subject_model_cheap` (Qwen3.6-35B) on `hive_mind_ipb` cell | descriptive | n/a | Δpp + Wilson 95% CI |
|
||||
|
||||
**S5 Negative control rationale:** the Temporal split is the hardest GAIA 2 split (all models), with failures driven by wall-clock latency, not memory architecture. hive-mind should not help here. If S5 shows large positive lift, it signals a confound in the experimental design.
|
||||
|
||||
Multi-comparisons policy: secondary endpoints are descriptive, no correction required. Primary hypothesis test is the only confirmatory statistical test.
|
||||
|
||||
---
|
||||
|
||||
## 3. Sample design
|
||||
|
||||
- **Cells:** four, run sequentially (concurrency = 1 per cell; within-cell parallelism TBD at Phase 2).
|
||||
|
||||
| Cell | Description |
|
||||
|---|---|
|
||||
| `no_memory_baseline` | Standard ARE ReAct loop, no memory injection, no hive-mind substrate |
|
||||
| `hive_mind_retrieval` | ARE ReAct loop + hive-mind `@waggle/core::HybridSearch` passive recall (no I/P/B frame writes) |
|
||||
| `hive_mind_ipb` | ARE ReAct loop + full hive-mind substrate (retrieval + I/P/B frame detection + contradiction gate) |
|
||||
| `hive_mind_ipb_strong` | Same as `hive_mind_ipb` with stronger subject model (Opus 4.x vs Qwen3.6) |
|
||||
|
||||
- **Primary splits:** Ambiguity + Adaptability (pooled for primary hypothesis test; per-split for S3/S4)
|
||||
- **Negative control split:** Temporal (S5)
|
||||
- **N per cell per split:** 50 scenarios × 2 splits = 100 scenarios per cell
|
||||
- **Total scenario executions:** 400 (4 cells × 100 scenarios)
|
||||
- **Instance selection seed:** `42`
|
||||
- **Matched design:** same 100 scenarios (50 Ambiguity + 50 Adaptability) flow through all four cells
|
||||
- **Negative control:** same 50 Temporal scenarios flow through `no_memory_baseline` and `hive_mind_ipb` only (100 additional executions)
|
||||
- **Total executions including negative control:** 500
|
||||
- **Concurrency (within cell):** `1` pending Phase 1 environment characterization; may be raised to `≤4` at Phase 2 gate if ARE rate limits permit, under PM authorization
|
||||
- **N justification:** N=50 per split per cell is the minimum for the 10pp threshold at α=0.10 with ~70% power. A larger N=100 per split per cell is preferred and reserved for PM authorization at Phase 2 if Phase 1 pilot (N=50 per split) is inconclusive.
|
||||
|
||||
---
|
||||
|
||||
## 4. Dataset — ARE/GAIA 2 scenarios
|
||||
|
||||
- **Source:** Meta AI ARE framework (arXiv:2509.17158), open-source MIT license, dataset CC BY 4.0
|
||||
- **Repository:** `https://github.com/facebookresearch/agentbenchmark` (canonical ARE repo — to be confirmed at Phase 1)
|
||||
- **Version:** ARE public release (800 scenarios across 10 universes), as of Phase 1 clone date
|
||||
- **Selected splits:** `ambiguity`, `adaptability`, `temporal` (negative control)
|
||||
- **Scenario selection:** deterministic seed-42 shuffle within each split, take-first-N
|
||||
- **Scenario count per split:**
|
||||
- Ambiguity: 50 (pilot); expand to 100 at Phase 2 gate if available
|
||||
- Adaptability: 50 (pilot); expand to 100 at Phase 2 gate if available
|
||||
- Temporal (negative control): 50 (no expansion planned)
|
||||
- **Dataset integrity:** SHA-256 of the scenario JSON files recorded at Phase 1 clone time in the run manifest
|
||||
- **No modifications** to scenario content, oracle traces, or verifier DAGs. ARE scenarios used as-is from the public release.
|
||||
|
||||
---
|
||||
|
||||
## 5. Model stack
|
||||
|
||||
### 5.1 Subject models
|
||||
|
||||
| Priority | Cell | Model alias | Provider | Thinking | Notes |
|
||||
|---|---|---|---|---|---|
|
||||
| primary | `no_memory_baseline`, `hive_mind_retrieval`, `hive_mind_ipb` | `qwen3.6-35b-a3b-via-dashscope-direct` | Alibaba DashScope-intl | on | Inherits v6 primary subject |
|
||||
| fallback | above cells | `qwen3.6-35b-a3b-via-openrouter` | OpenRouter | on | Inherits v6 fallback |
|
||||
| primary | `hive_mind_ipb_strong` | `claude-opus-4-x` | Anthropic direct | n/a | Exact alias pinned at Phase 1 |
|
||||
| fallback | `hive_mind_ipb_strong` | `claude-sonnet-4-x` | Anthropic direct | n/a | Fallback if Opus rate-limits |
|
||||
|
||||
Subject model pricing (Qwen3.6-35B): $0.20 / $0.80 per M in/out (inherits v6).
|
||||
Subject model pricing (Claude Opus 4.x): to be recorded at Phase 1 from Anthropic pricing page.
|
||||
|
||||
### 5.2 Judge protocol — ARE verifier (primary) + LLM soft judge (secondary)
|
||||
|
||||
GAIA 2 scoring uses a **two-layer judge**:
|
||||
|
||||
**Layer 1 — ARE verifier (deterministic):**
|
||||
- Hard checks: argument-level exact match for deterministic fields (dates, names, IDs)
|
||||
- Causality and relative-time constraint validation
|
||||
- Implemented in the ARE scenario DAG verifier; no LLM call
|
||||
- Score: binary pass/fail per hard-check field
|
||||
|
||||
**Layer 2 — LLM soft judge (for open-ended fields):**
|
||||
- Per ARE paper (arXiv:2509.17158): `Llama 3.3 Instruct 70B` used in paper baseline
|
||||
- **v8 policy:** replicate ARE paper's judge (`llama-3.3-70b-instruct-via-openrouter`) as Layer 2 primary for reproducibility with the leaderboard baseline
|
||||
- **v8 secondary judge:** `claude-opus-4-x` (Anthropic) for cross-validation of open-ended verdicts on a 20% random sample
|
||||
- Temperature: 0 for both judges
|
||||
- κ calibration: compute pairwise Cohen's κ (ARE-Llama vs. Opus) on the 20% sample at Phase 1 exit. Threshold: κ ≥ 0.65 PASS / κ < 0.60 HALT.
|
||||
|
||||
**Score aggregation:** final scenario score = arithmetic mean of (all hard-check pass rates + soft-judge pass rates) per ARE paper §4. Reported as a decimal in [0, 1].
|
||||
|
||||
**v8 does NOT use the waggle-os LoCoMo judge ensemble (Opus + GPT + MiniMax).** That ensemble was designed for factoid string-match accuracy. GAIA 2 scenarios require structural verifier checks that cannot be delegated to a general-purpose judge.
|
||||
|
||||
### 5.3 Health-check predicate
|
||||
|
||||
Pre-cell health check required before each cell execution:
|
||||
- ARE environment boot: `python -c "from agents_benchmark import ARE; ARE().ping()"` — must succeed
|
||||
- LiteLLM liveness: `/v1/chat/completions` ping on all active subject aliases
|
||||
- LLM soft judge: ping on `llama-3.3-70b-instruct-via-openrouter`
|
||||
- Ollama embedder (for `hive_mind_retrieval` + `hive_mind_ipb` cells): `curl http://localhost:11434/api/embeddings` with `nomic-embed-text`
|
||||
|
||||
Failure on any probe → halt before cell, PM raise.
|
||||
|
||||
---
|
||||
|
||||
## 6. Substrate
|
||||
|
||||
### 6.1 hive-mind substrate (`@waggle/core`)
|
||||
|
||||
Inherited from hive-mind v5 architecture (`hive-mind` repo, `feature/v5-distilled-dense` or equivalent frozen branch — to be confirmed at Phase 1).
|
||||
|
||||
**Components:**
|
||||
- `MindDB` (SQLite + sqlite-vec) — in-memory per scenario (`:memory:` path), ephemeral per run
|
||||
- `FrameStore` — I/P/B-frame CRUD + FTS5 auto-index
|
||||
- `HybridSearch` — RRF-fused FTS5 + vec0 search
|
||||
- `SessionStore` — per-scenario session lifecycle
|
||||
- `createOllamaEmbedder()` — `nomic-embed-text`, 1024 dims, local, $0
|
||||
|
||||
**Cell-specific substrate config:**
|
||||
|
||||
| Cell | `MindDB` | `FrameStore` | Frame types written | `HybridSearch` |
|
||||
|---|---|---|---|---|
|
||||
| `no_memory_baseline` | none | none | none | none |
|
||||
| `hive_mind_retrieval` | `:memory:` | active (read-only after ingest) | I-frames only (past turns ingested) | conv-scoped, top-K=20 |
|
||||
| `hive_mind_ipb` | `:memory:` | active (read+write during run) | I + P + B frames | conv-scoped, top-K=20 |
|
||||
| `hive_mind_ipb_strong` | same as `hive_mind_ipb` | same | same | same |
|
||||
|
||||
### 6.2 I/P/B frame integration contract for GAIA 2
|
||||
|
||||
This is the **core novel integration** that does not exist in the current codebase and must be built during Phase 1. The integration contract is pre-specified here to prevent methodology drift:
|
||||
|
||||
**Intra-frames (I-frames):** written after each user message and each environment event notification is received. Content = the raw turn text. Standard frame, equivalent to what LoCoMo ingest uses.
|
||||
|
||||
**Predicted-frames (P-frames):** written **before** each write-action tool call. Content = the agent's stated intent and predicted post-state (extracted from the agent's reasoning trace). Purpose: create a retrievable record of what the agent expected to happen, enabling contradiction detection when the environment responds differently.
|
||||
|
||||
**Bidirectional-frames (B-frames):** written when the agent detects a contradiction between:
|
||||
- A new user message and a previously recorded P-frame (instruction conflict → Ambiguity)
|
||||
- An environment event and a previously recorded P-frame (plan invalidated → Adaptability)
|
||||
|
||||
B-frames store: the conflicting pair (new content + prior P-frame ID), the resolved intent (if any), and a `conflict_type` label (`instruction_conflict` or `plan_invalidation`).
|
||||
|
||||
**Contradiction gate:** a lightweight check (single LLM call or heuristic rule, TBD at Phase 1 code-freeze) that fires before each write-action tool call when there are ≥ 1 P-frames in the substrate. If triggered, the gate:
|
||||
1. Retrieves the top-3 P-frames by semantic similarity to the current action
|
||||
2. Asks the subject model (or a cheap dedicated gate model, TBD): "Does the proposed action contradict any of these prior plans?"
|
||||
3. If YES → writes a B-frame and injects a `# Contradiction Warning` block into the agent's context before the write-action call
|
||||
4. If NO → proceeds without injection
|
||||
|
||||
The contradiction gate is the primary mechanism by which `hive_mind_ipb` is expected to outperform `hive_mind_retrieval` on the Ambiguity and Adaptability splits.
|
||||
|
||||
**Gate model (contradiction check):** default = same subject model as the outer agent loop (Qwen3.6-35B). Alternative: a lightweight dedicated model (e.g., `qwen3.6-8b` via OpenRouter) to reduce latency and cost. Decision deferred to Phase 1 gate after latency profiling.
|
||||
|
||||
### 6.3 Substrate lifecycle per scenario
|
||||
|
||||
1. `createSubstrate({dbPath: ':memory:'})` at scenario start
|
||||
2. Ingest prior turns (if scenario has a conversation history preamble) as I-frames
|
||||
3. Agent loop runs; P/B frames written dynamically during execution
|
||||
4. Scenario exits (success or failure per ARE verifier)
|
||||
5. `substrate.close()` — `:memory:` freed; no persistence between scenarios
|
||||
6. ARE verifier evaluates the recorded action trace independently of the substrate
|
||||
|
||||
Substrate state does NOT affect the ARE verifier. The verifier scores the agent's actual tool calls in the environment, not the memory contents.
|
||||
|
||||
---
|
||||
|
||||
## 7. Agent architecture — ARE ReAct loop with hive-mind hooks
|
||||
|
||||
**Base architecture:** ARE paper's standard uniform ReAct loop (temperature=0.5, max generation=16K tokens per ARE §4.2). This is the architecture all GAIA 2 baseline models use; it is adopted for comparability.
|
||||
|
||||
**Hive-mind hooks (new for v8):**
|
||||
|
||||
The `no_memory_baseline` cell uses the ARE loop unmodified.
|
||||
|
||||
The `hive_mind_*` cells wrap the ARE loop with three hooks:
|
||||
1. **Pre-turn hook:** `search_memory(query=current_user_message, topK=20, scopeToSession=true)` → inject `# Recalled Memories` block into system context (as in LoCoMo agentic cell). Skipped on turn 1 if no prior I-frames exist.
|
||||
2. **Pre-write-action hook:** contradiction gate (§6.2) — fires before any `write`-type ARE tool call
|
||||
3. **Post-turn hook:** write I-frame (always) + write P-frame if the turn contained a write-action intent
|
||||
|
||||
**Tool allowlist:** all 101 ARE tools remain available. No tool is blocked. The hive-mind substrate is additive, not restrictive.
|
||||
|
||||
**ARE environment:** Mobile universe (smartphone mock-up with Email, Calendar, Contacts, Shopping, FileSystem) unless the Ambiguity/Adaptability split scenarios span multiple universes — to be confirmed at Phase 1 dataset inspection. If multiple universes are present, universe distribution is reported in run metadata.
|
||||
|
||||
**SIGALRM blocker fix (prerequisite for Phase 1 execution):**
|
||||
|
||||
The smoke run `smoke-c2-2026-04-30` failed on all 3 runs with:
|
||||
```
|
||||
AttributeError: module 'signal' has no attribute 'SIGALRM'
|
||||
```
|
||||
This is a Windows/non-Unix signal error. Resolution options (in priority order):
|
||||
1. **WSL2 Linux environment** — run ARE inside WSL2 where `signal.SIGALRM` is available (zero code change, recommended)
|
||||
2. **Docker container** — run `benchmarks/harness` + ARE inside a Linux container (Docker Desktop on Windows)
|
||||
3. **SIGALRM shim** — patch ARE's timeout mechanism to use `asyncio.wait_for` instead of `signal.SIGALRM` (code change to ARE; requires upstreaming or local fork maintenance)
|
||||
|
||||
Option 1 (WSL2) is the pre-registered preferred path. If WSL2 is unavailable, Option 2 (Docker) is the fallback. Option 3 is a last resort requiring PM approval as a scope-deviation.
|
||||
|
||||
Phase 1 does NOT begin until the SIGALRM blocker is resolved and a clean smoke run (`0 exceptions, ≥ 1 scenario scored ≥ 0.5`) is recorded.
|
||||
|
||||
---
|
||||
|
||||
## 8. Stopping rules
|
||||
|
||||
| # | Rule | Trigger | Action |
|
||||
|---|---|---|---|
|
||||
| §8.1 | Budget hard halt | Cumulative spend ≥ **$80.00** | Halt, persist partial JSONL, PM raise |
|
||||
| §8.2 | Scenario exception rate | > 20% of scenarios in a cell raise Python exceptions | Halt cell, PM raise before next cell |
|
||||
| §8.3 | ARE environment health | ARE ping fails on pre-cell health check | Halt before cell |
|
||||
| §8.4 | Subject fetch failures | 5 consecutive LiteLLM 5xx / fetch errors | Halt, persist partial |
|
||||
| §8.5 | κ failure (Phase 1 exit) | κ(ARE-Llama, Opus) < 0.60 on 20% calibration sample | Halt, PM raise — Phase 2 requires judge renegotiation |
|
||||
| §8.6 | Deviation from §1–§9 | Any change detected during run | Immediate halt + PM raise + re-pre-registration required (v9+) |
|
||||
| §8.7 | SIGALRM unresolved | Phase 1 smoke run still fails after resolution attempt | Halt, report blocker to PM, await environment decision |
|
||||
|
||||
**No interim looks at primary hypothesis during execution.** Budget monitoring is continuous (non-statistical). Halts are operational, not inferential.
|
||||
|
||||
---
|
||||
|
||||
## 9. Post-hoc exclusion policy: NONE
|
||||
|
||||
All 500 scenario executions (400 primary + 100 negative control) enter the denominator. `verifier_error` and `environment_crash` scenarios are counted in denominator and reported as `execution_loss` separately. No scenario whitelist/blacklist. Selective exclusion forbidden ex-ante.
|
||||
|
||||
Exception: if the ARE environment produces a scenario with a known bug acknowledged in the ARE issue tracker, PM may authorize exclusion of that specific scenario ID with audit trail. This requires a new decision document (not a manifest amendment).
|
||||
|
||||
---
|
||||
|
||||
## 10. Deviation policy
|
||||
|
||||
Any deviation from §1–§9 during run → (1) immediate halt, (2) PM raise, (3) re-pre-registration (manifest v9+) if accepted.
|
||||
|
||||
**Permitted non-deviations (do not require v9):**
|
||||
- SIGALRM fix choice between Option 1 (WSL2) and Option 2 (Docker), per §7
|
||||
- Gate model selection for contradiction check (subject model vs. dedicated lightweight), per §6.2
|
||||
- Minor litellm-config.yaml amendments (new alias additions only, no semantic changes to existing aliases)
|
||||
- κ calibration sample size adjustment ±10% due to split availability (if < 50 scenarios exist in a split at Phase 1, N adjusts to available count — primary hypothesis threshold adjusts proportionally)
|
||||
|
||||
---
|
||||
|
||||
## 11. Code freeze
|
||||
|
||||
Code is frozen at the commit that adds this manifest to the repository. The following paths are frozen for the duration of v8 execution:
|
||||
|
||||
**Frozen paths (substrate and harness):**
|
||||
- `packages/core/src/mind/search.ts` (HybridSearch)
|
||||
- `packages/core/src/mind/frames.ts` (FrameStore, I/P/B frame types)
|
||||
- `packages/core/src/mind/sessions.ts` (SessionStore)
|
||||
- `packages/core/src/mind/db.ts` (MindDB)
|
||||
- `packages/agent/src/agent-loop.ts` (runAgentLoop)
|
||||
- `benchmarks/harness/src/substrate.ts`
|
||||
|
||||
**Frozen paths (v8-specific, to be created at Phase 1 code-freeze):**
|
||||
- `benchmarks/gaia2/src/adapter.ts` — ARE scenario ↔ harness schema adapter
|
||||
- `benchmarks/gaia2/src/cells-gaia2.ts` — four v8 cell implementations
|
||||
- `benchmarks/gaia2/src/contradiction-gate.ts` — I/P/B contradiction gate
|
||||
- `benchmarks/gaia2/src/runner-gaia2.ts` — v8 runner (wraps ARE loop)
|
||||
- `benchmarks/gaia2/src/judge-gaia2.ts` — ARE verifier + Llama soft-judge integration
|
||||
|
||||
These paths are frozen at Phase 1 code-freeze commit (separate from this pre-registration commit). Any post-freeze modification to the above paths triggers §8.6 (halt + PM raise).
|
||||
|
||||
**Permitted delta during run:** new JSONL files emitted to `benchmarks/gaia2/runs/` and `benchmarks/gaia2/results/`.
|
||||
|
||||
---
|
||||
|
||||
## 12. Scope boundaries
|
||||
|
||||
### Can claim at Gate D (post-run):
|
||||
- Magnitude + significance of I/P/B hive-mind lift on GAIA 2 Ambiguity + Adaptability splits under the pre-registered cell stack and subject models
|
||||
- Per-cell, per-split scenario success rates with Wilson 95% CIs
|
||||
- Negative control result (Temporal split)
|
||||
- Contradiction gate firing rate and per-firing outcome (resolved vs. unresolved)
|
||||
- Cost-per-solved-scenario across cells
|
||||
- Substrate-is-the-moat finding (if S7 shows Opus ≈ Qwen on `hive_mind_ipb`, replicating LoCoMo finding in GAIA 2 context)
|
||||
|
||||
### Cannot claim at Gate D:
|
||||
- General GAIA 2 leaderboard rank (v8 tests 2 of 7 splits; full leaderboard requires all splits)
|
||||
- Multi-model generalization beyond Qwen3.6-35B + Opus 4.x
|
||||
- Production agent performance on real mobile environments
|
||||
- Direct comparison to GAIA 2 paper baseline models (different scaffold; comparisons are indicative only)
|
||||
|
||||
### Reserved for PM:
|
||||
- Public claim phrasing and venue
|
||||
- Full 7-split GAIA 2 leaderboard submission (requires v9 pre-registration)
|
||||
- Publication timing and co-author decisions
|
||||
- "beats Opus 4.x without hive-mind" framing (requires v9 + explicit cross-model ablation design)
|
||||
|
||||
---
|
||||
|
||||
## 13. PM gates
|
||||
|
||||
### Gate P+ (Phase 1: environment unblock + adapter build)
|
||||
|
||||
**Trigger:** SIGALRM fix verified + clean smoke run + Phase 1 code-freeze commit
|
||||
|
||||
**Pre-kick checks required:**
|
||||
- SIGALRM resolution option confirmed (WSL2 or Docker)
|
||||
- Clean smoke run: `≥ 1 scenario scored ≥ 0.5`, `0 signal.SIGALRM exceptions`
|
||||
- All frozen paths (§11 v8-specific) committed at Phase 1 code-freeze
|
||||
- κ calibration plan confirmed (20% sample, Llama + Opus)
|
||||
- LiteLLM config includes all v8 subject model aliases
|
||||
- Ollama `nomic-embed-text` liveness confirmed in execution environment
|
||||
|
||||
**Action:** CC (agent) halts; awaits `PM-RATIFY-V8-PHASE1` before Phase 2 execution.
|
||||
|
||||
### Gate P++ (Phase 2: N=500 execution kick)
|
||||
|
||||
**Trigger:** `PM-RATIFY-V8-PHASE1` received after Gate P+ ratification.
|
||||
|
||||
**Pre-kick checks required:**
|
||||
- Phase 1 exit report at `benchmarks/gaia2/preregistration/phase1-exit-report.md`
|
||||
- κ(ARE-Llama, Opus) ≥ 0.65 on calibration sample
|
||||
- Budget envelope confirmed ($80 hard halt)
|
||||
- Cell order confirmed (no_memory_baseline → hive_mind_retrieval → hive_mind_ipb → hive_mind_ipb_strong)
|
||||
|
||||
**Action:** kick N=500 execution via CLI invocation template (§14).
|
||||
|
||||
### Gate D (post-run, pre-claim)
|
||||
|
||||
**Trigger:** N=500 run exit (clean or halted per §8)
|
||||
|
||||
**Action:** CC writes Gate D exit report at `benchmarks/gaia2/results/v8-gate-d-exit-report.md`. Halts. PM decides claim composition, venue, and whether to proceed to full 7-split leaderboard submission (v9).
|
||||
|
||||
**No self-advance at any gate.**
|
||||
|
||||
---
|
||||
|
||||
## 14. Budget
|
||||
|
||||
- **Hard halt:** $80.00
|
||||
- **Cap:** $90.00
|
||||
- **Expected burn:** ~$55–$70
|
||||
|
||||
| Component | Est. cost |
|
||||
|---|---|
|
||||
| Phase 1: smoke run validation, adapter testing, κ calibration | ~$5–10 |
|
||||
| N=500 executions — Qwen3.6-35B subject (3 cells × 100 scenarios × ~8K avg tokens) | ~$10–15 |
|
||||
| N=100 executions — Opus 4.x subject (1 cell × 100 scenarios) | ~$20–30 |
|
||||
| ARE Llama soft judge (N=500 × soft-check fields, est. avg 2 soft fields/scenario) | ~$5–8 |
|
||||
| Opus soft-judge cross-validation (20% sample = 100 scenarios) | ~$3–5 |
|
||||
| Contradiction gate calls (fired on est. 30% of Ambiguity/Adaptability turns) | ~$3–5 |
|
||||
| Ollama embedding (local, $0) | $0 |
|
||||
| **Total expected** | **~$41–73** |
|
||||
|
||||
Wall-clock estimate: Phase 1 ≤ 1 day; Phase 2 N=500 ≈ 4–8 hours (ARE scenarios have longer execution traces than LoCoMo turns; temporal scenarios have real wait periods that may be simulated or skipped).
|
||||
|
||||
---
|
||||
|
||||
## 15. Related artefacts
|
||||
|
||||
### v8-specific
|
||||
- **v8 anchor commit:** THIS COMMIT
|
||||
- **Phase 1 smoke run (failed):** `benchmarks/gaia2/runs/smoke-c2-2026-04-30/` (SIGALRM blocker — audit trail)
|
||||
- **ARE paper:** Froger et al., arXiv:2509.17158, Sept 2025
|
||||
- **GAIA 2 comparison report:** `docs/gaia-comparison.md` (research background, 2026-06-04)
|
||||
|
||||
### Prior waggle-os manifests (independent lineage)
|
||||
- v6 LoCoMo Stage 3 anchor: `fc169250c3c27cd3`
|
||||
- v7 GEPA Faza 1 anchor: see manifest-v7-gepa-faza1.yaml
|
||||
- Bench-Spec LOCK v1: `PM-Waggle-OS/decisions/2026-04-22-bench-spec-locked.manifest.yaml`
|
||||
|
||||
### hive-mind lineage
|
||||
- LoCoMo v5 result (73.1% Opus, 73.4% Qwen): `hive-mind/benchmarks/locomo/RESULTS.md`
|
||||
- Trio-strict re-judge (67.8% AND-of-3): `hive-mind/data/judgments/trio-judgments-v5-retrieval.v2.jsonl`
|
||||
- METHODOLOGY.md: `hive-mind/benchmarks/METHODOLOGY.md`
|
||||
|
||||
---
|
||||
|
||||
## Appendix A — Integration work required before Phase 1 code-freeze
|
||||
|
||||
This appendix documents the build tasks needed. It is informational; it is NOT part of the pre-registered methodology (§1–§9 are the invariant sections).
|
||||
|
||||
| Task | File to create | Description |
|
||||
|---|---|---|
|
||||
| A1 | `benchmarks/gaia2/src/adapter.ts` | Map ARE scenario JSON schema → harness `DatasetInstance` equivalent; extract split label, oracle trace, universe ID |
|
||||
| A2 | `benchmarks/gaia2/src/cells-gaia2.ts` | Four cell implementations wrapping ARE loop with hive-mind hooks (§7) |
|
||||
| A3 | `benchmarks/gaia2/src/contradiction-gate.ts` | P-frame retrieval + conflict check + B-frame write + context injection (§6.2) |
|
||||
| A4 | `benchmarks/gaia2/src/runner-gaia2.ts` | Scenario runner: load ARE env, run cell, call ARE verifier, emit JSONL |
|
||||
| A5 | `benchmarks/gaia2/src/judge-gaia2.ts` | ARE verifier wrapper + Llama 3.3 70B soft-judge + Opus cross-validation |
|
||||
| A6 | SIGALRM fix | WSL2 or Docker environment setup (§7); clean smoke run gate |
|
||||
| A7 | `litellm-config.yaml` amendment | Add `llama-3.3-70b-instruct-via-openrouter` alias + Opus 4.x alias (if not already present) |
|
||||
|
||||
---
|
||||
|
||||
_End of Manifest v8 pre-registration. This document is the anchor for all analysis choices at GAIA 2 Gate D exit. Prior waggle-os manifests (v2–v7) remain audit-immutable and govern the LoCoMo/GEPA programme independently._
|
||||
@@ -0,0 +1,498 @@
|
||||
# Manifest v8 — GAIA 2 Ambiguity+Adaptability Ablation Pre-Registration
|
||||
# Canonical markdown surface: manifest-v8-gaia2-preregistration.md
|
||||
# Authority: PM (Marko Markovic) — v8 emission 2026-06-04
|
||||
# SHA-256 of this YAML's bytes recorded in v8 anchor commit message.
|
||||
|
||||
manifest_version: v8.0.0-preregistration
|
||||
manifest_type: gaia2_ambiguity_adaptability_hive_mind_ablation
|
||||
locked_date: 2026-06-04
|
||||
authority: "PM (Marko Markovic) — v8 emission under full PM authority. Extends waggle-os benchmark programme to ARE/GAIA 2. Does not supersede v6/v7."
|
||||
sprint: 13
|
||||
task: "3.1_gaia2_hive_mind_ablation"
|
||||
stage: 1
|
||||
phase: preregistration
|
||||
branch: feature/gaia2-hive-mind-ablation
|
||||
supersedes: NONE # v8 extends the programme; v6/v7 LoCoMo/GEPA lineage continues independently
|
||||
inherits_from:
|
||||
- hive_mind_v5_architecture_locomo_73pct
|
||||
- waggle_os_bench_spec_lock_v1
|
||||
extends:
|
||||
- manifest_v6_locomo_stage3
|
||||
- manifest_v7_gepa_faza1
|
||||
|
||||
# ── Motivation ───────────────────────────────────────────────────────────────
|
||||
|
||||
motivation:
|
||||
parent_finding: "Substrate >> subject model: Opus 4.7 and Qwen3.6-35B converge to 73.1%/73.4% on identical retrieval substrate (hive-mind LoCoMo v5, N=320)"
|
||||
research_question: "Do I/P/B frames provide incremental lift on GAIA 2 tasks requiring instruction-conflict detection (Ambiguity split) and plan-invalidation recovery (Adaptability split)?"
|
||||
frame_alignment:
|
||||
P_frames: "Record agent hypotheses and planned actions before execution"
|
||||
B_frames: "Record corrections and resolved contradictions"
|
||||
Ambiguity_split: "Requires detecting that two instructions conflict before acting — directly maps to P-frame retrieval + B-frame write"
|
||||
Adaptability_split: "Requires updating a prior committed plan when new information arrives — directly maps to B-frame plan_invalidation type"
|
||||
known_blocker: "signal.SIGALRM unavailable on Windows — smoke run smoke-c2-2026-04-30 failed 3/3 scenarios with AttributeError"
|
||||
|
||||
# ── Primary Hypothesis ───────────────────────────────────────────────────────
|
||||
|
||||
primary_hypothesis:
|
||||
name: ipb_lift_ambiguity_adaptability
|
||||
direction: one_sided_positive
|
||||
statement: "score(hive_mind_ipb) - score(no_memory_baseline) >= 10pp on pooled Ambiguity+Adaptability splits"
|
||||
test: fisher_exact_one_sided
|
||||
alpha_threshold: 0.10
|
||||
effect_size_threshold_pp: 10
|
||||
pooling: ambiguity_plus_adaptability
|
||||
one_sided_justification:
|
||||
- "Theory-driven: I/P/B architecture explicitly designed for contradiction tracking"
|
||||
- "LoCoMo Gate C monotonicity: no-memory 0.0% < retrieval 35% < agentic 40% < oracle 55% at N=20"
|
||||
- "Semantic alignment: both selected splits require capabilities that map onto P/B frame semantics"
|
||||
threshold_rationale:
|
||||
- "10pp (vs 5pp LoCoMo standard) because GAIA 2 write-action scoring is harder (argument-level exact+soft vs string match)"
|
||||
- "I/P/B frames are structural advantage on contradiction tasks, not marginal"
|
||||
- "Power: ~70% at N=100 per cell, alpha=0.10 (Wilson-based estimate)"
|
||||
failure_policy:
|
||||
action: "Report full cell distribution; PM adjudication on larger N or revised integration"
|
||||
constraint: "Do NOT claim negative result without replication; N=100 pilot may underpower marginal effects"
|
||||
|
||||
# ── Secondary Endpoints ──────────────────────────────────────────────────────
|
||||
|
||||
secondary_endpoints:
|
||||
S1:
|
||||
name: monotonicity_baseline_to_retrieval
|
||||
statement: "no_memory_baseline <= hive_mind_retrieval"
|
||||
direction: one_sided_positive
|
||||
threshold_pp: 0
|
||||
test: fisher_exact_one_sided
|
||||
alpha: 0.20
|
||||
S2:
|
||||
name: monotonicity_retrieval_to_ipb
|
||||
statement: "hive_mind_retrieval <= hive_mind_ipb"
|
||||
direction: one_sided_positive
|
||||
threshold_pp: 0
|
||||
test: fisher_exact_one_sided
|
||||
alpha: 0.20
|
||||
S3:
|
||||
name: ambiguity_split_ipb_vs_baseline
|
||||
statement: "hive_mind_ipb - no_memory_baseline on Ambiguity split only"
|
||||
direction: descriptive
|
||||
threshold_pp: 0
|
||||
report: [point_estimate, wilson_95_ci]
|
||||
S4:
|
||||
name: adaptability_split_ipb_vs_baseline
|
||||
statement: "hive_mind_ipb - no_memory_baseline on Adaptability split only"
|
||||
direction: descriptive
|
||||
threshold_pp: 0
|
||||
report: [point_estimate, wilson_95_ci]
|
||||
S5:
|
||||
name: temporal_split_negative_control
|
||||
statement: "hive_mind_ipb vs no_memory_baseline on Temporal split (null expected)"
|
||||
direction: descriptive_null_expected
|
||||
rationale: "Temporal failures driven by wall-clock latency, not memory. Positive lift here signals confound."
|
||||
cells: [no_memory_baseline, hive_mind_ipb]
|
||||
report: [point_estimate]
|
||||
S6:
|
||||
name: cost_efficiency
|
||||
statement: "cost_per_solved_scenario across cells"
|
||||
direction: descriptive
|
||||
report: [median, iqr_usd]
|
||||
S7:
|
||||
name: substrate_vs_model_strength
|
||||
statement: "Opus 4.x vs Qwen3.6-35B on hive_mind_ipb cell"
|
||||
direction: descriptive
|
||||
rationale: "Replication of LoCoMo substrate-is-the-moat finding in GAIA 2 context"
|
||||
report: [delta_pp, wilson_95_ci]
|
||||
|
||||
# ── Sample Design ─────────────────────────────────────────────────────────────
|
||||
|
||||
sample:
|
||||
cells:
|
||||
- name: no_memory_baseline
|
||||
description: "Standard ARE ReAct loop, no memory injection, no hive-mind substrate"
|
||||
subject: qwen3.6-35b-a3b
|
||||
splits: [ambiguity, adaptability, temporal_negative_control]
|
||||
- name: hive_mind_retrieval
|
||||
description: "ARE ReAct loop + HybridSearch passive recall, I-frames only, no P/B frame writes"
|
||||
subject: qwen3.6-35b-a3b
|
||||
splits: [ambiguity, adaptability]
|
||||
- name: hive_mind_ipb
|
||||
description: "ARE ReAct loop + full hive-mind substrate (retrieval + I/P/B frames + contradiction gate)"
|
||||
subject: qwen3.6-35b-a3b
|
||||
splits: [ambiguity, adaptability, temporal_negative_control]
|
||||
- name: hive_mind_ipb_strong
|
||||
description: "Same as hive_mind_ipb with Opus 4.x subject"
|
||||
subject: claude-opus-4-x
|
||||
splits: [ambiguity, adaptability]
|
||||
n_per_cell_per_split:
|
||||
primary: 50
|
||||
pilot_expand_to: 100 # authorized at Phase 2 gate by PM if inconclusive
|
||||
n_primary_scenarios: 100 # 50 ambiguity + 50 adaptability per cell
|
||||
n_negative_control_scenarios: 50 # temporal, no_memory_baseline + hive_mind_ipb only
|
||||
total_executions: 500 # 4*100 primary + 2*50 negative control
|
||||
instance_selection_seed: 42
|
||||
instance_selection_method: "shuffle-then-take-first-N per split, deterministic given seed"
|
||||
matched_design: true # same scenarios flow through all cells
|
||||
concurrency: 1 # within-cell; may raise to <=4 at Phase 2 gate with PM authorization
|
||||
concurrency_rationale: "Conservative default pending ARE rate-limit characterization at Phase 1"
|
||||
|
||||
# ── Dataset ───────────────────────────────────────────────────────────────────
|
||||
|
||||
dataset:
|
||||
name: gaia2_are
|
||||
paper: "Froger et al., arXiv:2509.17158, September 2025"
|
||||
paper_url: https://arxiv.org/abs/2509.17158
|
||||
license: CC_BY_4.0
|
||||
framework_license: MIT
|
||||
canonical_repo: "https://github.com/facebookresearch/agentbenchmark" # to be confirmed at Phase 1
|
||||
total_public_scenarios: 800
|
||||
universes: 10
|
||||
selected_splits:
|
||||
- name: ambiguity
|
||||
description: "Clarification of conflicting/ambiguous requests (e.g., scheduling conflicts)"
|
||||
target_n: 50
|
||||
primary: true
|
||||
- name: adaptability
|
||||
description: "Response to dynamic environment changes (e.g., update email based on follow-up)"
|
||||
target_n: 50
|
||||
primary: true
|
||||
- name: temporal
|
||||
description: "Time-sensitive actions with deadlines — negative control"
|
||||
target_n: 50
|
||||
primary: false
|
||||
dataset_integrity:
|
||||
sha256_recorded_at: "Phase 1 clone time, in run manifest"
|
||||
modification_policy: "No modifications to scenario content, oracle traces, or verifier DAGs"
|
||||
|
||||
# ── Model Stack ───────────────────────────────────────────────────────────────
|
||||
|
||||
subject_models:
|
||||
primary:
|
||||
alias: qwen3.6-35b-a3b-via-dashscope-direct
|
||||
provider: alibaba_dashscope_intl
|
||||
thinking: on
|
||||
price_per_million_input_usd: 0.20
|
||||
price_per_million_output_usd: 0.80
|
||||
cells: [no_memory_baseline, hive_mind_retrieval, hive_mind_ipb]
|
||||
primary_fallback:
|
||||
alias: qwen3.6-35b-a3b-via-openrouter
|
||||
provider: openrouter
|
||||
thinking: on
|
||||
cells: [no_memory_baseline, hive_mind_retrieval, hive_mind_ipb]
|
||||
trigger: fetch_error_on_primary
|
||||
strong:
|
||||
alias: claude-opus-4-x # exact alias pinned at Phase 1
|
||||
provider: anthropic_direct
|
||||
cells: [hive_mind_ipb_strong]
|
||||
strong_fallback:
|
||||
alias: claude-sonnet-4-x # fallback if Opus rate-limits
|
||||
provider: anthropic_direct
|
||||
cells: [hive_mind_ipb_strong]
|
||||
trigger: rate_limit_on_opus
|
||||
|
||||
judge_protocol:
|
||||
layer_1_are_verifier:
|
||||
type: deterministic
|
||||
description: "ARE scenario DAG verifier — argument-level exact match for hard fields + causality/time constraint validation"
|
||||
llm_calls: 0
|
||||
layer_2_soft_judge:
|
||||
primary:
|
||||
alias: llama-3.3-70b-instruct-via-openrouter
|
||||
rationale: "Matches ARE paper baseline judge (arXiv:2509.17158 §4.2) for leaderboard comparability"
|
||||
temperature: 0
|
||||
applies_to: open_ended_fields_only
|
||||
cross_validation:
|
||||
alias: claude-opus-4-x
|
||||
sample_fraction: 0.20
|
||||
purpose: "Pairwise κ calibration: κ(ARE-Llama, Opus) >= 0.65 required for Phase 2 gate"
|
||||
temperature: 0
|
||||
score_aggregation: "arithmetic mean of (hard-check pass rates + soft-judge pass rates) per ARE §4"
|
||||
note: "v8 does NOT use the waggle-os LoCoMo judge ensemble (Opus+GPT+MiniMax). GAIA 2 requires structural verifier checks."
|
||||
|
||||
kappa_calibration:
|
||||
judges: [llama-3.3-70b-instruct, claude-opus-4-x]
|
||||
sample_fraction: 0.20
|
||||
metric: pairwise_cohen_kappa
|
||||
pass_threshold: 0.65
|
||||
halt_threshold: 0.60
|
||||
action_on_halt: "Phase 2 requires judge renegotiation; PM raise"
|
||||
|
||||
# ── Substrate ──────────────────────────────────────────────────────────────────
|
||||
|
||||
substrate:
|
||||
implementation: "@waggle/core (hive-mind v5 architecture)"
|
||||
components:
|
||||
MindDB: "SQLite + sqlite-vec, :memory: per scenario"
|
||||
FrameStore: "I/P/B-frame CRUD + FTS5 auto-index"
|
||||
HybridSearch: "RRF-fused FTS5 + vec0"
|
||||
SessionStore: "per-scenario session lifecycle"
|
||||
embedder: "createOllamaEmbedder(), nomic-embed-text, 1024 dims, local $0"
|
||||
lifecycle: "createSubstrate(:memory:) at scenario start; substrate.close() at scenario exit; no persistence between scenarios"
|
||||
verifier_independence: "ARE verifier scores actual tool calls in environment, not memory contents"
|
||||
cell_config:
|
||||
no_memory_baseline:
|
||||
MindDB: none
|
||||
FrameStore: none
|
||||
frame_types_written: none
|
||||
HybridSearch: none
|
||||
hive_mind_retrieval:
|
||||
MindDB: ":memory:"
|
||||
FrameStore: "active (read-only after ingest)"
|
||||
frame_types_written: [I_frames]
|
||||
HybridSearch: "conv-scoped, top-K=20"
|
||||
hive_mind_ipb:
|
||||
MindDB: ":memory:"
|
||||
FrameStore: "active (read+write during run)"
|
||||
frame_types_written: [I_frames, P_frames, B_frames]
|
||||
HybridSearch: "conv-scoped, top-K=20"
|
||||
hive_mind_ipb_strong:
|
||||
MindDB: ":memory:"
|
||||
FrameStore: "active (read+write during run)"
|
||||
frame_types_written: [I_frames, P_frames, B_frames]
|
||||
HybridSearch: "conv-scoped, top-K=20"
|
||||
|
||||
ipb_frame_contract:
|
||||
I_frames:
|
||||
written_when: "after each user message and each environment event notification"
|
||||
content: "raw turn text"
|
||||
P_frames:
|
||||
written_when: "BEFORE each write-action tool call"
|
||||
content: "agent's stated intent + predicted post-state (extracted from reasoning trace)"
|
||||
purpose: "retrievable record of expected outcome for contradiction detection"
|
||||
B_frames:
|
||||
written_when: "when contradiction detected between new content and prior P-frame"
|
||||
trigger_types:
|
||||
instruction_conflict: "new user message contradicts prior P-frame (Ambiguity split)"
|
||||
plan_invalidation: "environment event contradicts prior P-frame (Adaptability split)"
|
||||
content: "conflicting pair (new content + prior P-frame ID), resolved intent if any, conflict_type label"
|
||||
|
||||
contradiction_gate:
|
||||
fires_when: ">=1 P-frames in substrate AND agent is about to execute a write-action tool call"
|
||||
steps:
|
||||
1: "Retrieve top-3 P-frames by semantic similarity to current action"
|
||||
2: "Ask gate model: 'Does the proposed action contradict any of these prior plans?'"
|
||||
3_if_YES: "Write B-frame; inject '# Contradiction Warning' block into agent context before write-action"
|
||||
3_if_NO: "Proceed without injection"
|
||||
gate_model: "default = same subject model as outer agent loop; alternative lightweight model deferred to Phase 1 profiling"
|
||||
gate_model_decision: "deferred to Phase 1 gate after latency profiling — NOT a methodology deviation"
|
||||
|
||||
# ── Agent Architecture ────────────────────────────────────────────────────────
|
||||
|
||||
agent_architecture:
|
||||
base: "ARE paper standard uniform ReAct loop (temperature=0.5, max_generation=16384 tokens, per arXiv:2509.17158 §4.2)"
|
||||
base_rationale: "Adopted for leaderboard comparability — all GAIA 2 baseline models use this loop"
|
||||
hive_mind_hooks:
|
||||
no_memory_baseline: "ARE loop unmodified — no hooks"
|
||||
hive_mind_cells:
|
||||
pre_turn: "search_memory(query=current_user_message, topK=20, scopeToSession=true) -> inject Recalled Memories block"
|
||||
pre_write_action: "contradiction_gate (§6.2)"
|
||||
post_turn: "write I-frame (always) + write P-frame if turn contained write-action intent"
|
||||
tool_allowlist: "all 101 ARE tools available — hive-mind substrate is additive, not restrictive"
|
||||
|
||||
sigalrm_fix:
|
||||
blocker: "AttributeError: module 'signal' has no attribute 'SIGALRM' — Windows OS"
|
||||
failed_smoke_run: "benchmarks/gaia2/runs/smoke-c2-2026-04-30 (3/3 failures)"
|
||||
resolution_priority:
|
||||
1:
|
||||
name: WSL2
|
||||
description: "Run ARE inside WSL2 Linux environment — zero code change"
|
||||
status: preferred
|
||||
2:
|
||||
name: Docker
|
||||
description: "Run harness + ARE inside Linux container via Docker Desktop"
|
||||
status: fallback
|
||||
3:
|
||||
name: asyncio_shim
|
||||
description: "Patch ARE timeout to use asyncio.wait_for instead of signal.SIGALRM"
|
||||
status: last_resort_requires_PM_approval
|
||||
phase_1_gate: "No execution begins until clean smoke run (0 SIGALRM exceptions, >=1 scenario scored >=0.5)"
|
||||
|
||||
# ── Stopping Rules ────────────────────────────────────────────────────────────
|
||||
|
||||
stopping_rules:
|
||||
budget_hard_halt_usd: 80.00
|
||||
budget_cap_usd: 90.00
|
||||
scenario_exception_rate:
|
||||
threshold: 0.20 # 20% of scenarios in a cell
|
||||
action: halt_cell_and_PM_raise_before_next_cell
|
||||
are_environment_health:
|
||||
check: pre_cell_are_ping
|
||||
action: halt_before_cell
|
||||
subject_fetch_failures:
|
||||
consecutive_failures: 5
|
||||
action: halt_persist_partial
|
||||
kappa_failure:
|
||||
threshold_lt: 0.60
|
||||
phase: Phase_1_exit
|
||||
action: halt_PM_raise_judge_renegotiation
|
||||
deviation_from_preregistration:
|
||||
action: immediate_halt_PM_raise_repreregistration_required_v9
|
||||
sigalrm_unresolved:
|
||||
action: halt_report_blocker_await_environment_decision
|
||||
no_interim_looks: true
|
||||
|
||||
# ── Post-Hoc Exclusion ────────────────────────────────────────────────────────
|
||||
|
||||
post_hoc_exclusion:
|
||||
policy: none
|
||||
denominator: all_500_executions
|
||||
execution_loss_handling:
|
||||
included_in_denominator: true
|
||||
reported_separately: true
|
||||
sources:
|
||||
- verifier_error
|
||||
- environment_crash
|
||||
known_bug_exception:
|
||||
requires: "PM decision document (not manifest amendment)"
|
||||
condition: "ARE-acknowledged issue tracker bug on specific scenario ID"
|
||||
|
||||
# ── Deviation Policy ──────────────────────────────────────────────────────────
|
||||
|
||||
deviation_policy:
|
||||
on_detection:
|
||||
- immediate_halt
|
||||
- PM_raise
|
||||
- re_preregister_v9
|
||||
permitted_non_deviations:
|
||||
- "SIGALRM fix option choice (WSL2 vs Docker) per §7"
|
||||
- "Gate model selection for contradiction check (subject model vs lightweight) per §6.2"
|
||||
- "litellm-config.yaml alias additions (no semantic changes to existing aliases)"
|
||||
- "N adjustment if split has <50 scenarios at Phase 1 — threshold adjusts proportionally; PM raise required"
|
||||
|
||||
# ── Code Freeze ───────────────────────────────────────────────────────────────
|
||||
|
||||
code_freeze:
|
||||
head: "recorded at v8 anchor commit"
|
||||
branch: feature/gaia2-hive-mind-ablation
|
||||
frozen_substrate_paths:
|
||||
- packages/core/src/mind/search.ts
|
||||
- packages/core/src/mind/frames.ts
|
||||
- packages/core/src/mind/sessions.ts
|
||||
- packages/core/src/mind/db.ts
|
||||
- packages/agent/src/agent-loop.ts
|
||||
- benchmarks/harness/src/substrate.ts
|
||||
frozen_v8_paths_at_phase1_codefreeze:
|
||||
- benchmarks/gaia2/src/adapter.ts
|
||||
- benchmarks/gaia2/src/cells-gaia2.ts
|
||||
- benchmarks/gaia2/src/contradiction-gate.ts
|
||||
- benchmarks/gaia2/src/runner-gaia2.ts
|
||||
- benchmarks/gaia2/src/judge-gaia2.ts
|
||||
permitted_delta_during_run:
|
||||
- "new JSONL files to benchmarks/gaia2/runs/"
|
||||
- "new artefacts to benchmarks/gaia2/results/"
|
||||
|
||||
# ── Scope Boundaries ──────────────────────────────────────────────────────────
|
||||
|
||||
scope_boundaries:
|
||||
claimable_at_gate_d:
|
||||
- "I/P/B hive-mind lift magnitude + significance on GAIA 2 Ambiguity + Adaptability (pre-registered cells + models)"
|
||||
- "Per-cell per-split scenario success rates with Wilson 95% CIs"
|
||||
- "Temporal split negative control result"
|
||||
- "Contradiction gate firing rate + per-firing outcome (resolved vs unresolved)"
|
||||
- "Cost-per-solved-scenario across cells"
|
||||
- "Substrate-is-the-moat finding (if S7 Opus ~ Qwen on hive_mind_ipb)"
|
||||
not_claimable_at_gate_d:
|
||||
- "General GAIA 2 leaderboard rank (v8 tests 2 of 7 splits)"
|
||||
- "Multi-model generalization beyond Qwen3.6-35B + Opus 4.x"
|
||||
- "Production agent performance on real environments"
|
||||
- "'beats Opus 4.x without hive-mind' framing (requires v9 + explicit cross-model ablation)"
|
||||
reserved_for_pm:
|
||||
- "Public claim phrasing and venue"
|
||||
- "Full 7-split leaderboard submission (requires v9)"
|
||||
- "Publication timing and co-author decisions"
|
||||
|
||||
# ── PM Gates ─────────────────────────────────────────────────────────────────
|
||||
|
||||
pm_gates:
|
||||
gate_p_plus_phase_1_environment_unblock:
|
||||
trigger: "SIGALRM fix verified + clean smoke run + Phase 1 code-freeze commit"
|
||||
pre_kick_checks:
|
||||
- sigalrm_resolution_option_confirmed
|
||||
- clean_smoke_run_at_least_1_scenario_score_gte_0.5_zero_exceptions
|
||||
- all_v8_specific_frozen_paths_committed_at_phase1_codefreeze
|
||||
- kappa_calibration_plan_confirmed_20pct_sample_llama_plus_opus
|
||||
- litellm_config_includes_all_v8_subject_aliases
|
||||
- ollama_nomic_embed_text_liveness_confirmed_in_execution_environment
|
||||
action: "CC halts; awaits PM-RATIFY-V8-PHASE1 before Phase 2 execution"
|
||||
self_advance: forbidden
|
||||
gate_p_plus_plus_phase_2_execution:
|
||||
trigger: PM-RATIFY-V8-PHASE1
|
||||
pre_kick_checks:
|
||||
- phase1_exit_report_at_benchmarks_gaia2_preregistration_phase1_exit_report.md
|
||||
- kappa_llama_opus_gte_0.65_on_calibration_sample
|
||||
- budget_envelope_confirmed_80_hard_halt
|
||||
- cell_execution_order_confirmed
|
||||
action: "kick N=500 execution"
|
||||
execution_order: [no_memory_baseline, hive_mind_retrieval, hive_mind_ipb, hive_mind_ipb_strong]
|
||||
self_advance: forbidden
|
||||
gate_d_post_run:
|
||||
trigger: "N=500 run exit (clean or halted)"
|
||||
action: "CC writes gate-d exit report at benchmarks/gaia2/results/v8-gate-d-exit-report.md; halts"
|
||||
pm_decides:
|
||||
- claim_composition_and_venue
|
||||
- full_7_split_leaderboard_submission_v9
|
||||
- publication_timing
|
||||
self_advance: forbidden
|
||||
|
||||
# ── Budget ────────────────────────────────────────────────────────────────────
|
||||
|
||||
budget:
|
||||
hard_halt_usd: 80.00
|
||||
cap_usd: 90.00
|
||||
expected_burn_usd_range: [41, 73]
|
||||
breakdown_expected:
|
||||
phase_1_smoke_adapter_kappa: "5-10"
|
||||
n500_qwen_subject_3cells_100scenarios: "10-15"
|
||||
n100_opus_subject_1cell: "20-30"
|
||||
are_llama_soft_judge_n500: "5-8"
|
||||
opus_crossval_20pct: "3-5"
|
||||
contradiction_gate_calls_30pct_firing_rate: "3-5"
|
||||
ollama_embedding_local: 0
|
||||
wall_clock_estimate:
|
||||
phase_1_days: 1
|
||||
phase_2_hours_range: [4, 8]
|
||||
phase_2_note: "ARE scenarios have longer execution traces than LoCoMo turns; temporal scenario wait periods may require simulation"
|
||||
|
||||
# ── Build Tasks (Appendix A) ──────────────────────────────────────────────────
|
||||
|
||||
build_tasks_before_phase1_codefreeze:
|
||||
- id: A1
|
||||
file: benchmarks/gaia2/src/adapter.ts
|
||||
description: "Map ARE scenario JSON schema -> harness DatasetInstance equivalent; extract split label, oracle trace, universe ID"
|
||||
- id: A2
|
||||
file: benchmarks/gaia2/src/cells-gaia2.ts
|
||||
description: "Four cell implementations wrapping ARE loop with hive-mind hooks"
|
||||
- id: A3
|
||||
file: benchmarks/gaia2/src/contradiction-gate.ts
|
||||
description: "P-frame retrieval + conflict check + B-frame write + context injection"
|
||||
- id: A4
|
||||
file: benchmarks/gaia2/src/runner-gaia2.ts
|
||||
description: "Scenario runner: load ARE env, run cell, call ARE verifier, emit JSONL"
|
||||
- id: A5
|
||||
file: benchmarks/gaia2/src/judge-gaia2.ts
|
||||
description: "ARE verifier wrapper + Llama 3.3 70B soft-judge + Opus cross-validation"
|
||||
- id: A6
|
||||
description: "SIGALRM fix — WSL2 or Docker environment setup; clean smoke run gate"
|
||||
- id: A7
|
||||
file: litellm-config.yaml
|
||||
description: "Add llama-3.3-70b-instruct-via-openrouter alias + Opus 4.x alias if not present"
|
||||
|
||||
# ── Related Artefacts ──────────────────────────────────────────────────────────
|
||||
|
||||
related:
|
||||
v8_specific:
|
||||
anchor_commit: THIS_COMMIT
|
||||
failed_smoke_run: "benchmarks/gaia2/runs/smoke-c2-2026-04-30 (SIGALRM blocker — audit trail)"
|
||||
are_paper: "Froger et al., arXiv:2509.17158, September 2025"
|
||||
gaia2_comparison_report: "docs/gaia-comparison.md (2026-06-04)"
|
||||
prior_locomo_lineage:
|
||||
v6_anchor: fc169250c3c27cd3
|
||||
v7_gepa: see_manifest_v7_gepa_faza1.yaml
|
||||
bench_spec_lock_v1: PM-Waggle-OS/decisions/2026-04-22-bench-spec-locked.manifest.yaml
|
||||
hive_mind:
|
||||
locomo_v5_results: "hive-mind/benchmarks/locomo/RESULTS.md (73.1% Opus / 73.4% Qwen)"
|
||||
trio_strict_rejudge: "hive-mind/data/judgments/trio-judgments-v5-retrieval.v2.jsonl (67.8% AND-of-3)"
|
||||
methodology: "hive-mind/benchmarks/METHODOLOGY.md"
|
||||
|
||||
# ── Manifest Lock ──────────────────────────────────────────────────────────────
|
||||
|
||||
manifest_path: benchmarks/preregistration/manifest-v8-gaia2-preregistration.yaml
|
||||
manifest_locked_at: 2026-06-04T00:00:00Z
|
||||
449
benchmarks/preregistration/manifest-v8.1-multi-benchmark.md
Normal file
449
benchmarks/preregistration/manifest-v8.1-multi-benchmark.md
Normal file
@@ -0,0 +1,449 @@
|
||||
# Manifest v8.1 — Multi-Benchmark Programme Amendment
|
||||
## GAIA 2 · BEAM · LongMemEval-V2 · Terminal-Bench 2.0
|
||||
|
||||
**Manifest version:** v8.1.0-preregistration (amends v8.0.0)
|
||||
**Preregistered date:** 2026-06-04
|
||||
**Authority:** PM (Marko Marković)
|
||||
**Supersedes:** manifest-v8-gaia2-preregistration (v8.0.0) for scope section only.
|
||||
All methodology in v8.0.0 §1–§9 for GAIA 2 remains intact and unmodified.
|
||||
This document adds three new benchmark tracks (BEAM, LongMemEval-V2, Terminal-Bench 2.0)
|
||||
and defines their preregistered methodology, runability status, and relationship to the
|
||||
central I/P/B frame architectural claim.
|
||||
**Machine-readable twin:** `manifest-v8.1-multi-benchmark.yaml` (companion file)
|
||||
|
||||
---
|
||||
|
||||
## 0. Amendment rationale
|
||||
|
||||
v8.0.0 scoped the v8 benchmark programme to GAIA 2 only. This amendment adds three
|
||||
parallel tracks following a landscape review (2026-06-04) that identified:
|
||||
|
||||
1. **BEAM** — the only benchmark with an explicit, unsolved Contradiction Resolution (CR)
|
||||
category. CR is the direct evaluation target for hive-mind I/P/B B-frames. Runnable
|
||||
today with minimal adapter work. Highest priority addition.
|
||||
|
||||
2. **LongMemEval-V2** — brand new (May 2026), no published competitor results, uses
|
||||
web agent trajectories as memory haystacks. A different evaluation surface from V1
|
||||
and from LoCoMo. High publication opportunity precisely because it's fresh.
|
||||
|
||||
3. **Terminal-Bench 2.0** — positioning data only. Not a memory or agentic-task benchmark.
|
||||
Submission-only (no public runner). Qwen3.6-35B baseline already on leaderboard at
|
||||
24.6% via `little-coder`. Included as a capability floor reference, not as a
|
||||
primary scientific claim.
|
||||
|
||||
---
|
||||
|
||||
## 1. Four-benchmark overview
|
||||
|
||||
| Track | Benchmark | Primary claim | Runability | Adapter effort | Timeline |
|
||||
|---|---|---|---|---|---|
|
||||
| **A** | **BEAM** | I/P/B B-frames solve unsolved CR category | **Runnable now** — `pip install` + download script | Low — JSONL conversation → DatasetInstance | **Sprint 13, Phase 1** |
|
||||
| **B** | **GAIA 2 / ARE** | I/P/B frames lift Ambiguity + Adaptability splits | Blocked — SIGALRM fix required | High — ARE adapter + contradiction gate | Sprint 14, Phase 1 after SIGALRM |
|
||||
| **C** | **LongMemEval-V2** | Substrate advantage on web agent trajectory memory | Partially runnable — haystacks available, no public runner yet | Medium-high — trajectory Insert/Query API, multimodal, 25M+ token haystacks | Sprint 14–15, after BEAM |
|
||||
| **D** | **Terminal-Bench 2.0** | Positioning floor for waggle on coding tasks | Submission-only | None — submit scaffold as-is | Anytime — submit existing agent |
|
||||
|
||||
**Execution priority:** A → B (parallel after SIGALRM fix) → C → D (asynchronous).
|
||||
|
||||
The central scientific claim across all tracks:
|
||||
> **Substrate (I/P/B hive-mind) > subject model selection.**
|
||||
> Demonstrated on memory recall (LoCoMo, done), contradiction resolution (BEAM, Track A),
|
||||
> stateful task completion (GAIA 2, Track B), and trajectory experience memory (LME-V2, Track C).
|
||||
|
||||
---
|
||||
|
||||
## 2. Track A — BEAM
|
||||
|
||||
### 2.1 What BEAM actually is
|
||||
|
||||
**Paper:** "Beyond a Million Tokens: Benchmarking and Enhancing Long-Term Memory in LLMs"
|
||||
(arXiv:2510.27246, ICLR 2026). Authors: Tavakoli, Salemi, Ye, Abdalla, Zamani, Mitchell.
|
||||
**Repo:** https://github.com/mohammadtavakoli78/BEAM
|
||||
**License:** Not explicitly stated on repo (academic use; no commercial restriction noted).
|
||||
**Dataset:** 100 conversations × 4 context scales (128K / 500K / 1M / 10M tokens).
|
||||
**Questions:** 2,000 probing questions across 10 memory ability categories.
|
||||
|
||||
### 2.2 BEAM categories and hive-mind relevance
|
||||
|
||||
| Category | Abbrev | Direct I/P/B relevance | Current SOTA status |
|
||||
|---|---|---|---|
|
||||
| **Contradiction Resolution** | CR | **Highest** — B-frames are designed for exactly this | **Unsolved** — worst-performing category across all tested models |
|
||||
| **Knowledge Update** | KU | High — B-frames track fact revision | Moderate performance |
|
||||
| **Instruction Following** | IF | High — P-frames record instructions; B-frames detect violations | Moderate |
|
||||
| **Preference Following** | PF | Medium — I-frames track preferences | Moderate |
|
||||
| Multi-Session Reasoning | MR | Medium | Moderate |
|
||||
| Temporal Reasoning | TR | Low (same signal as LoCoMo negative control) | Moderate |
|
||||
| Information Extraction | IE | Low | High |
|
||||
| Event Ordering | EO | Low | Moderate |
|
||||
| Abstention | ABS | Low | High |
|
||||
| Summarization | SUM | Low | High |
|
||||
|
||||
**Primary focus for Track A:** CR + KU as confirmatory; IF + PF as secondary descriptive.
|
||||
IE, EO, ABS, SUM as negative controls (hive-mind should not help here).
|
||||
|
||||
### 2.3 BEAM runability assessment
|
||||
|
||||
```bash
|
||||
# Installation — no known blockers
|
||||
git clone https://github.com/mohammadtavakoli78/BEAM
|
||||
pip install -r requirements.txt
|
||||
|
||||
# Dataset download (pre-built, no generation needed for evaluation)
|
||||
python src/beam/download_dataset.py
|
||||
# → downloads to data/ directory; chat sizes 128K/500K/1M/10M
|
||||
|
||||
# Answer generation
|
||||
EVAL_TYPE="rag" # hive-mind cells use RAG eval type
|
||||
bash src/model_inference/answer_generation.sh
|
||||
|
||||
# Evaluation
|
||||
python -m src.evaluation.run_evaluation \
|
||||
--input_directory results/128K \
|
||||
--chat_size 128K \
|
||||
--start_index 0 --end_index 20 \
|
||||
--max_workers 4 \
|
||||
--allowed_result_files [cell_output_files]
|
||||
```
|
||||
|
||||
**No SIGALRM issue. No Windows signal dependency. Python-native pipeline.**
|
||||
|
||||
LLM config: `src/llms_config.json` — add Qwen3.6-35B DashScope + Opus 4.x entries.
|
||||
Judge: LLM-as-judge (configurable; use Llama-3.3-70B for leaderboard comparability).
|
||||
|
||||
### 2.4 Harness adapter for BEAM (Track A build tasks)
|
||||
|
||||
The waggle-os harness `DatasetInstance` schema maps cleanly:
|
||||
|
||||
| BEAM field | DatasetInstance field | Notes |
|
||||
|---|---|---|
|
||||
| conversation text (full) | `context` | Truncated per BEAM's chat_size (128K/500K/1M/10M) |
|
||||
| probing question | `question` | One question per instance |
|
||||
| reference answer | `expected[]` | BEAM uses nugget scoring; adapter extracts string answers |
|
||||
| conversation_id | `conversation_id` | BEAM conversation index |
|
||||
| memory_ability category | metadata only | Not in DatasetInstance schema; stored in output JSONL |
|
||||
|
||||
**`DatasetSpec.id`:** `'beam'` — add to the union in `types.ts` alongside existing `'locomo' | 'longmemeval' | 'synthetic'`.
|
||||
|
||||
**New build task B1:** `benchmarks/harness/scripts/build-beam-canonical.ts` — analogous to `build-locomo-canonical.ts`. Converts BEAM JSON format to JSONL with `DatasetInstance` schema. Emit one instance per (conversation_id × question) pair. Record SHA-256 of source download.
|
||||
|
||||
**New build task B2:** `benchmarks/harness/src/cells-beam.ts` — four cells for BEAM (no_memory_baseline, hive_mind_retrieval, hive_mind_ipb, hive_mind_ipb_strong). Identical architecture to GAIA 2 cells except:
|
||||
- Input is a long conversation (not an ARE scenario)
|
||||
- Output scoring uses BEAM's LLM judge, not ARE verifier
|
||||
- No write-action oracle — BEAM is read-only (QA over memory)
|
||||
|
||||
**New build task B3:** BEAM judge integration — call `src/evaluation/run_evaluation.py` from the TypeScript harness via child_process, or replicate the judge logic in `judge-beam.ts` using the same Llama-3.3-70B soft judge as GAIA 2.
|
||||
|
||||
### 2.5 BEAM primary hypothesis
|
||||
|
||||
> **I/P/B-augmented hive-mind improves Contradiction Resolution (CR) score on BEAM
|
||||
> vs. no-memory baseline by ≥ 8 percentage points, at α = 0.10 one-sided.**
|
||||
|
||||
**Scope:** 128K tier primary (all 20 conversations, CR + KU + IF questions only, N ≈ 200–300 questions depending on BEAM distribution).
|
||||
**Extension:** 1M tier as secondary descriptive (hive-mind advantage should widen at scale).
|
||||
**10M tier:** not in v8.1 scope (wall-clock cost + `:memory:` SQLite constraint at 10M tokens — requires chunked ingestion; deferred to v9).
|
||||
|
||||
**Secondary BEAM endpoints:**
|
||||
- S_B1: KU monotonicity (no_memory ≤ hive_mind_retrieval ≤ hive_mind_ipb)
|
||||
- S_B2: Negative control — IE, EO, SUM: hive_mind_ipb ≈ no_memory_baseline (≤ 3pp Δ)
|
||||
- S_B3: Substrate-is-the-moat — Opus 4.x vs Qwen3.6 on hive_mind_ipb (CR category)
|
||||
|
||||
### 2.6 BEAM cells
|
||||
|
||||
| Cell | Substrate | Frame types |
|
||||
|---|---|---|
|
||||
| `no_memory_baseline` | none | none |
|
||||
| `hive_mind_retrieval` | HybridSearch, I-frames only | I |
|
||||
| `hive_mind_ipb` | Full substrate | I + P + B |
|
||||
| `hive_mind_ipb_strong` | Full substrate | I + P + B |
|
||||
|
||||
**Note:** For BEAM, P-frames are written when the probing question is asked (recording the agent's retrieval intent). B-frames are written when retrieved content contradicts an earlier I-frame. The contradiction gate fires before answer generation (not before a write action, as in GAIA 2).
|
||||
|
||||
### 2.7 BEAM budget
|
||||
|
||||
| Component | Est. cost |
|
||||
|---|---|
|
||||
| N≈300 questions × 4 cells × 128K tier, Qwen3.6 | ~$8–12 |
|
||||
| Opus 4.x cell (hive_mind_ipb_strong) | ~$15–20 |
|
||||
| BEAM LLM judge (Llama-3.3-70B, N=1200 questions) | ~$3–5 |
|
||||
| **Total expected** | **~$26–37** |
|
||||
|
||||
**Hard halt:** $50 for Track A.
|
||||
|
||||
---
|
||||
|
||||
## 3. Track B — GAIA 2 / ARE
|
||||
|
||||
No changes to v8.0.0 methodology. Full preregistration at
|
||||
`benchmarks/preregistration/manifest-v8-gaia2-preregistration.md`.
|
||||
|
||||
**Status:** Blocked on SIGALRM fix. Track A (BEAM) runs first in Sprint 13.
|
||||
GAIA 2 Phase 1 begins in Sprint 14 after:
|
||||
1. SIGALRM resolution (WSL2 preferred)
|
||||
2. BEAM Track A results available (inform whether I/P/B contradiction gate works as expected before building GAIA 2 adapter)
|
||||
|
||||
**Dependency:** Track A is the methodological pilot for the contradiction gate.
|
||||
If B-frames don't lift CR on BEAM (Track A fails), revisit the gate design before
|
||||
investing Sprint 14 effort in the GAIA 2 adapter.
|
||||
|
||||
---
|
||||
|
||||
## 4. Track C — LongMemEval-V2
|
||||
|
||||
### 4.1 What LME-V2 actually is
|
||||
|
||||
**Paper:** "LongMemEval-V2: Evaluating Long-Term Agent Memory on Web Agent Trajectories"
|
||||
(arXiv:2605.12493, May 2026). Lead: Xiaowu Li (UCLA).
|
||||
**Website:** https://xiaowu0162.github.io/longmemeval-v2/
|
||||
**Questions:** 451 manually curated.
|
||||
**Context scale:** 25M tokens (Small, 100 trajectories) / 115M tokens (Medium, 500 trajectories).
|
||||
**Tiers:** LME-V2-Small (100-trajectory shared haystack) / LME-V2-Medium (500-trajectory question-specific).
|
||||
|
||||
**Critical distinction from V1:** LME-V2 is a **web agent experience memory** benchmark.
|
||||
Memory haystacks are web browsing trajectories (screenshot + accessibility tree + BrowserGym action).
|
||||
It is NOT a conversational memory benchmark. This is a fundamentally different evaluation surface.
|
||||
|
||||
**Five memory abilities in V2:**
|
||||
1. **Static State Recall** — remember a fact from agent history (closest to LoCoMo V1)
|
||||
2. **Dynamic State Tracking** — track evolving state across trajectory steps
|
||||
3. **Workflow Knowledge** — remember procedural patterns from past agent sessions
|
||||
4. **Environment Gotchas** — recall known failure modes in the current environment
|
||||
5. **Premise Awareness** — detect that a question assumes something false about the environment
|
||||
|
||||
**hive-mind fit by category:**
|
||||
- Dynamic State Tracking → **high** (P-frames record state before actions; B-frames track state changes)
|
||||
- Environment Gotchas → **high** (I-frames accumulate error observations; B-frames flag recurrence)
|
||||
- Premise Awareness → **medium** (B-frames can detect false premises if prior I-frames contain contradicting evidence)
|
||||
- Static State Recall → low (same as LoCoMo; retrieval is sufficient)
|
||||
- Workflow Knowledge → low (procedural; I-frame accumulation is sufficient)
|
||||
|
||||
### 4.2 LME-V2 runability assessment
|
||||
|
||||
**Current status: partially runnable.**
|
||||
- Trajectory dataset: available via project website (haystacks can be downloaded)
|
||||
- Evaluation harness: **no public runner released yet** as of 2026-06-04
|
||||
- Paper uses Codex + GPT-5.4-mini as the evaluation agent
|
||||
- AgentLab framework (ServiceNow): https://github.com/ServiceNow/AgentLab — provides the execution environment
|
||||
- **The Insert/Query API is the evaluation interface**, not a CLI benchmark runner
|
||||
|
||||
**Adapter complexity:** High. Each trajectory contains screenshot + accessibility tree + BrowserGym action. Ingesting into hive-mind requires:
|
||||
1. Stripping screenshots (or OCR-ing them) for I-frame text content
|
||||
2. Treating each trajectory step as an I-frame with metadata (trajectory_id, step_index, action)
|
||||
3. Implementing `Insert(trajectory)` and `Query(question)` over hive-mind's HybridSearch
|
||||
|
||||
**Recommendation:** Target **LME-V2-Small only** in v8.1. 100 trajectories per question, text-only (accessibility tree, no screenshots). Defer multimodal screenshots to v9.
|
||||
|
||||
### 4.3 LME-V2 preregistration (conditional)
|
||||
|
||||
**LME-V2 execution is GATED on Track A (BEAM) completion AND PM-RATIFY-V8C.**
|
||||
|
||||
Rationale: LME-V2 is a significant new adapter build (~comparable to GAIA 2). Running it before BEAM validates the I/P/B integration would be premature. PM ratification is required before Sprint 15 build allocation.
|
||||
|
||||
**Provisional primary hypothesis for LME-V2 (subject to PM ratification):**
|
||||
> I/P/B-augmented hive-mind improves Dynamic State Tracking + Environment Gotchas
|
||||
> on LME-V2-Small vs. no-memory baseline by ≥ 8 percentage points.
|
||||
|
||||
**Provisional cells:** identical to BEAM (4 cells). Reader model: Qwen3.5-9B per paper baseline.
|
||||
**N:** 451 questions on Small tier (100-trajectory haystack).
|
||||
|
||||
**LME-V2 build tasks (pre-ratification design only):**
|
||||
- C1: `benchmarks/longmemeval-v2/src/adapter.ts` — trajectory → I-frame ingestion (text-only, accessibility tree)
|
||||
- C2: `benchmarks/longmemeval-v2/src/cells-lmev2.ts` — 4 cells implementing Insert/Query protocol
|
||||
- C3: `benchmarks/longmemeval-v2/src/judge-lmev2.ts` — normalized string match (structured) + LLM judge (free-form)
|
||||
|
||||
### 4.4 LME-V2 budget (provisional)
|
||||
|
||||
| Component | Est. cost |
|
||||
|---|---|
|
||||
| N=451 × 4 cells × Qwen3.6 reader | ~$10–15 |
|
||||
| Opus 4.x strong cell | ~$20–30 |
|
||||
| LLM judge (GPT-5.2 medium per paper) | ~$5–10 |
|
||||
| **Total expected** | **~$35–55** |
|
||||
|
||||
**Hard halt:** $65 for Track C.
|
||||
|
||||
---
|
||||
|
||||
## 5. Track D — Terminal-Bench 2.0
|
||||
|
||||
### 5.1 Status and rationale
|
||||
|
||||
Terminal-Bench 2.0 tests long-horizon agentic coding and system administration in
|
||||
terminal environments. Waggle is not a coding system. This track is **positioning
|
||||
data only** — it establishes waggle's floor on a broadly-followed leaderboard and
|
||||
provides a signal about whether the general agent scaffold (not the memory substrate)
|
||||
is competitive.
|
||||
|
||||
**Critical fact: Qwen3.6-35B via `little-coder` is already on the leaderboard at
|
||||
entries #118 and #123 at 24.6% ± 3.2 and 23.0% respectively (submitted 2026-05-14).
|
||||
This baseline exists.** No new run is required to have a data point.
|
||||
|
||||
### 5.2 What a waggle submission would add
|
||||
|
||||
The `little-coder` entries (#118/#123) use Qwen3.6-35B but no waggle scaffold and
|
||||
no hive-mind. A waggle-scaffolded submission would test:
|
||||
- Whether waggle's tool-calling loop (ReAct + Plan-Execute + Critic hybrid per v7 GEPA)
|
||||
outperforms a bare `little-coder` harness on terminal tasks
|
||||
- Whether hive-mind memory helps on long-running tasks (task-state persistence)
|
||||
|
||||
**Expected result:** moderate improvement from waggle scaffold (architectural advantage);
|
||||
small or zero improvement from hive-mind (terminal tasks are not memory-intensive in
|
||||
the same way as LoCoMo/BEAM/GAIA 2).
|
||||
|
||||
### 5.3 Terminal-Bench 2.0 runability
|
||||
|
||||
**Submission-only via `harborframework/terminal-bench-2-leaderboard`.**
|
||||
No public runner or local evaluation. Requires submitting agent code; Terminal-Bench
|
||||
team runs the evaluation.
|
||||
|
||||
**Current leaderboard context (as of 2026-06-02):**
|
||||
- Top: `vix` + Claude Opus 4.7 = 90.2%
|
||||
- Claude Code + Claude Opus 4.6 = 58.0% (#52)
|
||||
- `little-coder` + Qwen3.6-35B-A3B = 24.6% (#118) — our baseline
|
||||
- Waggle scaffold target: ≥ 35% (matching Claude Opus 4.5 era baselines)
|
||||
|
||||
### 5.4 Terminal-Bench submission plan
|
||||
|
||||
**No preregistration required** — Terminal-Bench is positioning data, not a confirmatory
|
||||
scientific claim. The submission is not governed by waggle-os preregistration policy.
|
||||
|
||||
**Execution:** submit to `harborframework/terminal-bench-2-leaderboard` after BEAM
|
||||
Track A completion (Sprint 13 end). Use Qwen3.6-35B + waggle scaffold, no hive-mind
|
||||
(isolate scaffold contribution). If scaffold submission scores ≥ 35%, add a second
|
||||
submission with hive-mind to test the memory lift.
|
||||
|
||||
**No budget cap required** — Terminal-Bench evaluations run on their infrastructure.
|
||||
|
||||
---
|
||||
|
||||
## 6. Amended execution timeline
|
||||
|
||||
```
|
||||
Sprint 13 (now)
|
||||
├── Track A: BEAM Phase 1
|
||||
│ ├── build-beam-canonical.ts (B1)
|
||||
│ ├── cells-beam.ts (B2)
|
||||
│ ├── judge-beam.ts (B3)
|
||||
│ └── N≈300 × 4 cells execution (Qwen3.6 + Opus 4.x)
|
||||
│
|
||||
├── Track D: Terminal-Bench submission
|
||||
│ └── Submit waggle scaffold (async, no gate)
|
||||
│
|
||||
Sprint 14
|
||||
├── Track B: GAIA 2 Phase 1 (after SIGALRM fix + BEAM CR result validates gate)
|
||||
│ ├── SIGALRM resolution (WSL2)
|
||||
│ ├── ARE adapter (A1–A5)
|
||||
│ └── N=500 × 4 cells execution
|
||||
│
|
||||
├── PM-RATIFY-V8C decision: proceed with LME-V2?
|
||||
│
|
||||
Sprint 15 (conditional)
|
||||
└── Track C: LongMemEval-V2 (gated on PM-RATIFY-V8C)
|
||||
├── Trajectory adapter (C1–C3)
|
||||
└── N=451 × 4 cells execution
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7. Cross-benchmark claim architecture
|
||||
|
||||
The four tracks build a layered argument:
|
||||
|
||||
```
|
||||
Layer 1 (DONE) LoCoMo v5 → substrate > model on factoid recall
|
||||
Qwen 73.4% ≈ Opus 73.1% with hive-mind
|
||||
|
||||
Layer 2 (BEAM) BEAM 128K CR/KU → I/P/B B-frames solve unsolved CR category
|
||||
[Track A] → first published system to address the open problem
|
||||
|
||||
Layer 3 (GAIA 2) ARE Ambiguity + → I/P/B lift carries into stateful write-action environment
|
||||
Adaptability → substrate > model on interactive agent tasks
|
||||
[Track B]
|
||||
|
||||
Layer 4 (LME-V2) Web agent → Dynamic State + Gotcha categories: P/B frames
|
||||
trajectories → track agent state and recurring failures
|
||||
[Track C]
|
||||
|
||||
Positioning Terminal-Bench → waggle scaffold competitiveness floor
|
||||
[Track D]
|
||||
```
|
||||
|
||||
The claim stacks: each layer adds a new evaluation surface while reusing the same
|
||||
architectural claim. The B-frame contradiction gate is the single mechanism tested
|
||||
across BEAM (QA), GAIA 2 (write-action), and LME-V2 (trajectory).
|
||||
|
||||
---
|
||||
|
||||
## 8. Amended gates
|
||||
|
||||
### Gate A-P+ (Track A Phase 1 kick)
|
||||
**Pre-kick checks:**
|
||||
- BEAM dataset download complete and SHA-256 recorded
|
||||
- `build-beam-canonical.ts` output validated (N instances ≥ 1,800 for 128K tier)
|
||||
- `cells-beam.ts` dry-run passes (all 4 cells, 5-instance smoke)
|
||||
- LiteLLM config includes Qwen3.6 DashScope + Llama-3.3-70B judge aliases
|
||||
- hive-mind `:memory:` substrate liveness confirmed
|
||||
- Budget envelope confirmed ($50 hard halt)
|
||||
|
||||
**Action:** kick N≈300 × 4 cells.
|
||||
|
||||
### Gate A-D (Track A post-run)
|
||||
**Action:** CC writes `benchmarks/beam/results/v8a-gate-d-exit-report.md`.
|
||||
**PM decides:**
|
||||
1. Is the CR lift ≥ 8pp? If yes → proceed to GAIA 2 (B) and LME-V2 gate
|
||||
2. Is the contradiction gate working? If no → redesign gate before GAIA 2
|
||||
3. Terminal-Bench submission: submit waggle scaffold result
|
||||
|
||||
### Gate V8C-PM (Track C authorization)
|
||||
Separate PM ratification required before LME-V2 build starts.
|
||||
|
||||
---
|
||||
|
||||
## 9. Budget summary
|
||||
|
||||
| Track | Hard halt | Expected burn |
|
||||
|---|---|---|
|
||||
| A — BEAM | $50 | $26–37 |
|
||||
| B — GAIA 2 | $80 | $41–73 (per v8.0.0) |
|
||||
| C — LME-V2 | $65 | $35–55 (provisional) |
|
||||
| D — Terminal-Bench | $0 (external infra) | $0 |
|
||||
| **Programme total** | **$195** | **$102–165** |
|
||||
|
||||
---
|
||||
|
||||
## 10. Scope boundaries (amended)
|
||||
|
||||
### Added at Gate A-D:
|
||||
- BEAM CR + KU lift magnitude + significance (pre-registered cells + models, 128K tier)
|
||||
- BEAM negative control result (IE, EO, SUM)
|
||||
- Contradiction gate firing rate on BEAM conversations
|
||||
|
||||
### Added at Gate D (GAIA 2, per v8.0.0):
|
||||
- Per v8.0.0 §12 (unchanged)
|
||||
|
||||
### Added at Gate C-D (LME-V2, conditional):
|
||||
- Dynamic State Tracking + Environment Gotchas lift on LME-V2-Small
|
||||
- Trajectory I-frame ingestion lift vs. no-memory baseline
|
||||
|
||||
### NOT claimable from Track D:
|
||||
- Terminal-Bench results are NOT a preregistered claim; they are positioning data
|
||||
|
||||
### Cannot claim from any track:
|
||||
- "Waggle OS beats [model X] on all benchmarks" — each track has specific splits and conditions
|
||||
- Generalization beyond the preregistered cells and models
|
||||
|
||||
---
|
||||
|
||||
## 11. Related artefacts
|
||||
|
||||
- **v8.0.0 anchor:** `benchmarks/preregistration/manifest-v8-gaia2-preregistration.md`
|
||||
(commit SHA `a3ae4cada43c` / `51b6bcc9c39c`)
|
||||
- **BEAM repo:** https://github.com/mohammadtavakoli78/BEAM (ICLR 2026)
|
||||
- **LME-V2 paper:** https://arxiv.org/html/2605.12493v1 (May 2026)
|
||||
- **Terminal-Bench 2.0 leaderboard:** https://www.tbench.ai/leaderboard/terminal-bench/2.0
|
||||
- **`little-coder` baseline entries:** #118 (24.6% ± 3.2) and #123 (23.0%), submitted 2026-05-14
|
||||
- **hive-mind LoCoMo v5 results:** `hive-mind/benchmarks/locomo/RESULTS.md`
|
||||
|
||||
---
|
||||
|
||||
_End of Manifest v8.1 amendment. v8.0.0 GAIA 2 methodology is unchanged._
|
||||
_v8.1 governs Track A (BEAM) and Track C (LME-V2) preregistration._
|
||||
_Track D (Terminal-Bench) is positioning data, not governed by preregistration policy._
|
||||
110
benchmarks/preregistration/manifest-v8.2-final.md
Normal file
110
benchmarks/preregistration/manifest-v8.2-final.md
Normal file
@@ -0,0 +1,110 @@
|
||||
# Manifest v8.2 — Final Scope Amendment (supersedes v8.0 and v8.1)
|
||||
|
||||
**Manifest version:** v8.2.0-preregistration (final scope)
|
||||
**Preregistered date:** 2026-06-04
|
||||
**Authority:** PM (Marko Marković)
|
||||
**Supersedes:** manifest-v8-gaia2-preregistration.md (v8.0.0) and
|
||||
manifest-v8.1-multi-benchmark.md (v8.1.0) for scope section only.
|
||||
All GAIA 2 methodology in v8.0.0 §1–§9 remains intact and unmodified.
|
||||
|
||||
---
|
||||
|
||||
## 0. Amendment rationale
|
||||
|
||||
v8.1.0 added BEAM (Track A), LongMemEval-V2 (Track C), and Terminal-Bench 2.0 (Track D).
|
||||
This amendment makes two scope changes:
|
||||
|
||||
1. **LongMemEval-V2 dropped** — deferred to v9. No public runner, 25 M+ token haystacks,
|
||||
and web-agent trajectory tooling are all pre-stable. Attempting V2 in v8 would block
|
||||
the release milestone without scientific payoff. The V1 benchmark is fully runnable
|
||||
today (harness already recognises `longmemeval`), and its `knowledge-update` split
|
||||
directly tests the same substrate claim.
|
||||
|
||||
2. **LongMemEval V1 added as Track A0** — becomes the fastest path to a new data point.
|
||||
500 questions, S-variant (115 K tokens/question), zero new harness work required.
|
||||
Hard budget cap $20. Runs before BEAM.
|
||||
|
||||
---
|
||||
|
||||
## 1. Final four-track scope
|
||||
|
||||
| Track | Benchmark | Primary claim surface | Status | Budget hard halt |
|
||||
|---|---|---|---|---|
|
||||
| **A0** | **LongMemEval V1** | Conversational memory — `knowledge-update` split | Runnable now | **$20** |
|
||||
| **A** | **BEAM** | Contradiction resolution (unsolved category) | Runnable now | **$50** |
|
||||
| **B** | **GAIA 2 / ARE** | Ambiguity + Adaptability stateful tasks | Blocked — SIGALRM fix | **$80** (per v8.0) |
|
||||
| **D** | **Terminal-Bench 2.0** | Capability floor / positioning | Async submission | **$0** |
|
||||
|
||||
> **LME-V2 (former Track C) is out of v8 scope.** It is preregistered as a v9 track
|
||||
> with no methodology commitments in this document.
|
||||
|
||||
---
|
||||
|
||||
## 2. Execution order
|
||||
|
||||
```
|
||||
A0 (LongMemEval V1) ← first: fastest path, harness already knows 'longmemeval'
|
||||
↓
|
||||
A (BEAM) ← second: pip install + download, low adapter effort
|
||||
↓
|
||||
B (GAIA 2) ← third: blocked on SIGALRM fix; starts in parallel after fix
|
||||
D (Terminal-Bench) ← async: submit existing scaffold any time, not gating release
|
||||
```
|
||||
|
||||
Rationale for A0-first ordering: the harness `longmemeval` dataset identifier is already
|
||||
wired; the canonical builder (`build-longmemeval-canonical.ts`) produces a drop-in JSONL;
|
||||
the S-variant is 500 questions at manageable cost. A0 provides an independent memory
|
||||
result to strengthen the central claim before BEAM runs.
|
||||
|
||||
---
|
||||
|
||||
## 3. Central scientific claim (unchanged from v8.1)
|
||||
|
||||
> **Substrate (I/P/B hive-mind) > subject model selection.**
|
||||
>
|
||||
> Demonstrated across three evaluation surfaces:
|
||||
> - **Conversational memory** — LongMemEval V1, `knowledge-update` split (Track A0)
|
||||
> - **Contradiction resolution** — BEAM CR category (Track A), the category that remains
|
||||
> unsolved for all subject models at the time of preregistration
|
||||
> - **Stateful agent tasks** — GAIA 2 Ambiguity + Adaptability splits (Track B)
|
||||
|
||||
Terminal-Bench 2.0 (Track D) provides **positioning data only**. It is not part of the
|
||||
scientific claim and will not be cited as evidence for the substrate > subject hypothesis.
|
||||
|
||||
---
|
||||
|
||||
## 4. Unchanged elements from v8.0.0
|
||||
|
||||
- All GAIA 2 methodology (§1–§9 of manifest-v8-gaia2-preregistration.md) is unmodified.
|
||||
- Track B $80 hard budget halt is unchanged.
|
||||
- Contradiction resolution as primary BEAM focus is unchanged.
|
||||
- The I/P/B frame taxonomy (I = Identity, P = Procedural, B = Belief/Contradiction) is
|
||||
unchanged.
|
||||
|
||||
---
|
||||
|
||||
## 5. LME-V2 deferral note (v9 placeholder)
|
||||
|
||||
LongMemEval-V2 (web-agent trajectory haystacks, multimodal, 25 M+ token context) is
|
||||
deferred to v9. No methodology is preregistered here. A v9 amendment will cover:
|
||||
trajectory Insert/Query API, multimodal evidence handling, and comparison to V1 baseline.
|
||||
|
||||
---
|
||||
|
||||
## 6. Budget summary
|
||||
|
||||
| Track | Hard halt | Notes |
|
||||
|---|---|---|
|
||||
| A0 — LongMemEval V1 | $20 | S-variant only (500 questions) |
|
||||
| A — BEAM | $50 | 128 K chat-size default |
|
||||
| B — GAIA 2 | $80 | Per v8.0.0 methodology |
|
||||
| D — Terminal-Bench | $0 | Submit only |
|
||||
| **Total exposure** | **$150** | Sequential gating limits concurrent spend |
|
||||
|
||||
---
|
||||
|
||||
## 7. Prior anchors
|
||||
|
||||
- `benchmarks/preregistration/manifest-v8-gaia2-preregistration.md` — v8.0.0 base
|
||||
- `benchmarks/preregistration/manifest-v8.1-multi-benchmark.md` — v8.1.0 amendment
|
||||
- This document (v8.2.0) supersedes both for scope; neither for methodology.
|
||||
Reference in New Issue
Block a user