moving

benchmarks/preregistration/manifest-v8.1-multi-benchmark.md

@@ -0,0 +1,449 @@
+# Manifest v8.1 — Multi-Benchmark Programme Amendment
+## GAIA 2 · BEAM · LongMemEval-V2 · Terminal-Bench 2.0
+
+**Manifest version:** v8.1.0-preregistration (amends v8.0.0)
+**Preregistered date:** 2026-06-04
+**Authority:** PM (Marko Marković)
+**Supersedes:** manifest-v8-gaia2-preregistration (v8.0.0) for scope section only.
+All methodology in v8.0.0 §1–§9 for GAIA 2 remains intact and unmodified.
+This document adds three new benchmark tracks (BEAM, LongMemEval-V2, Terminal-Bench 2.0)
+and defines their preregistered methodology, runability status, and relationship to the
+central I/P/B frame architectural claim.
+**Machine-readable twin:** `manifest-v8.1-multi-benchmark.yaml` (companion file)
+
+---
+
+## 0. Amendment rationale
+
+v8.0.0 scoped the v8 benchmark programme to GAIA 2 only. This amendment adds three
+parallel tracks following a landscape review (2026-06-04) that identified:
+
+1. **BEAM** — the only benchmark with an explicit, unsolved Contradiction Resolution (CR)
+   category. CR is the direct evaluation target for hive-mind I/P/B B-frames. Runnable
+   today with minimal adapter work. Highest priority addition.
+
+2. **LongMemEval-V2** — brand new (May 2026), no published competitor results, uses
+   web agent trajectories as memory haystacks. A different evaluation surface from V1
+   and from LoCoMo. High publication opportunity precisely because it's fresh.
+
+3. **Terminal-Bench 2.0** — positioning data only. Not a memory or agentic-task benchmark.
+   Submission-only (no public runner). Qwen3.6-35B baseline already on leaderboard at
+   24.6% via `little-coder`. Included as a capability floor reference, not as a
+   primary scientific claim.
+
+---
+
+## 1. Four-benchmark overview
+
+| Track | Benchmark | Primary claim | Runability | Adapter effort | Timeline |
+|---|---|---|---|---|---|
+| **A** | **BEAM** | I/P/B B-frames solve unsolved CR category | **Runnable now** — `pip install` + download script | Low — JSONL conversation → DatasetInstance | **Sprint 13, Phase 1** |
+| **B** | **GAIA 2 / ARE** | I/P/B frames lift Ambiguity + Adaptability splits | Blocked — SIGALRM fix required | High — ARE adapter + contradiction gate | Sprint 14, Phase 1 after SIGALRM |
+| **C** | **LongMemEval-V2** | Substrate advantage on web agent trajectory memory | Partially runnable — haystacks available, no public runner yet | Medium-high — trajectory Insert/Query API, multimodal, 25M+ token haystacks | Sprint 14–15, after BEAM |
+| **D** | **Terminal-Bench 2.0** | Positioning floor for waggle on coding tasks | Submission-only | None — submit scaffold as-is | Anytime — submit existing agent |
+
+**Execution priority:** A → B (parallel after SIGALRM fix) → C → D (asynchronous).
+
+The central scientific claim across all tracks:
+> **Substrate (I/P/B hive-mind) > subject model selection.**
+> Demonstrated on memory recall (LoCoMo, done), contradiction resolution (BEAM, Track A),
+> stateful task completion (GAIA 2, Track B), and trajectory experience memory (LME-V2, Track C).
+
+---
+
+## 2. Track A — BEAM
+
+### 2.1 What BEAM actually is
+
+**Paper:** "Beyond a Million Tokens: Benchmarking and Enhancing Long-Term Memory in LLMs"
+(arXiv:2510.27246, ICLR 2026). Authors: Tavakoli, Salemi, Ye, Abdalla, Zamani, Mitchell.
+**Repo:** https://github.com/mohammadtavakoli78/BEAM
+**License:** Not explicitly stated on repo (academic use; no commercial restriction noted).
+**Dataset:** 100 conversations × 4 context scales (128K / 500K / 1M / 10M tokens).
+**Questions:** 2,000 probing questions across 10 memory ability categories.
+
+### 2.2 BEAM categories and hive-mind relevance
+
+| Category | Abbrev | Direct I/P/B relevance | Current SOTA status |
+|---|---|---|---|
+| **Contradiction Resolution** | CR | **Highest** — B-frames are designed for exactly this | **Unsolved** — worst-performing category across all tested models |
+| **Knowledge Update** | KU | High — B-frames track fact revision | Moderate performance |
+| **Instruction Following** | IF | High — P-frames record instructions; B-frames detect violations | Moderate |
+| **Preference Following** | PF | Medium — I-frames track preferences | Moderate |
+| Multi-Session Reasoning | MR | Medium | Moderate |
+| Temporal Reasoning | TR | Low (same signal as LoCoMo negative control) | Moderate |
+| Information Extraction | IE | Low | High |
+| Event Ordering | EO | Low | Moderate |
+| Abstention | ABS | Low | High |
+| Summarization | SUM | Low | High |
+
+**Primary focus for Track A:** CR + KU as confirmatory; IF + PF as secondary descriptive.
+IE, EO, ABS, SUM as negative controls (hive-mind should not help here).
+
+### 2.3 BEAM runability assessment
+
+```bash
+# Installation — no known blockers
+git clone https://github.com/mohammadtavakoli78/BEAM
+pip install -r requirements.txt
+
+# Dataset download (pre-built, no generation needed for evaluation)
+python src/beam/download_dataset.py
+# → downloads to data/ directory; chat sizes 128K/500K/1M/10M
+
+# Answer generation
+EVAL_TYPE="rag"  # hive-mind cells use RAG eval type
+bash src/model_inference/answer_generation.sh
+
+# Evaluation
+python -m src.evaluation.run_evaluation \
+  --input_directory results/128K \
+  --chat_size 128K \
+  --start_index 0 --end_index 20 \
+  --max_workers 4 \
+  --allowed_result_files [cell_output_files]
+```
+
+**No SIGALRM issue. No Windows signal dependency. Python-native pipeline.**
+
+LLM config: `src/llms_config.json` — add Qwen3.6-35B DashScope + Opus 4.x entries.
+Judge: LLM-as-judge (configurable; use Llama-3.3-70B for leaderboard comparability).
+
+### 2.4 Harness adapter for BEAM (Track A build tasks)
+
+The waggle-os harness `DatasetInstance` schema maps cleanly:
+
+| BEAM field | DatasetInstance field | Notes |
+|---|---|---|
+| conversation text (full) | `context` | Truncated per BEAM's chat_size (128K/500K/1M/10M) |
+| probing question | `question` | One question per instance |
+| reference answer | `expected[]` | BEAM uses nugget scoring; adapter extracts string answers |
+| conversation_id | `conversation_id` | BEAM conversation index |
+| memory_ability category | metadata only | Not in DatasetInstance schema; stored in output JSONL |
+
+**`DatasetSpec.id`:** `'beam'` — add to the union in `types.ts` alongside existing `'locomo' | 'longmemeval' | 'synthetic'`.
+
+**New build task B1:** `benchmarks/harness/scripts/build-beam-canonical.ts` — analogous to `build-locomo-canonical.ts`. Converts BEAM JSON format to JSONL with `DatasetInstance` schema. Emit one instance per (conversation_id × question) pair. Record SHA-256 of source download.
+
+**New build task B2:** `benchmarks/harness/src/cells-beam.ts` — four cells for BEAM (no_memory_baseline, hive_mind_retrieval, hive_mind_ipb, hive_mind_ipb_strong). Identical architecture to GAIA 2 cells except:
+- Input is a long conversation (not an ARE scenario)
+- Output scoring uses BEAM's LLM judge, not ARE verifier
+- No write-action oracle — BEAM is read-only (QA over memory)
+
+**New build task B3:** BEAM judge integration — call `src/evaluation/run_evaluation.py` from the TypeScript harness via child_process, or replicate the judge logic in `judge-beam.ts` using the same Llama-3.3-70B soft judge as GAIA 2.
+
+### 2.5 BEAM primary hypothesis
+
+> **I/P/B-augmented hive-mind improves Contradiction Resolution (CR) score on BEAM
+> vs. no-memory baseline by ≥ 8 percentage points, at α = 0.10 one-sided.**
+
+**Scope:** 128K tier primary (all 20 conversations, CR + KU + IF questions only, N ≈ 200–300 questions depending on BEAM distribution).
+**Extension:** 1M tier as secondary descriptive (hive-mind advantage should widen at scale).
+**10M tier:** not in v8.1 scope (wall-clock cost + `:memory:` SQLite constraint at 10M tokens — requires chunked ingestion; deferred to v9).
+
+**Secondary BEAM endpoints:**
+- S_B1: KU monotonicity (no_memory ≤ hive_mind_retrieval ≤ hive_mind_ipb)
+- S_B2: Negative control — IE, EO, SUM: hive_mind_ipb ≈ no_memory_baseline (≤ 3pp Δ)
+- S_B3: Substrate-is-the-moat — Opus 4.x vs Qwen3.6 on hive_mind_ipb (CR category)
+
+### 2.6 BEAM cells
+
+| Cell | Substrate | Frame types |
+|---|---|---|
+| `no_memory_baseline` | none | none |
+| `hive_mind_retrieval` | HybridSearch, I-frames only | I |
+| `hive_mind_ipb` | Full substrate | I + P + B |
+| `hive_mind_ipb_strong` | Full substrate | I + P + B |
+
+**Note:** For BEAM, P-frames are written when the probing question is asked (recording the agent's retrieval intent). B-frames are written when retrieved content contradicts an earlier I-frame. The contradiction gate fires before answer generation (not before a write action, as in GAIA 2).
+
+### 2.7 BEAM budget
+
+| Component | Est. cost |
+|---|---|
+| N≈300 questions × 4 cells × 128K tier, Qwen3.6 | ~$8–12 |
+| Opus 4.x cell (hive_mind_ipb_strong) | ~$15–20 |
+| BEAM LLM judge (Llama-3.3-70B, N=1200 questions) | ~$3–5 |
+| **Total expected** | **~$26–37** |
+
+**Hard halt:** $50 for Track A.
+
+---
+
+## 3. Track B — GAIA 2 / ARE
+
+No changes to v8.0.0 methodology. Full preregistration at
+`benchmarks/preregistration/manifest-v8-gaia2-preregistration.md`.
+
+**Status:** Blocked on SIGALRM fix. Track A (BEAM) runs first in Sprint 13.
+GAIA 2 Phase 1 begins in Sprint 14 after:
+1. SIGALRM resolution (WSL2 preferred)
+2. BEAM Track A results available (inform whether I/P/B contradiction gate works as expected before building GAIA 2 adapter)
+
+**Dependency:** Track A is the methodological pilot for the contradiction gate.
+If B-frames don't lift CR on BEAM (Track A fails), revisit the gate design before
+investing Sprint 14 effort in the GAIA 2 adapter.
+
+---
+
+## 4. Track C — LongMemEval-V2
+
+### 4.1 What LME-V2 actually is
+
+**Paper:** "LongMemEval-V2: Evaluating Long-Term Agent Memory on Web Agent Trajectories"
+(arXiv:2605.12493, May 2026). Lead: Xiaowu Li (UCLA).
+**Website:** https://xiaowu0162.github.io/longmemeval-v2/
+**Questions:** 451 manually curated.
+**Context scale:** 25M tokens (Small, 100 trajectories) / 115M tokens (Medium, 500 trajectories).
+**Tiers:** LME-V2-Small (100-trajectory shared haystack) / LME-V2-Medium (500-trajectory question-specific).
+
+**Critical distinction from V1:** LME-V2 is a **web agent experience memory** benchmark.
+Memory haystacks are web browsing trajectories (screenshot + accessibility tree + BrowserGym action).
+It is NOT a conversational memory benchmark. This is a fundamentally different evaluation surface.
+
+**Five memory abilities in V2:**
+1. **Static State Recall** — remember a fact from agent history (closest to LoCoMo V1)
+2. **Dynamic State Tracking** — track evolving state across trajectory steps
+3. **Workflow Knowledge** — remember procedural patterns from past agent sessions
+4. **Environment Gotchas** — recall known failure modes in the current environment
+5. **Premise Awareness** — detect that a question assumes something false about the environment
+
+**hive-mind fit by category:**
+- Dynamic State Tracking → **high** (P-frames record state before actions; B-frames track state changes)
+- Environment Gotchas → **high** (I-frames accumulate error observations; B-frames flag recurrence)
+- Premise Awareness → **medium** (B-frames can detect false premises if prior I-frames contain contradicting evidence)
+- Static State Recall → low (same as LoCoMo; retrieval is sufficient)
+- Workflow Knowledge → low (procedural; I-frame accumulation is sufficient)
+
+### 4.2 LME-V2 runability assessment
+
+**Current status: partially runnable.**
+- Trajectory dataset: available via project website (haystacks can be downloaded)
+- Evaluation harness: **no public runner released yet** as of 2026-06-04
+- Paper uses Codex + GPT-5.4-mini as the evaluation agent
+- AgentLab framework (ServiceNow): https://github.com/ServiceNow/AgentLab — provides the execution environment
+- **The Insert/Query API is the evaluation interface**, not a CLI benchmark runner
+
+**Adapter complexity:** High. Each trajectory contains screenshot + accessibility tree + BrowserGym action. Ingesting into hive-mind requires:
+1. Stripping screenshots (or OCR-ing them) for I-frame text content
+2. Treating each trajectory step as an I-frame with metadata (trajectory_id, step_index, action)
+3. Implementing `Insert(trajectory)` and `Query(question)` over hive-mind's HybridSearch
+
+**Recommendation:** Target **LME-V2-Small only** in v8.1. 100 trajectories per question, text-only (accessibility tree, no screenshots). Defer multimodal screenshots to v9.
+
+### 4.3 LME-V2 preregistration (conditional)
+
+**LME-V2 execution is GATED on Track A (BEAM) completion AND PM-RATIFY-V8C.**
+
+Rationale: LME-V2 is a significant new adapter build (~comparable to GAIA 2). Running it before BEAM validates the I/P/B integration would be premature. PM ratification is required before Sprint 15 build allocation.
+
+**Provisional primary hypothesis for LME-V2 (subject to PM ratification):**
+> I/P/B-augmented hive-mind improves Dynamic State Tracking + Environment Gotchas
+> on LME-V2-Small vs. no-memory baseline by ≥ 8 percentage points.
+
+**Provisional cells:** identical to BEAM (4 cells). Reader model: Qwen3.5-9B per paper baseline.
+**N:** 451 questions on Small tier (100-trajectory haystack).
+
+**LME-V2 build tasks (pre-ratification design only):**
+- C1: `benchmarks/longmemeval-v2/src/adapter.ts` — trajectory → I-frame ingestion (text-only, accessibility tree)
+- C2: `benchmarks/longmemeval-v2/src/cells-lmev2.ts` — 4 cells implementing Insert/Query protocol
+- C3: `benchmarks/longmemeval-v2/src/judge-lmev2.ts` — normalized string match (structured) + LLM judge (free-form)
+
+### 4.4 LME-V2 budget (provisional)
+
+| Component | Est. cost |
+|---|---|
+| N=451 × 4 cells × Qwen3.6 reader | ~$10–15 |
+| Opus 4.x strong cell | ~$20–30 |
+| LLM judge (GPT-5.2 medium per paper) | ~$5–10 |
+| **Total expected** | **~$35–55** |
+
+**Hard halt:** $65 for Track C.
+
+---
+
+## 5. Track D — Terminal-Bench 2.0
+
+### 5.1 Status and rationale
+
+Terminal-Bench 2.0 tests long-horizon agentic coding and system administration in
+terminal environments. Waggle is not a coding system. This track is **positioning
+data only** — it establishes waggle's floor on a broadly-followed leaderboard and
+provides a signal about whether the general agent scaffold (not the memory substrate)
+is competitive.
+
+**Critical fact: Qwen3.6-35B via `little-coder` is already on the leaderboard at
+entries #118 and #123 at 24.6% ± 3.2 and 23.0% respectively (submitted 2026-05-14).
+This baseline exists.** No new run is required to have a data point.
+
+### 5.2 What a waggle submission would add
+
+The `little-coder` entries (#118/#123) use Qwen3.6-35B but no waggle scaffold and
+no hive-mind. A waggle-scaffolded submission would test:
+- Whether waggle's tool-calling loop (ReAct + Plan-Execute + Critic hybrid per v7 GEPA)
+  outperforms a bare `little-coder` harness on terminal tasks
+- Whether hive-mind memory helps on long-running tasks (task-state persistence)
+
+**Expected result:** moderate improvement from waggle scaffold (architectural advantage);
+small or zero improvement from hive-mind (terminal tasks are not memory-intensive in
+the same way as LoCoMo/BEAM/GAIA 2).
+
+### 5.3 Terminal-Bench 2.0 runability
+
+**Submission-only via `harborframework/terminal-bench-2-leaderboard`.**
+No public runner or local evaluation. Requires submitting agent code; Terminal-Bench
+team runs the evaluation.
+
+**Current leaderboard context (as of 2026-06-02):**
+- Top: `vix` + Claude Opus 4.7 = 90.2%
+- Claude Code + Claude Opus 4.6 = 58.0% (#52)
+- `little-coder` + Qwen3.6-35B-A3B = 24.6% (#118) — our baseline
+- Waggle scaffold target: ≥ 35% (matching Claude Opus 4.5 era baselines)
+
+### 5.4 Terminal-Bench submission plan
+
+**No preregistration required** — Terminal-Bench is positioning data, not a confirmatory
+scientific claim. The submission is not governed by waggle-os preregistration policy.
+
+**Execution:** submit to `harborframework/terminal-bench-2-leaderboard` after BEAM
+Track A completion (Sprint 13 end). Use Qwen3.6-35B + waggle scaffold, no hive-mind
+(isolate scaffold contribution). If scaffold submission scores ≥ 35%, add a second
+submission with hive-mind to test the memory lift.
+
+**No budget cap required** — Terminal-Bench evaluations run on their infrastructure.
+
+---
+
+## 6. Amended execution timeline
+
+```
+Sprint 13 (now)
+├── Track A: BEAM Phase 1
+│   ├── build-beam-canonical.ts (B1)
+│   ├── cells-beam.ts (B2)
+│   ├── judge-beam.ts (B3)
+│   └── N≈300 × 4 cells execution (Qwen3.6 + Opus 4.x)
+│
+├── Track D: Terminal-Bench submission
+│   └── Submit waggle scaffold (async, no gate)
+│
+Sprint 14
+├── Track B: GAIA 2 Phase 1 (after SIGALRM fix + BEAM CR result validates gate)
+│   ├── SIGALRM resolution (WSL2)
+│   ├── ARE adapter (A1–A5)
+│   └── N=500 × 4 cells execution
+│
+├── PM-RATIFY-V8C decision: proceed with LME-V2?
+│
+Sprint 15 (conditional)
+└── Track C: LongMemEval-V2 (gated on PM-RATIFY-V8C)
+    ├── Trajectory adapter (C1–C3)
+    └── N=451 × 4 cells execution
+```
+
+---
+
+## 7. Cross-benchmark claim architecture
+
+The four tracks build a layered argument:
+
+```
+Layer 1 (DONE)     LoCoMo v5         → substrate > model on factoid recall
+                                       Qwen 73.4% ≈ Opus 73.1% with hive-mind
+
+Layer 2 (BEAM)     BEAM 128K CR/KU   → I/P/B B-frames solve unsolved CR category
+                   [Track A]          → first published system to address the open problem
+
+Layer 3 (GAIA 2)   ARE Ambiguity +   → I/P/B lift carries into stateful write-action environment
+                   Adaptability       → substrate > model on interactive agent tasks
+                   [Track B]
+
+Layer 4 (LME-V2)   Web agent         → Dynamic State + Gotcha categories: P/B frames
+                   trajectories       → track agent state and recurring failures
+                   [Track C]
+
+Positioning        Terminal-Bench    → waggle scaffold competitiveness floor
+                   [Track D]
+```
+
+The claim stacks: each layer adds a new evaluation surface while reusing the same
+architectural claim. The B-frame contradiction gate is the single mechanism tested
+across BEAM (QA), GAIA 2 (write-action), and LME-V2 (trajectory).
+
+---
+
+## 8. Amended gates
+
+### Gate A-P+ (Track A Phase 1 kick)
+**Pre-kick checks:**
+- BEAM dataset download complete and SHA-256 recorded
+- `build-beam-canonical.ts` output validated (N instances ≥ 1,800 for 128K tier)
+- `cells-beam.ts` dry-run passes (all 4 cells, 5-instance smoke)
+- LiteLLM config includes Qwen3.6 DashScope + Llama-3.3-70B judge aliases
+- hive-mind `:memory:` substrate liveness confirmed
+- Budget envelope confirmed ($50 hard halt)
+
+**Action:** kick N≈300 × 4 cells.
+
+### Gate A-D (Track A post-run)
+**Action:** CC writes `benchmarks/beam/results/v8a-gate-d-exit-report.md`.
+**PM decides:**
+1. Is the CR lift ≥ 8pp? If yes → proceed to GAIA 2 (B) and LME-V2 gate
+2. Is the contradiction gate working? If no → redesign gate before GAIA 2
+3. Terminal-Bench submission: submit waggle scaffold result
+
+### Gate V8C-PM (Track C authorization)
+Separate PM ratification required before LME-V2 build starts.
+
+---
+
+## 9. Budget summary
+
+| Track | Hard halt | Expected burn |
+|---|---|---|
+| A — BEAM | $50 | $26–37 |
+| B — GAIA 2 | $80 | $41–73 (per v8.0.0) |
+| C — LME-V2 | $65 | $35–55 (provisional) |
+| D — Terminal-Bench | $0 (external infra) | $0 |
+| **Programme total** | **$195** | **$102–165** |
+
+---
+
+## 10. Scope boundaries (amended)
+
+### Added at Gate A-D:
+- BEAM CR + KU lift magnitude + significance (pre-registered cells + models, 128K tier)
+- BEAM negative control result (IE, EO, SUM)
+- Contradiction gate firing rate on BEAM conversations
+
+### Added at Gate D (GAIA 2, per v8.0.0):
+- Per v8.0.0 §12 (unchanged)
+
+### Added at Gate C-D (LME-V2, conditional):
+- Dynamic State Tracking + Environment Gotchas lift on LME-V2-Small
+- Trajectory I-frame ingestion lift vs. no-memory baseline
+
+### NOT claimable from Track D:
+- Terminal-Bench results are NOT a preregistered claim; they are positioning data
+
+### Cannot claim from any track:
+- "Waggle OS beats [model X] on all benchmarks" — each track has specific splits and conditions
+- Generalization beyond the preregistered cells and models
+
+---
+
+## 11. Related artefacts
+
+- **v8.0.0 anchor:** `benchmarks/preregistration/manifest-v8-gaia2-preregistration.md`
+  (commit SHA `a3ae4cada43c` / `51b6bcc9c39c`)
+- **BEAM repo:** https://github.com/mohammadtavakoli78/BEAM (ICLR 2026)
+- **LME-V2 paper:** https://arxiv.org/html/2605.12493v1 (May 2026)
+- **Terminal-Bench 2.0 leaderboard:** https://www.tbench.ai/leaderboard/terminal-bench/2.0
+- **`little-coder` baseline entries:** #118 (24.6% ± 3.2) and #123 (23.0%), submitted 2026-05-14
+- **hive-mind LoCoMo v5 results:** `hive-mind/benchmarks/locomo/RESULTS.md`
+
+---
+
+_End of Manifest v8.1 amendment. v8.0.0 GAIA 2 methodology is unchanged._
+_v8.1 governs Track A (BEAM) and Track C (LME-V2) preregistration._
+_Track D (Terminal-Bench) is positioning data, not governed by preregistration policy._