waggle-os/benchmarks/preregistration/manifest-v8-gaia2-preregistration.md

# Manifest v8 — GAIA 2 Ambiguity+Adaptability Ablation Pre-Registration

**Manifest version:** v8.0.0-preregistration
**Manifest type:** `gaia2_ambiguity_adaptability_hive_mind_ablation`
**Preregistered date:** 2026-06-04
**Authority:** PM (Marko Marković) — v8 emission under full PM authority. Extends waggle-os benchmark programme to the ARE/GAIA 2 evaluation environment. Does not supersede v6/v7 (LoCoMo / GEPA lineage continues independently). v8 governs all GAIA 2 work from this pre-registration forward.
**Branch:** `feature/gaia2-hive-mind-ablation` (to be created at code-freeze)
**Machine-readable twin:** [`manifest-v8-gaia2-preregistration.yaml`](manifest-v8-gaia2-preregistration.yaml)

---

## 0. Status

**PRE-REGISTERED — PENDING PM RATIFICATION FOR PHASE 1 (ENVIRONMENT UNBLOCK + ADAPTER BUILD).**

Any change to §1–§9 of this document after the anchor commit invalidates the pre-registration and requires a new PM-ratified manifest (v9+). The anchor commit SHA and SHA-256 of both MD and YAML files are recorded in the commit message body.

**Anchor commit SHA:** recorded in the git commit that adds these files.
**Manifest SHA-256 (MD + YAML bytes):** `sha256sum benchmarks/preregistration/manifest-v8-gaia2-preregistration.{md,yaml}` — recorded in the commit message body.

---

## 0.1. Motivation and lineage

This pre-registration extends the waggle-os benchmark programme from its LoCoMo memory-recall focus (v2–v7) to the **GAIA 2 ARE** (Agent Runtime Environments) evaluation environment. The scientific motivation is a direct continuation of the hive-mind LoCoMo finding:

> **Substrate ≫ subject model**: Opus 4.7 and Qwen3.6-35B converge to 73.1% / 73.4% on identical retrieval substrate (hive-mind LoCoMo v5, N=320).

The LoCoMo benchmark tests single-turn factoid recall from conversational memory. The central open question is whether the **I/P/B frame architecture** (Intra, Predicted, Bidirectional frames in `@waggle/core::FrameStore`) provides incremental lift on tasks that require:

1. **Detecting and tracking conflicting instructions** across turns (GAIA 2 Ambiguity split)
2. **Replanning in response to environment changes** that contradict earlier agent decisions (GAIA 2 Adaptability split)

These two splits were selected because they map directly onto the semantic purpose of I/P/B frames — specifically:
- **P-frames (Predicted):** record agent hypotheses and planned actions before execution
- **B-frames (Bidirectional):** record corrections and resolved contradictions
- The Ambiguity split requires an agent to detect that two instructions conflict before acting
- The Adaptability split requires an agent to update an earlier committed plan when new information arrives

The ARE SIGALRM blocker (`module 'signal' has no attribute 'SIGALRM'` — smoke run `smoke-c2-2026-04-30`) must be resolved as a **Phase 1 prerequisite** before any benchmark execution.

---

## 0.2. Relationship to prior manifests

| Manifest | Dataset | Claim | Status |
|---|---|---|---|
| v2–v5 | LoCoMo | Memory lift proof (retrieval cell) | Completed |
| v6 | LoCoMo | Judge ensemble swap (MiniMax) | Completed |
| v7 | LoCoMo (GEPA) | Prompt-shape evolution | Active |
| **v8 (this)** | **GAIA 2 ARE** | **I/P/B frame lift on Ambiguity + Adaptability** | **Pre-registered** |

v8 does NOT inherit v6/v7 code freeze or judge ensemble choices. v8 is a new benchmark programme on a new evaluation framework. All sections are v8-native.

---

## 1. Primary hypothesis (directional, confirmatory)

> **I/P/B-augmented hive-mind substrate improves agent task success on GAIA 2 Ambiguity and Adaptability splits vs. a substrate-free baseline, by a margin of ≥ 10 percentage points.**
>
> `score(hive_mind_ipb) − score(no_memory_baseline) ≥ 10pp`
>
> evaluated at **Fisher exact one-sided** p-value **< 0.10** on the pooled Ambiguity + Adaptability scenario set.

**One-sided justification:** theory-driven directional claim. The I/P/B frame architecture was explicitly designed to track conflicting and corrective information. Both selected splits require this capability. No prior GAIA 2 / hive-mind result exists; direction is grounded in hive-mind LoCoMo monotonicity chain (no-memory 0.0% → retrieval 35% → agentic 40% → oracle 55% at N=20, Stage 2 Gate C) and the semantic alignment of I/P/B frames with Ambiguity + Adaptability task structure.

**Threshold rationale:** 10pp chosen over the LoCoMo-standard 5pp because:
- GAIA 2 write-action scoring is harder (argument-level exact + soft checks vs. string match)
- I/P/B frames are a structural advantage, not a marginal one, on contradiction-requiring tasks
- A weaker effect at <10pp would be scientifically interesting but would not support a strong architectural claim
- Power: 10pp effect size at N=100 per cell gives ~70% power at α=0.10 (Wilson-based estimate); adequate for a pilot

**Failure mode:** If primary fails (Δ < 10pp or p ≥ 0.10):
- Report full cell distribution
- PM adjudication on whether to run a larger N or revise the architectural integration
- Do NOT claim negative result without replication; pilot N=100 may underpower marginal effects

---

## 2. Secondary endpoints (ex-ante, non-blocking on primary)

| # | Endpoint | Direction | Threshold | Test |
|---|---|---|---|---|
| S1 | `no_memory_baseline ≤ hive_mind_retrieval_only` | positive | ≥ 0pp | Fisher one-sided p < 0.20 |
| S2 | `hive_mind_retrieval_only ≤ hive_mind_ipb` | positive | ≥ 0pp | Fisher one-sided p < 0.20 |
| S3 | Ambiguity split: `hive_mind_ipb − no_memory_baseline` | descriptive | ≥ 0pp | point estimate + Wilson 95% CI |
| S4 | Adaptability split: `hive_mind_ipb − no_memory_baseline` | descriptive | ≥ 0pp | point estimate + Wilson 95% CI |
| S5 | Temporal split: `hive_mind_ipb − no_memory_baseline` | descriptive (null expected) | n/a | point estimate (negative control) |
| S6 | Cost-efficiency: `hive_mind_ipb` cost per solved scenario vs. no-memory | descriptive | n/a | median + IQR in USD |
| S7 | `subject_model_strong` (Opus 4.x) vs `subject_model_cheap` (Qwen3.6-35B) on `hive_mind_ipb` cell | descriptive | n/a | Δpp + Wilson 95% CI |

**S5 Negative control rationale:** the Temporal split is the hardest GAIA 2 split (all models), with failures driven by wall-clock latency, not memory architecture. hive-mind should not help here. If S5 shows large positive lift, it signals a confound in the experimental design.

Multi-comparisons policy: secondary endpoints are descriptive, no correction required. Primary hypothesis test is the only confirmatory statistical test.

---

## 3. Sample design

- **Cells:** four, run sequentially (concurrency = 1 per cell; within-cell parallelism TBD at Phase 2).

  | Cell | Description |
  |---|---|
  | `no_memory_baseline` | Standard ARE ReAct loop, no memory injection, no hive-mind substrate |
  | `hive_mind_retrieval` | ARE ReAct loop + hive-mind `@waggle/core::HybridSearch` passive recall (no I/P/B frame writes) |
  | `hive_mind_ipb` | ARE ReAct loop + full hive-mind substrate (retrieval + I/P/B frame detection + contradiction gate) |
  | `hive_mind_ipb_strong` | Same as `hive_mind_ipb` with stronger subject model (Opus 4.x vs Qwen3.6) |

- **Primary splits:** Ambiguity + Adaptability (pooled for primary hypothesis test; per-split for S3/S4)
- **Negative control split:** Temporal (S5)
- **N per cell per split:** 50 scenarios × 2 splits = 100 scenarios per cell
- **Total scenario executions:** 400 (4 cells × 100 scenarios)
- **Instance selection seed:** `42`
- **Matched design:** same 100 scenarios (50 Ambiguity + 50 Adaptability) flow through all four cells
- **Negative control:** same 50 Temporal scenarios flow through `no_memory_baseline` and `hive_mind_ipb` only (100 additional executions)
- **Total executions including negative control:** 500
- **Concurrency (within cell):** `1` pending Phase 1 environment characterization; may be raised to `≤4` at Phase 2 gate if ARE rate limits permit, under PM authorization
- **N justification:** N=50 per split per cell is the minimum for the 10pp threshold at α=0.10 with ~70% power. A larger N=100 per split per cell is preferred and reserved for PM authorization at Phase 2 if Phase 1 pilot (N=50 per split) is inconclusive.

---

## 4. Dataset — ARE/GAIA 2 scenarios

- **Source:** Meta AI ARE framework (arXiv:2509.17158), open-source MIT license, dataset CC BY 4.0
- **Repository:** `https://github.com/facebookresearch/agentbenchmark` (canonical ARE repo — to be confirmed at Phase 1)
- **Version:** ARE public release (800 scenarios across 10 universes), as of Phase 1 clone date
- **Selected splits:** `ambiguity`, `adaptability`, `temporal` (negative control)
- **Scenario selection:** deterministic seed-42 shuffle within each split, take-first-N
- **Scenario count per split:**
  - Ambiguity: 50 (pilot); expand to 100 at Phase 2 gate if available
  - Adaptability: 50 (pilot); expand to 100 at Phase 2 gate if available
  - Temporal (negative control): 50 (no expansion planned)
- **Dataset integrity:** SHA-256 of the scenario JSON files recorded at Phase 1 clone time in the run manifest
- **No modifications** to scenario content, oracle traces, or verifier DAGs. ARE scenarios used as-is from the public release.

---

## 5. Model stack

### 5.1 Subject models

| Priority | Cell | Model alias | Provider | Thinking | Notes |
|---|---|---|---|---|---|
| primary | `no_memory_baseline`, `hive_mind_retrieval`, `hive_mind_ipb` | `qwen3.6-35b-a3b-via-dashscope-direct` | Alibaba DashScope-intl | on | Inherits v6 primary subject |
| fallback | above cells | `qwen3.6-35b-a3b-via-openrouter` | OpenRouter | on | Inherits v6 fallback |
| primary | `hive_mind_ipb_strong` | `claude-opus-4-x` | Anthropic direct | n/a | Exact alias pinned at Phase 1 |
| fallback | `hive_mind_ipb_strong` | `claude-sonnet-4-x` | Anthropic direct | n/a | Fallback if Opus rate-limits |

Subject model pricing (Qwen3.6-35B): $0.20 / $0.80 per M in/out (inherits v6).
Subject model pricing (Claude Opus 4.x): to be recorded at Phase 1 from Anthropic pricing page.

### 5.2 Judge protocol — ARE verifier (primary) + LLM soft judge (secondary)

GAIA 2 scoring uses a **two-layer judge**:

**Layer 1 — ARE verifier (deterministic):**
- Hard checks: argument-level exact match for deterministic fields (dates, names, IDs)
- Causality and relative-time constraint validation
- Implemented in the ARE scenario DAG verifier; no LLM call
- Score: binary pass/fail per hard-check field

**Layer 2 — LLM soft judge (for open-ended fields):**
- Per ARE paper (arXiv:2509.17158): `Llama 3.3 Instruct 70B` used in paper baseline
- **v8 policy:** replicate ARE paper's judge (`llama-3.3-70b-instruct-via-openrouter`) as Layer 2 primary for reproducibility with the leaderboard baseline
- **v8 secondary judge:** `claude-opus-4-x` (Anthropic) for cross-validation of open-ended verdicts on a 20% random sample
- Temperature: 0 for both judges
- κ calibration: compute pairwise Cohen's κ (ARE-Llama vs. Opus) on the 20% sample at Phase 1 exit. Threshold: κ ≥ 0.65 PASS / κ < 0.60 HALT.

**Score aggregation:** final scenario score = arithmetic mean of (all hard-check pass rates + soft-judge pass rates) per ARE paper §4. Reported as a decimal in [0, 1].

**v8 does NOT use the waggle-os LoCoMo judge ensemble (Opus + GPT + MiniMax).** That ensemble was designed for factoid string-match accuracy. GAIA 2 scenarios require structural verifier checks that cannot be delegated to a general-purpose judge.

### 5.3 Health-check predicate

Pre-cell health check required before each cell execution:
- ARE environment boot: `python -c "from agents_benchmark import ARE; ARE().ping()"` — must succeed
- LiteLLM liveness: `/v1/chat/completions` ping on all active subject aliases
- LLM soft judge: ping on `llama-3.3-70b-instruct-via-openrouter`
- Ollama embedder (for `hive_mind_retrieval` + `hive_mind_ipb` cells): `curl http://localhost:11434/api/embeddings` with `nomic-embed-text`

Failure on any probe → halt before cell, PM raise.

---

## 6. Substrate

### 6.1 hive-mind substrate (`@waggle/core`)

Inherited from hive-mind v5 architecture (`hive-mind` repo, `feature/v5-distilled-dense` or equivalent frozen branch — to be confirmed at Phase 1).

**Components:**
- `MindDB` (SQLite + sqlite-vec) — in-memory per scenario (`:memory:` path), ephemeral per run
- `FrameStore` — I/P/B-frame CRUD + FTS5 auto-index
- `HybridSearch` — RRF-fused FTS5 + vec0 search
- `SessionStore` — per-scenario session lifecycle
- `createOllamaEmbedder()` — `nomic-embed-text`, 1024 dims, local, $0

**Cell-specific substrate config:**

| Cell | `MindDB` | `FrameStore` | Frame types written | `HybridSearch` |
|---|---|---|---|---|
| `no_memory_baseline` | none | none | none | none |
| `hive_mind_retrieval` | `:memory:` | active (read-only after ingest) | I-frames only (past turns ingested) | conv-scoped, top-K=20 |
| `hive_mind_ipb` | `:memory:` | active (read+write during run) | I + P + B frames | conv-scoped, top-K=20 |
| `hive_mind_ipb_strong` | same as `hive_mind_ipb` | same | same | same |

### 6.2 I/P/B frame integration contract for GAIA 2

This is the **core novel integration** that does not exist in the current codebase and must be built during Phase 1. The integration contract is pre-specified here to prevent methodology drift:

**Intra-frames (I-frames):** written after each user message and each environment event notification is received. Content = the raw turn text. Standard frame, equivalent to what LoCoMo ingest uses.

**Predicted-frames (P-frames):** written **before** each write-action tool call. Content = the agent's stated intent and predicted post-state (extracted from the agent's reasoning trace). Purpose: create a retrievable record of what the agent expected to happen, enabling contradiction detection when the environment responds differently.

**Bidirectional-frames (B-frames):** written when the agent detects a contradiction between:
- A new user message and a previously recorded P-frame (instruction conflict → Ambiguity)
- An environment event and a previously recorded P-frame (plan invalidated → Adaptability)

B-frames store: the conflicting pair (new content + prior P-frame ID), the resolved intent (if any), and a `conflict_type` label (`instruction_conflict` or `plan_invalidation`).

**Contradiction gate:** a lightweight check (single LLM call or heuristic rule, TBD at Phase 1 code-freeze) that fires before each write-action tool call when there are ≥ 1 P-frames in the substrate. If triggered, the gate:
1. Retrieves the top-3 P-frames by semantic similarity to the current action
2. Asks the subject model (or a cheap dedicated gate model, TBD): "Does the proposed action contradict any of these prior plans?"
3. If YES → writes a B-frame and injects a `# Contradiction Warning` block into the agent's context before the write-action call
4. If NO → proceeds without injection

The contradiction gate is the primary mechanism by which `hive_mind_ipb` is expected to outperform `hive_mind_retrieval` on the Ambiguity and Adaptability splits.

**Gate model (contradiction check):** default = same subject model as the outer agent loop (Qwen3.6-35B). Alternative: a lightweight dedicated model (e.g., `qwen3.6-8b` via OpenRouter) to reduce latency and cost. Decision deferred to Phase 1 gate after latency profiling.

### 6.3 Substrate lifecycle per scenario

1. `createSubstrate({dbPath: ':memory:'})` at scenario start
2. Ingest prior turns (if scenario has a conversation history preamble) as I-frames
3. Agent loop runs; P/B frames written dynamically during execution
4. Scenario exits (success or failure per ARE verifier)
5. `substrate.close()` — `:memory:` freed; no persistence between scenarios
6. ARE verifier evaluates the recorded action trace independently of the substrate

Substrate state does NOT affect the ARE verifier. The verifier scores the agent's actual tool calls in the environment, not the memory contents.

---

## 7. Agent architecture — ARE ReAct loop with hive-mind hooks

**Base architecture:** ARE paper's standard uniform ReAct loop (temperature=0.5, max generation=16K tokens per ARE §4.2). This is the architecture all GAIA 2 baseline models use; it is adopted for comparability.

**Hive-mind hooks (new for v8):**

The `no_memory_baseline` cell uses the ARE loop unmodified.

The `hive_mind_*` cells wrap the ARE loop with three hooks:
1. **Pre-turn hook:** `search_memory(query=current_user_message, topK=20, scopeToSession=true)` → inject `# Recalled Memories` block into system context (as in LoCoMo agentic cell). Skipped on turn 1 if no prior I-frames exist.
2. **Pre-write-action hook:** contradiction gate (§6.2) — fires before any `write`-type ARE tool call
3. **Post-turn hook:** write I-frame (always) + write P-frame if the turn contained a write-action intent

**Tool allowlist:** all 101 ARE tools remain available. No tool is blocked. The hive-mind substrate is additive, not restrictive.

**ARE environment:** Mobile universe (smartphone mock-up with Email, Calendar, Contacts, Shopping, FileSystem) unless the Ambiguity/Adaptability split scenarios span multiple universes — to be confirmed at Phase 1 dataset inspection. If multiple universes are present, universe distribution is reported in run metadata.

**SIGALRM blocker fix (prerequisite for Phase 1 execution):**

The smoke run `smoke-c2-2026-04-30` failed on all 3 runs with:
```
AttributeError: module 'signal' has no attribute 'SIGALRM'
```
This is a Windows/non-Unix signal error. Resolution options (in priority order):
1. **WSL2 Linux environment** — run ARE inside WSL2 where `signal.SIGALRM` is available (zero code change, recommended)
2. **Docker container** — run `benchmarks/harness` + ARE inside a Linux container (Docker Desktop on Windows)
3. **SIGALRM shim** — patch ARE's timeout mechanism to use `asyncio.wait_for` instead of `signal.SIGALRM` (code change to ARE; requires upstreaming or local fork maintenance)

Option 1 (WSL2) is the pre-registered preferred path. If WSL2 is unavailable, Option 2 (Docker) is the fallback. Option 3 is a last resort requiring PM approval as a scope-deviation.

Phase 1 does NOT begin until the SIGALRM blocker is resolved and a clean smoke run (`0 exceptions, ≥ 1 scenario scored ≥ 0.5`) is recorded.

---

## 8. Stopping rules

| # | Rule | Trigger | Action |
|---|---|---|---|
| §8.1 | Budget hard halt | Cumulative spend ≥ **$80.00** | Halt, persist partial JSONL, PM raise |
| §8.2 | Scenario exception rate | > 20% of scenarios in a cell raise Python exceptions | Halt cell, PM raise before next cell |
| §8.3 | ARE environment health | ARE ping fails on pre-cell health check | Halt before cell |
| §8.4 | Subject fetch failures | 5 consecutive LiteLLM 5xx / fetch errors | Halt, persist partial |
| §8.5 | κ failure (Phase 1 exit) | κ(ARE-Llama, Opus) < 0.60 on 20% calibration sample | Halt, PM raise — Phase 2 requires judge renegotiation |
| §8.6 | Deviation from §1–§9 | Any change detected during run | Immediate halt + PM raise + re-pre-registration required (v9+) |
| §8.7 | SIGALRM unresolved | Phase 1 smoke run still fails after resolution attempt | Halt, report blocker to PM, await environment decision |

**No interim looks at primary hypothesis during execution.** Budget monitoring is continuous (non-statistical). Halts are operational, not inferential.

---

## 9. Post-hoc exclusion policy: NONE

All 500 scenario executions (400 primary + 100 negative control) enter the denominator. `verifier_error` and `environment_crash` scenarios are counted in denominator and reported as `execution_loss` separately. No scenario whitelist/blacklist. Selective exclusion forbidden ex-ante.

Exception: if the ARE environment produces a scenario with a known bug acknowledged in the ARE issue tracker, PM may authorize exclusion of that specific scenario ID with audit trail. This requires a new decision document (not a manifest amendment).

---

## 10. Deviation policy

Any deviation from §1–§9 during run → (1) immediate halt, (2) PM raise, (3) re-pre-registration (manifest v9+) if accepted.

**Permitted non-deviations (do not require v9):**
- SIGALRM fix choice between Option 1 (WSL2) and Option 2 (Docker), per §7
- Gate model selection for contradiction check (subject model vs. dedicated lightweight), per §6.2
- Minor litellm-config.yaml amendments (new alias additions only, no semantic changes to existing aliases)
- κ calibration sample size adjustment ±10% due to split availability (if < 50 scenarios exist in a split at Phase 1, N adjusts to available count — primary hypothesis threshold adjusts proportionally)

---

## 11. Code freeze

Code is frozen at the commit that adds this manifest to the repository. The following paths are frozen for the duration of v8 execution:

**Frozen paths (substrate and harness):**
- `packages/core/src/mind/search.ts` (HybridSearch)
- `packages/core/src/mind/frames.ts` (FrameStore, I/P/B frame types)
- `packages/core/src/mind/sessions.ts` (SessionStore)
- `packages/core/src/mind/db.ts` (MindDB)
- `packages/agent/src/agent-loop.ts` (runAgentLoop)
- `benchmarks/harness/src/substrate.ts`

**Frozen paths (v8-specific, to be created at Phase 1 code-freeze):**
- `benchmarks/gaia2/src/adapter.ts` — ARE scenario ↔ harness schema adapter
- `benchmarks/gaia2/src/cells-gaia2.ts` — four v8 cell implementations
- `benchmarks/gaia2/src/contradiction-gate.ts` — I/P/B contradiction gate
- `benchmarks/gaia2/src/runner-gaia2.ts` — v8 runner (wraps ARE loop)
- `benchmarks/gaia2/src/judge-gaia2.ts` — ARE verifier + Llama soft-judge integration

These paths are frozen at Phase 1 code-freeze commit (separate from this pre-registration commit). Any post-freeze modification to the above paths triggers §8.6 (halt + PM raise).

**Permitted delta during run:** new JSONL files emitted to `benchmarks/gaia2/runs/` and `benchmarks/gaia2/results/`.

---

## 12. Scope boundaries

### Can claim at Gate D (post-run):
- Magnitude + significance of I/P/B hive-mind lift on GAIA 2 Ambiguity + Adaptability splits under the pre-registered cell stack and subject models
- Per-cell, per-split scenario success rates with Wilson 95% CIs
- Negative control result (Temporal split)
- Contradiction gate firing rate and per-firing outcome (resolved vs. unresolved)
- Cost-per-solved-scenario across cells
- Substrate-is-the-moat finding (if S7 shows Opus ≈ Qwen on `hive_mind_ipb`, replicating LoCoMo finding in GAIA 2 context)

### Cannot claim at Gate D:
- General GAIA 2 leaderboard rank (v8 tests 2 of 7 splits; full leaderboard requires all splits)
- Multi-model generalization beyond Qwen3.6-35B + Opus 4.x
- Production agent performance on real mobile environments
- Direct comparison to GAIA 2 paper baseline models (different scaffold; comparisons are indicative only)

### Reserved for PM:
- Public claim phrasing and venue
- Full 7-split GAIA 2 leaderboard submission (requires v9 pre-registration)
- Publication timing and co-author decisions
- "beats Opus 4.x without hive-mind" framing (requires v9 + explicit cross-model ablation design)

---

## 13. PM gates

### Gate P+ (Phase 1: environment unblock + adapter build)

**Trigger:** SIGALRM fix verified + clean smoke run + Phase 1 code-freeze commit

**Pre-kick checks required:**
- SIGALRM resolution option confirmed (WSL2 or Docker)
- Clean smoke run: `≥ 1 scenario scored ≥ 0.5`, `0 signal.SIGALRM exceptions`
- All frozen paths (§11 v8-specific) committed at Phase 1 code-freeze
- κ calibration plan confirmed (20% sample, Llama + Opus)
- LiteLLM config includes all v8 subject model aliases
- Ollama `nomic-embed-text` liveness confirmed in execution environment

**Action:** CC (agent) halts; awaits `PM-RATIFY-V8-PHASE1` before Phase 2 execution.

### Gate P++ (Phase 2: N=500 execution kick)

**Trigger:** `PM-RATIFY-V8-PHASE1` received after Gate P+ ratification.

**Pre-kick checks required:**
- Phase 1 exit report at `benchmarks/gaia2/preregistration/phase1-exit-report.md`
- κ(ARE-Llama, Opus) ≥ 0.65 on calibration sample
- Budget envelope confirmed ($80 hard halt)
- Cell order confirmed (no_memory_baseline → hive_mind_retrieval → hive_mind_ipb → hive_mind_ipb_strong)

**Action:** kick N=500 execution via CLI invocation template (§14).

### Gate D (post-run, pre-claim)

**Trigger:** N=500 run exit (clean or halted per §8)

**Action:** CC writes Gate D exit report at `benchmarks/gaia2/results/v8-gate-d-exit-report.md`. Halts. PM decides claim composition, venue, and whether to proceed to full 7-split leaderboard submission (v9).

**No self-advance at any gate.**

---

## 14. Budget

- **Hard halt:** $80.00
- **Cap:** $90.00
- **Expected burn:** ~$55–$70

| Component | Est. cost |
|---|---|
| Phase 1: smoke run validation, adapter testing, κ calibration | ~$5–10 |
| N=500 executions — Qwen3.6-35B subject (3 cells × 100 scenarios × ~8K avg tokens) | ~$10–15 |
| N=100 executions — Opus 4.x subject (1 cell × 100 scenarios) | ~$20–30 |
| ARE Llama soft judge (N=500 × soft-check fields, est. avg 2 soft fields/scenario) | ~$5–8 |
| Opus soft-judge cross-validation (20% sample = 100 scenarios) | ~$3–5 |
| Contradiction gate calls (fired on est. 30% of Ambiguity/Adaptability turns) | ~$3–5 |
| Ollama embedding (local, $0) | $0 |
| **Total expected** | **~$41–73** |

Wall-clock estimate: Phase 1 ≤ 1 day; Phase 2 N=500 ≈ 4–8 hours (ARE scenarios have longer execution traces than LoCoMo turns; temporal scenarios have real wait periods that may be simulated or skipped).

---

## 15. Related artefacts

### v8-specific
- **v8 anchor commit:** THIS COMMIT
- **Phase 1 smoke run (failed):** `benchmarks/gaia2/runs/smoke-c2-2026-04-30/` (SIGALRM blocker — audit trail)
- **ARE paper:** Froger et al., arXiv:2509.17158, Sept 2025
- **GAIA 2 comparison report:** `docs/gaia-comparison.md` (research background, 2026-06-04)

### Prior waggle-os manifests (independent lineage)
- v6 LoCoMo Stage 3 anchor: `fc169250c3c27cd3`
- v7 GEPA Faza 1 anchor: see manifest-v7-gepa-faza1.yaml
- Bench-Spec LOCK v1: `PM-Waggle-OS/decisions/2026-04-22-bench-spec-locked.manifest.yaml`

### hive-mind lineage
- LoCoMo v5 result (73.1% Opus, 73.4% Qwen): `hive-mind/benchmarks/locomo/RESULTS.md`
- Trio-strict re-judge (67.8% AND-of-3): `hive-mind/data/judgments/trio-judgments-v5-retrieval.v2.jsonl`
- METHODOLOGY.md: `hive-mind/benchmarks/METHODOLOGY.md`

---

## Appendix A — Integration work required before Phase 1 code-freeze

This appendix documents the build tasks needed. It is informational; it is NOT part of the pre-registered methodology (§1–§9 are the invariant sections).

| Task | File to create | Description |
|---|---|---|
| A1 | `benchmarks/gaia2/src/adapter.ts` | Map ARE scenario JSON schema → harness `DatasetInstance` equivalent; extract split label, oracle trace, universe ID |
| A2 | `benchmarks/gaia2/src/cells-gaia2.ts` | Four cell implementations wrapping ARE loop with hive-mind hooks (§7) |
| A3 | `benchmarks/gaia2/src/contradiction-gate.ts` | P-frame retrieval + conflict check + B-frame write + context injection (§6.2) |
| A4 | `benchmarks/gaia2/src/runner-gaia2.ts` | Scenario runner: load ARE env, run cell, call ARE verifier, emit JSONL |
| A5 | `benchmarks/gaia2/src/judge-gaia2.ts` | ARE verifier wrapper + Llama 3.3 70B soft-judge + Opus cross-validation |
| A6 | SIGALRM fix | WSL2 or Docker environment setup (§7); clean smoke run gate |
| A7 | `litellm-config.yaml` amendment | Add `llama-3.3-70b-instruct-via-openrouter` alias + Opus 4.x alias (if not already present) |

---

_End of Manifest v8 pre-registration. This document is the anchor for all analysis choices at GAIA 2 Gate D exit. Prior waggle-os manifests (v2–v7) remain audit-immutable and govern the LoCoMo/GEPA programme independently._