111 lines
4.4 KiB
Markdown
111 lines
4.4 KiB
Markdown
# Manifest v8.2 — Final Scope Amendment (supersedes v8.0 and v8.1)
|
||
|
||
**Manifest version:** v8.2.0-preregistration (final scope)
|
||
**Preregistered date:** 2026-06-04
|
||
**Authority:** PM (Marko Marković)
|
||
**Supersedes:** manifest-v8-gaia2-preregistration.md (v8.0.0) and
|
||
manifest-v8.1-multi-benchmark.md (v8.1.0) for scope section only.
|
||
All GAIA 2 methodology in v8.0.0 §1–§9 remains intact and unmodified.
|
||
|
||
---
|
||
|
||
## 0. Amendment rationale
|
||
|
||
v8.1.0 added BEAM (Track A), LongMemEval-V2 (Track C), and Terminal-Bench 2.0 (Track D).
|
||
This amendment makes two scope changes:
|
||
|
||
1. **LongMemEval-V2 dropped** — deferred to v9. No public runner, 25 M+ token haystacks,
|
||
and web-agent trajectory tooling are all pre-stable. Attempting V2 in v8 would block
|
||
the release milestone without scientific payoff. The V1 benchmark is fully runnable
|
||
today (harness already recognises `longmemeval`), and its `knowledge-update` split
|
||
directly tests the same substrate claim.
|
||
|
||
2. **LongMemEval V1 added as Track A0** — becomes the fastest path to a new data point.
|
||
500 questions, S-variant (115 K tokens/question), zero new harness work required.
|
||
Hard budget cap $20. Runs before BEAM.
|
||
|
||
---
|
||
|
||
## 1. Final four-track scope
|
||
|
||
| Track | Benchmark | Primary claim surface | Status | Budget hard halt |
|
||
|---|---|---|---|---|
|
||
| **A0** | **LongMemEval V1** | Conversational memory — `knowledge-update` split | Runnable now | **$20** |
|
||
| **A** | **BEAM** | Contradiction resolution (unsolved category) | Runnable now | **$50** |
|
||
| **B** | **GAIA 2 / ARE** | Ambiguity + Adaptability stateful tasks | Blocked — SIGALRM fix | **$80** (per v8.0) |
|
||
| **D** | **Terminal-Bench 2.0** | Capability floor / positioning | Async submission | **$0** |
|
||
|
||
> **LME-V2 (former Track C) is out of v8 scope.** It is preregistered as a v9 track
|
||
> with no methodology commitments in this document.
|
||
|
||
---
|
||
|
||
## 2. Execution order
|
||
|
||
```
|
||
A0 (LongMemEval V1) ← first: fastest path, harness already knows 'longmemeval'
|
||
↓
|
||
A (BEAM) ← second: pip install + download, low adapter effort
|
||
↓
|
||
B (GAIA 2) ← third: blocked on SIGALRM fix; starts in parallel after fix
|
||
D (Terminal-Bench) ← async: submit existing scaffold any time, not gating release
|
||
```
|
||
|
||
Rationale for A0-first ordering: the harness `longmemeval` dataset identifier is already
|
||
wired; the canonical builder (`build-longmemeval-canonical.ts`) produces a drop-in JSONL;
|
||
the S-variant is 500 questions at manageable cost. A0 provides an independent memory
|
||
result to strengthen the central claim before BEAM runs.
|
||
|
||
---
|
||
|
||
## 3. Central scientific claim (unchanged from v8.1)
|
||
|
||
> **Substrate (I/P/B hive-mind) > subject model selection.**
|
||
>
|
||
> Demonstrated across three evaluation surfaces:
|
||
> - **Conversational memory** — LongMemEval V1, `knowledge-update` split (Track A0)
|
||
> - **Contradiction resolution** — BEAM CR category (Track A), the category that remains
|
||
> unsolved for all subject models at the time of preregistration
|
||
> - **Stateful agent tasks** — GAIA 2 Ambiguity + Adaptability splits (Track B)
|
||
|
||
Terminal-Bench 2.0 (Track D) provides **positioning data only**. It is not part of the
|
||
scientific claim and will not be cited as evidence for the substrate > subject hypothesis.
|
||
|
||
---
|
||
|
||
## 4. Unchanged elements from v8.0.0
|
||
|
||
- All GAIA 2 methodology (§1–§9 of manifest-v8-gaia2-preregistration.md) is unmodified.
|
||
- Track B $80 hard budget halt is unchanged.
|
||
- Contradiction resolution as primary BEAM focus is unchanged.
|
||
- The I/P/B frame taxonomy (I = Identity, P = Procedural, B = Belief/Contradiction) is
|
||
unchanged.
|
||
|
||
---
|
||
|
||
## 5. LME-V2 deferral note (v9 placeholder)
|
||
|
||
LongMemEval-V2 (web-agent trajectory haystacks, multimodal, 25 M+ token context) is
|
||
deferred to v9. No methodology is preregistered here. A v9 amendment will cover:
|
||
trajectory Insert/Query API, multimodal evidence handling, and comparison to V1 baseline.
|
||
|
||
---
|
||
|
||
## 6. Budget summary
|
||
|
||
| Track | Hard halt | Notes |
|
||
|---|---|---|
|
||
| A0 — LongMemEval V1 | $20 | S-variant only (500 questions) |
|
||
| A — BEAM | $50 | 128 K chat-size default |
|
||
| B — GAIA 2 | $80 | Per v8.0.0 methodology |
|
||
| D — Terminal-Bench | $0 | Submit only |
|
||
| **Total exposure** | **$150** | Sequential gating limits concurrent spend |
|
||
|
||
---
|
||
|
||
## 7. Prior anchors
|
||
|
||
- `benchmarks/preregistration/manifest-v8-gaia2-preregistration.md` — v8.0.0 base
|
||
- `benchmarks/preregistration/manifest-v8.1-multi-benchmark.md` — v8.1.0 amendment
|
||
- This document (v8.2.0) supersedes both for scope; neither for methodology.
|