# Manifest v6 — Task 2.5 Stage 3 N=400 Pre-Registration (Judge Ensemble Swap) **Manifest version:** v6.0.0-preregistration **Manifest type:** `stage_3_n400_preregistration_v6_ensemble_swap` **Preregistered date:** 2026-04-24 **Authority:** PM (Marko Marković) — §2.0 + §2.1 Phase 1 authorization 2026-04-24 on closure of the full §1.3f → §1.3h-C judge swap validation sequence. Inherits §1.1 lock-semantics waiver + §1.2 RCA + §1.3 throttle chain ratifications from v5. **Branch:** `feature/c3-v3-wrapper` **Code freeze:** HEAD `373516c2784807da8536dbc0c194c54f4e4cd4be` (short `373516c`) — frozen EXCEPT the v5 §5.2 Gemini `rpm:20` addendum (retained as audit artefact) and the v6 `litellm-config.yaml` supersession amendment committed separately as Phase 1 Commit 2 (adds `minimax-m27-via-openrouter` + `kimi-k26-direct` aliases). **Supersedes:** Manifest v5 (anchor commit `fc16925`). v5 remains audit-immutable predecessor. v6 governs all Stage 3 N=400 re-kick from this commit forward. **Inherits:** Bench-Spec LOCK v1 (`decisions/2026-04-22-bench-spec-locked.manifest.yaml`). **Machine-readable twin:** [`manifest-v6-preregistration.yaml`](manifest-v6-preregistration.yaml) --- ## 0. Status **PRE-REGISTERED — PHASE 1 KICK (κ RE-CAL + CONFIG AMENDMENT). PENDING PM RATIFICATION FOR PHASE 2 (N=400 EXECUTION).** This document supersedes manifest v5 (anchor `fc16925`). v6's trigger and scope deltas are enumerated in §0.5 below. All other sections inherit v5 verbatim for tamper-evident standalone audit. Any change to v6's success criteria, statistical tests, sample design, or scope after v6's anchor commit invalidates the pre-registration and requires a new PM-ratified decision document (manifest v7+). **Anchor commit SHA:** recorded in the git commit that adds these files. **Manifest SHA-256 (MD + YAML bytes):** computed at anchor-commit time via `sha256sum benchmarks/preregistration/manifest-v6-preregistration.{md,yaml}` — recorded in the commit message body for tamper-evident audit trail. --- ## 0.5. v6 Delta Log (NEW — v6-specific) ### Trigger **§1.3f-§1.3h-C judge swap validation sequence (2026-04-24):** Starting from v5's §1.3 Gate P+ probe FAIL + §1.3c throttle probe PASS + §1.3e RPD feasibility check INFEASIBLE@250 / FEASIBLE@2500, Stage 3 N=400 remained blocked by Google quota ceiling for `gemini-3.1-pro-preview`. PM approved two parallel paths: (A) Google quota relief ticket (unresolved; indefinite wait), (B) judge swap to a non-Google flagship reasoning model. **§1.3f (anchor `8ad0567`, 2026-04-24):** Vertex AI Batch Prediction eligibility probe for `gemini-3.1-pro-preview`. Outcome: **INFEASIBLE**. Vertex v1beta does not list this preview model in the batch-eligible catalog; no publisher/model endpoint accepts batch ingestion. Branch A closed. **§1.3g (anchor `8a2f0e6`, 2026-04-24):** 4-candidate judge swap κ probe (Kimi K2.6 + MiniMax M2.7 + DeepSeek V4 Pro + Zhipu GLM-5.1) on 20 stratified instances (first-4-per-cell from the 100-row v5 κ calibration set at `benchmarks/results/locomo-mini-n20-retry-2026-04-24T00-02-12Z.jsonl`). Outcome: **MULTI_PASS** with methodological caveat — κ=1.0 across all 4 on the unanimous-biased subset (0/20 Opus-GPT splits vs full-set 7% split rate). Operational ranking (Zhipu > DeepSeek > MiniMax > Kimi on speed × parse × direct) was heuristic only, not empirical κ discrimination. **§1.3h (anchor `ae0d312`, 2026-04-24):** PM-adjudicated stratified discriminating re-probe on the 7 available Opus≠GPT split cases (PM-amended min 7 under §1.3H-POOL-SHORTAGE OPTION 1). Executed 28 calls (7 × 4 candidates) with MiniMax direct-first routing test. Outcome: **INCONCLUSIVE_BUT_OPERATIONAL_SIGNAL** — split-only κ structurally degenerate (all 7 splits Opus=correct / GPT=incorrect → reference column has no variance). Informative signal = correctness on oriented splits (agreement with verified-correct Opus reference): - MiniMax: 6/7 = **86%** (best) - Kimi: 4/5 = 80% - DeepSeek: 2/5 = 40% (mis-calibrated) - Zhipu: 0/6 = **0% — GPT-echo, DISQUALIFIED** (violates ensemble independence assumption) MiniMax direct routing failed both `api.minimaxi.com` and `api.minimax.chat` v2 endpoints (MINIMAX_GROUP_ID did not unblock); OpenRouter fallback 7/7 parse. **§1.3h-C (anchor `005a19a`, 2026-04-24):** DeepSeek `max_tokens` 1024→2048 bump verification on same 7-split sample. Outcome: **truncation_fixable_but_correctness_regressed** — parse 5/7 → 7/7 (truncation confirmed as root cause of NULLs), but correctness 40% → 14% (longer reasoning budget made DeepSeek more GPT-strict, moving further from verified-correct Opus reference). DeepSeek DISQUALIFIED on correctness grounds regardless of parse fix. ### Final ensemble selection ratified 2026-04-24 | Role | Model | Selection rationale | Routing | |------|-------|----------------------|---------| | primary_judge_1 | Claude Opus 4.7 | inherited from v5 (unchanged) | anthropic direct | | primary_judge_2 | GPT-5.4 | inherited from v5 (unchanged) | openai direct | | **primary_judge_3** | **MiniMax M2.7** | **86% correct on splits (best empirical fit), 100% parse via OR** | **openrouter (direct failed)** | | **backup_judge** | **Kimi K2.6** | **80% correct on splits, per-instance failover on primary_judge_3 failure** | **moonshot direct** | ### Disqualified candidates (audit trail) | Model | DQ reason | Evidence anchor | |-------|-----------|------------------| | Gemini 3.1 Pro Preview | Google per-model 25 RPM cap + Vertex batch INFEASIBLE | §1.3 `66dcd5a` + §1.3e `1d3851d` + §1.3f `8ad0567` | | Zhipu GLM-5.1 | 100% GPT-echo on splits (p_opus=0%, p_gpt=100%) — violates ensemble independence | §1.3h `ae0d312` | | DeepSeek V4-Pro | 14% correctness on splits at mt=2048 (regressed from 40% at mt=1024); GPT-alignment escalates with reasoning depth | §1.3h-C `005a19a` | ### Changes from v5 | # | Section | v5 | v6 | |---|---------|-----|-----| | §5.2 | Judge ensemble | Opus + GPT + Gemini-3.1-Pro-Preview primary; Grok-4.20 reserve (1/1/1 only) | **Opus + GPT + MiniMax-M2.7 primary; Kimi-K2.6 backup (per-instance failover)** | | §5.2 | Tie-break policy | majority + Grok-4.20 on 1/1/1 split | **primary 3-judge majority; backup activates per-instance on MiniMax failure; three-way 1/1/1 → PM escalation (no reserve judge in v6)** | | §5.2 | Rate-limit metadata | `rpm: 20` on `gemini-3.1-pro-preview` (v5 addendum) | **No active rpm:20 in judge path (Gemini alias retained but unused); MiniMax OR rpm + Kimi Moonshot rpm TBD at §1.3c-v6 probe time** | | §11 | Code freeze `litellm-config.yaml` | Frozen except v5 §5.2 Gemini rpm:20 addendum | **v5 freeze superseded; v6 amendment adds `minimax-m27-via-openrouter` + `kimi-k26-direct` aliases + retains all v5 entries (Gemini alias with rpm:20 kept as orphan audit artefact). Post-amendment state pinned by v6 §11.** | | §14 | Budget envelope | $30 cap / $28 halt / ~$23 expected | **$60 cap / $55 halt / ~$50 expected (Phase 1 κ re-cal ~$25 + Phase 2 N=400 ~$25; no preview-model premium)** | | §0.5 | Delta log | v4→v5 trigger from §1.3 probe FAIL + §1.3b IN_SCOPE + naming reconciliation | **v5→v6 trigger from §1.3f → §1.3h-C sequence closure; MiniMax primary + Kimi backup selection rationale; Zhipu/DeepSeek DQ; κ re-cal methodology** | | §13 | PM gates | Gate P+ (v5 pre-run) + Gate D (post-run) | **Gate P++ (v6 Phase 1: κ re-cal + config amendment) + Gate P+++ (v6 Phase 2 kick = PM-RATIFY-V6-KAPPA) + Gate D (post-run unchanged)** | ### UNCHANGED from v5 (verbatim inheritance) - **§1** primary hypothesis (Fisher one-sided p<0.10 on retrieval − no-context ≥ 5pp) - **§2** secondary endpoints (S1–S5) - **§3** sample design (concurrency=1; five cells sequential; N=400 per cell; seed=42) - **§4** dataset (LoCoMo 1531 instances, raw SHA `79fa87e9...`, canonical SHA `39e415e2...`) - **§5.1** subject route table (Qwen 3.6-35B-A3B-Thinking, DashScope-intl direct primary, OR fallback) - **§5.3** health-check predicate - **§6** substrate (HybridSearch conv-scope top-K=20, nomic-embed-text) - **§7** SYSTEM_AGENTIC verbatim bytes (SHA-256 `6facae6d...`, 1467 bytes) - **§8** stopping rules (budget + streak + pre-cell health + §1.1 waiver + deviation) - **§9** post-hoc exclusion policy NONE - **§10** deviation policy (halt + restart-required) - **§12** scope boundaries (claim/not-claim + SOTA composition reserved for PM) - **§15** related artefacts (predecessor chain extended to include v6 ancestry) ### Parent chain (extended) | Phase | Anchor | Note | |-------|--------|------| | v4 | `dedd698` | obsolete predecessor pre-reg | | §1.1 lock waiver | `67eb899` | ratified | | §1.2 RCA | `274e987` | ratified | | §1.3 probe FAIL | `66dcd5a` | preview 25 RPM discovery | | §1.3b scope audit | `69a14708` | IN_SCOPE verdict | | v5 emission | `fc16925` | manifest-v5 anchor (throttle config) | | §5.2 rpm:20 edit | `ad324cc` | v5 §11 exception | | §1.3c throttle probe PASS | `3a146ef` | empirical verification | | Fold-in 3.5b sibling mirror | `d0ab680` | defensive rpm:20 on sibling alias | | §1.3e RPD feasibility | `1d3851d` | INFEASIBLE@250, FEASIBLE@2500 | | §1.3f Vertex Batch | `8ad0567` | INFEASIBLE → Branch A closed | | §1.3g Judge swap MULTI_PASS | `8a2f0e6` | 4-candidate κ=1.0 (unanimous-biased) | | §1.3h Stratified re-probe | `ae0d312` | INCONCLUSIVE_BUT_OPERATIONAL_SIGNAL (bias exposed) | | §1.3h-C DeepSeek mt bump | `005a19a` | truncation_fixable_but_correctness_regressed | | **v6 emission** | **THIS COMMIT** | **manifest-v6 anchor** | --- ## 1. Primary hypothesis (directional, confirmatory) _Inherited verbatim from manifest v5 §1 (which inherited verbatim from v4 §1). No change._ > **Memory-lift at conv-scope retrieval exceeds zero-memory baseline.** > > `retrieval_judge_accuracy − no-context_judge_accuracy ≥ 5pp` > > evaluated at **Fisher exact one-sided** p-value **< 0.10**. **One-sided justification:** theory-driven directional claim; ex-ante scaffolding from Gate B whole-corpus-leak dry-run (8/20 other-conv leak) + Gate C monotonicity chain (no-context 0.10 < retrieval 0.35 < agentic 0.40 < oracle 0.55 at N=20). **Failure mode:** <2% probability at N=400 given Gate C's +25pp effect size (5× threshold). If primary fails despite coherent chain → PM adjudication on power-vs-signal question. --- ## 2. Secondary endpoints (ex-ante, non-blocking on primary) _Inherited verbatim from manifest v5 §2. No change._ | # | Endpoint | Direction | Threshold | Test | |---|----------|-----------|-----------|------| | S1 | no-context ≤ retrieval | positive | ≥ 0pp | Fisher one-sided p < 0.20 | | S2 | retrieval ≤ agentic | positive | ≥ 0pp | Fisher one-sided p < 0.20 | | S3 | agentic ≤ oracle-context | positive | ≥ 0pp | Fisher one-sided p < 0.20 | | S4 | agentic − retrieval | positive | ≥ 0pp | descriptive + Wilson 95% CI | | S5 | oracle-context − full-context | positive (expected) | descriptive | descriptive (SYSTEM_EVOLVED strict-abstain diagnostic) | Bench-Spec LOCK v1 multi-comparisons policy: secondary = descriptive, no correction required. --- ## 3. Sample design _Inherited verbatim from manifest v5 §3 (concurrency=1 retained). No change from v5._ - **Cells:** five, run in a single invocation. Definitions unchanged. 1. `no-context` — true zero-memory baseline. 2. `oracle-context` — PM-facing alias for harness `raw`. 3. `full-context` — oracle + SYSTEM_EVOLVED strict-abstain. 4. `retrieval` — conv-scope HybridSearch top-K=20. 5. `agentic` — softened SYSTEM_AGENTIC + bound search_memory + forced-fallback. - **N per cell:** 400. - **Total evaluations:** 2000. - **Instance selection seed:** `42`. - **Matched-pairs design:** same 400 instances flow through all cells. - **Concurrency:** `--parallel-concurrency 1`. Five cells sequential. --- ## 4. Dataset _Inherited verbatim from manifest v5 §4. No change._ - **Source:** `benchmarks/data/locomo10.json` (snap-research LoCoMo). - **Raw archive SHA-256:** `79fa87e90f04081343b8c8debecb80a9a6842b76a7aa537dc9fdf651ea698ff4`. - **Canonical build:** `benchmarks/data/locomo/locomo-1540.jsonl` (1531 instances). - **Canonical dataset SHA-256:** `39e415e2f3a0fa1bd3cb1804a58d0b440b50d3070b2100698437e4ec402a5b24`. - **Selection:** 400 per cell via seed-42 shuffle + take-first-400. --- ## 5. Model stack ### 5.1 Subject route table (Qwen 3.6-35B-A3B-Thinking) _Inherited verbatim from manifest v5 §5.1. No change._ | Priority | alias | thinking | max_tokens | |----------|-------|----------|------------| | primary | `qwen3.6-35b-a3b-via-dashscope-direct` | `on` | 16000 | | fallback_1 | `qwen3.6-35b-a3b-via-openrouter` | `on` | 64000 | | fallback_2 | `NOT_AVAILABLE` | — | — | Pricing $0.20 / $0.80 per M in/out. `floating_alias` pinning. B3 addendum § 5. ### 5.2 Judge ensemble — **CHANGED (ensemble swap + backup policy)** | Slot | alias (LiteLLM) | role | routing | rate-limit (v6) | |------|------------------|------|---------|-------------------| | primary_judge_1 | `claude-opus-4-7` | primary | anthropic direct | none (Anthropic immutable) | | primary_judge_2 | `gpt-5.4` | primary | openai direct | none | | **primary_judge_3** | **`minimax-m27-via-openrouter`** | **primary** | **openrouter (direct failed per §1.3h)** | **TBD at §1.3c-v6 probe time (OR tier-dependent; default 60 RPM if unspecified)** | | **backup_judge** | **`kimi-k26-direct`** | **backup (per-instance failover)** | **moonshot direct api.moonshot.ai/v1** | **TBD at §1.3c-v6 probe time (Moonshot tier-dependent)** | | ~~tiebreak_reserve~~ | ~~`grok-4.20`~~ | — (RETIRED in v6) | — | — | **Backup activation policy (new in v6):** - Primary judges (Opus + GPT + MiniMax) execute majority vote per instance. - If MiniMax primary fails (API error / parse failure / 60s timeout / non-200 HTTP), Kimi K2.6 backup is activated **for that single instance only** (per-instance failover). - If both MiniMax and Kimi fail for a single instance → `judge_ensemble_fail` marker; instance excluded from final analysis per post-hoc exclusion policy §9 (counted as `evaluator_loss` in denominator). - Three-way 1/1/1 split on primary trio → PM escalation (no reserve judge in v6; Grok-4.20 retired from tie-break role). - 2/2 defensive tie → PM escalation (unchanged from v5 policy). **Consistency constraint:** One judge call per instance per primary judge; backup called only on primary_judge_3 failure. No prompt-level batching. Identical prompt template per `failure-mode-judge.ts:245-258` verbatim. Temperature=0.0. Matched max_tokens per model (MiniMax/Kimi: 4096 per §1.3h findings; Opus/GPT per v5). **κ monitoring (κ re-cal phase, §5.4):** three pairwise Cohen's κ + conservative trio min. Thresholds from Bench-Spec LOCK v1 (pass ≥ 0.65; borderline 0.60-0.65; halt ≤ 0.60) retained. v6 κ re-cal success criterion ≥ 0.70 substantial agreement (tighter than operational halt threshold). #### 5.2.1 Failover behavior on MiniMax unavailability (clarification — added 2026-04-24 post-Phase-2 pre-flight, under v6 authority; canonical anchor `60d061e` preserved) The pre-registered backup activation ("Kimi K2.6 per-instance failover") is RETRACTED based on §1.3g-h-C Kimi reliability findings (parse rate 67-71% on challenging samples, p50 32s latency, p95 exceeds 60s timeout threshold). Kimi retirement from v6 ensemble is a clarification, not substantive methodology change: ensemble membership (Opus+GPT+MiniMax trio), primary hypothesis test, and κ baseline remain unchanged. Quorum policy on MiniMax failure (API error, parse fail, timeout >60s after standard 3-retry judge-runner policy): - If Opus and GPT agree → majority verdict = their consensus (2-of-2 quorum) - If Opus and GPT disagree → evaluator_loss marker, instance excluded from primary hypothesis analysis - Expected MiniMax failure rate <1% per Phase 1 empirical evidence (100/100 parse, 0 routing errors) - Expected evaluator_loss rate projected <1% of N=400 This clarification **supersedes** the pre-retraction "Backup activation policy" paragraph above (Kimi per-instance failover + both-fail `judge_ensemble_fail`). The retained text is kept in place as audit-visible "pre-amendment intent" for tamper-evident trail; operational Phase 2 runs per §5.2.1 quorum policy. #### 5.2.2 Kimi alias retention in litellm-config. Kimi alias (`kimi-k26-direct`) retained in `litellm-config.yaml` as orphan declaration (not invoked by runner). Removal would require additional config amendment commit; retention preserves audit trail of v6 Phase 1 intent and is zero-cost operationally. ### 5.3 Health-check predicate _Inherited from manifest v5 §5.3 (health-check.ts frozen), amended for new aliases._ Pre-cell health check must verify liveness on all v6 active aliases: - `claude-opus-4-7` via `/v1/chat/completions` ping - `gpt-5.4` via ping - `minimax-m27-via-openrouter` via ping - `kimi-k26-direct` via ping - Subject aliases per v5 No code change to `health-check.ts` itself (retained as §11 frozen path); new aliases consumed via LiteLLM config lookup. ### 5.4 κ re-calibration methodology (NEW in v6 — Phase 1 gate) Conducted at Phase 1 before any Stage 3 N=400 execution. Gates Phase 2 authorization. **Sample:** full 100-instance κ calibration set from v5 at `benchmarks/results/locomo-mini-n20-retry-2026-04-24T00-02-12Z.jsonl` (same authoritative source used for §1.3h split analysis). Identical sample → new trio κ is directly comparable to v5's original κ=0.7458 three-way baseline. **Judge verdicts reused:** - `claude-opus-4-7`: 100 existing verdicts from `judge_ensemble` field. Zero new calls. - `gpt-5.4`: 100 existing verdicts from `judge_ensemble` field. Zero new calls. **Judge verdicts new (Phase 1 execution):** - `minimax-m27-via-openrouter`: 100 new calls, verbatim prompt from `failure-mode-judge.ts:245-258`, temperature=0.0, max_tokens=4096. **Total new API calls at Phase 1: 100 (MiniMax only).** **Computation (three pairwise Cohen's κ):** - κ(Opus, GPT): should match v5's historical baseline (~0.74-0.82 range) - κ(Opus, MiniMax): new measurement - κ(GPT, MiniMax): new measurement **Conservative trio κ = min(three pairwise κ values).** **Also reported:** - Raw agreement % per pair - Confusion matrix per pair - Per-cell breakdown (no-context / oracle-context / full-context / retrieval / agentic) **Success criteria (v6 Phase 1 κ re-cal gate):** - `κ_conservative_trio ≥ 0.70` → **PASS**, halt with `PM-RATIFY-V6-KAPPA` for Phase 2 authorization - `0.60 ≤ κ_conservative_trio < 0.70` → **BORDERLINE**, halt with PM adjudication request - `κ_conservative_trio < 0.60` → **FAIL**, halt with swap-path-re-evaluation request (may require Kimi promoted to primary or backup-judge strategy rework) **Operational hedge:** during 100-call execution, log parse rate (target ≥95/100), latency p50 (target ≤25s) + p95, OpenRouter routing errors. If parse rate <90/100, halt before κ compute and raise PM flag. --- ## 6. Substrate (conv-scope retrieval) _Inherited verbatim from manifest v5 §6. No change._ - `@waggle/core::HybridSearch` (RRF-fused FTS5 + vec0). - `gopId = conversation_id` scope filter at `search.ts:14`. - Top-K default 20; upper clamp 50. - `createOllamaEmbedder()` + `nomic-embed-text` (1024 dims, local, $0). - Ingest batch 200. ### 6.1 Agentic-cell tool binding _Inherited verbatim from manifest v5 §6.1. No change._ `makeSearchMemoryTool(substrate, 20, instance.conversation_id)`. `maxTurns=3`. 180 s timeout. Forced-fallback on empty content + tool-use (0 firings at Gate C; load-bearing insurance). --- ## 7. SYSTEM_AGENTIC prompt — verbatim bytes locked _Inherited verbatim from manifest v5 §7. No change._ **SHA-256:** `6facae6decc44a6404290514accb4f7cb364081b32d02847a20f8e871633e328` (1467 bytes, no trailing newline). Source: `benchmarks/harness/src/cells.ts` lines 75–102. Softened text from Stage 2-Retry Gate A (commit `373516c`). --- ## 8. Stopping rules _Inherited verbatim from manifest v5 §8 (v5 §7.4 update under concurrency=1). No change._ | # | Rule | Source | Trigger | Action | |---|------|--------|---------|--------| | §7.1 | Budget hard halt | `runner.ts` | cumulative spend ≥ **$55.00** (v6 budget) | halt, persist partial, exit ping | | §7.2 | Streak halt | `streak-tracker.ts` | 3 consecutive subject fetch failures | halt, persist partial | | §7.3 | Pre-cell health check fail | `health-check.ts` | 5xx / fetch-error on subject or any judge probe | halt before cell | | §7.4 | Runner lock | `runner-lock.ts` | concurrent cross-process invocation detected | halt (§1.1 waiver unchanged) | | §7.5 | Pre-registration deviation | this document | any change to §1–§9 during run | halt + PM raise | Note: v6 budget hard halt at $55 (was $28 in v5) reflects expanded envelope for κ re-cal + N=400 combined. See §14. **No interim looks.** Halt only on the five conditions above. --- ## 9. Post-hoc exclusion policy: **NONE** _Inherited verbatim from manifest v5 §9. No change. `judge_ensemble_fail` (from v6 §5.2 backup-failover failure) counts in denominator as `evaluator_loss`._ All 2000 evals enter the denominator. `evaluator_loss` (judge-triple failure, including MiniMax+Kimi both-failed failover) counted in denominator, reported separately. No instance whitelist/blacklist. Selective exclusion forbidden ex-ante. --- ## 10. Deviation policy _Inherited verbatim from manifest v5 §10. No change._ Any deviation from §1–§9 during run → (1) immediate halt, (2) PM raise, (3) re-pre-registration (manifest v7+) if accepted. --- ## 11. Code freeze — **updated via v6 supersession of v5 §11** The following code is **frozen at HEAD `373516c`** for the duration of Stage 3 N=400 under v6. v6 emits the single permitted amendment to `litellm-config.yaml` as Phase 1 Commit 2 (under v6 authority — explicit supersession of v5 §11 freeze per PM authorization 2026-04-24). **v6 post-amendment state pinned:** `litellm-config.yaml` at Phase 1 Commit 2's tree state. The amendment adds `minimax-m27-via-openrouter` + `kimi-k26-direct` aliases. All v5 entries retained (including the Gemini `gemini-3.1-pro` alias with `rpm:20` — retained as orphan audit artefact; not routed in v6 judge ensemble). Frozen paths (inherited from v5 §11, unchanged EXCEPT `litellm-config.yaml`): - Cell semantics (`benchmarks/harness/src/cells.ts`). - Substrate (`benchmarks/harness/src/substrate.ts`, `@waggle/core::HybridSearch`, `@waggle/core::FrameStore`, `@waggle/core::SessionStore`). - SYSTEM_AGENTIC + SYSTEM_AGENTIC_FORCED_FALLBACK prompts. - Agent loop (`@waggle/agent::runAgentLoop`, `@waggle/agent::tools.ts`). - Judge ensemble + routing (`benchmarks/harness/src/judge-*.ts`, `benchmarks/harness/src/failure-mode-judge.ts`, `config/models.json`). - Runner + health-check (`benchmarks/harness/src/runner.ts`, `benchmarks/harness/src/health-check.ts`, `benchmarks/harness/src/runner-lock.ts`, `benchmarks/harness/src/streak-tracker.ts`). - Subject route table entries within `config/models.json`. - Test suite. - `litellm-config.yaml` pinned at **v6 Phase 1 Commit 2's tree state** (supersedes v5's pre-amendment pin). Execution-only delta during N=400 run: new JSONL files emitted to `benchmarks/results/` (κ re-cal output goes to `benchmarks/calibration/v6-kappa-recal/`). No code file modifications during or after run. --- ## 12. Scope boundaries _Inherited verbatim from manifest v5 §12. No change._ ### Can claim at Gate D: - Magnitude + significance of conv-scope retrieval memory-lift at Qwen-3.6-35b-a3b under HEAD 373516c. - Per-cell judge-accuracy with Wilson 95% CIs. - Monotonicity chain. - Conv-scope fair-comparison methodology. - Agentic discipline numbers. ### Cannot claim at Gate D: - Direct comparability to Mem0 91.6% (different scope + memory-synthesis layer). - Multi-model generalization (Qwen-only). - Production performance. ### Reserved for PM: - Public-claim phrasing + venue. - Matched-scope Mem0 co-run. - Publication timing. **CC-1 does NOT compose public SOTA claim.** Scope + data only. --- ## 13. PM gates — **Gate P++ + Gate P+++ new; Gate D unchanged** ### Gate P++ (v6 Phase 1: κ re-cal + config amendment) - Trigger: Phase 1 completion = v6 emission commit + `litellm-config.yaml` amendment commit + κ re-cal analysis commit on `feature/c3-v3-wrapper`. - Halt: CC-1 stops; no Phase 2 N=400 kick without PM-RATIFY-V6-KAPPA. - PM checks: v6 content matches brief §1–§5; κ_conservative_trio ≥ 0.70; MiniMax parse + latency + routing operational metrics acceptable. ### Gate P+++ (v6 Phase 2 kick = post-κ ratification) - Trigger: PM-RATIFY-V6-KAPPA received after Phase 1 ratification. - Action: CC-1 kicks N=400 execution via v5's `cli_invocation_template` patched for v6 aliases (`--judge-ensemble claude-opus-4-7,gpt-5.4,minimax-m27-via-openrouter --backup-judge kimi-k26-direct`). ### Gate D (post-run, pre-SOTA-claim) - Trigger: N=400 run exit (clean or halted per §8). - Action: CC-1 writes exit report at `PM-Waggle-OS/sessions/2026-04-24-task25-stage3-n400-complete.md`. - PM decides SOTA claim composition / publish gate / further scope. No self-advance at any gate. --- ## 14. Budget — **envelope expanded for Phase 1 + Phase 2** - **v6 total cap:** $60.00 (v5: $30.00) - **v6 total hard halt:** $55.00 (v5: $28.00) - **v6 expected total burn:** ~$50.00 (v5: ~$23.00) - Phase 1 κ re-cal: ~$25 (100 MiniMax calls via OR @ $0.30 prompt + $1.20 completion per M; ~250K prompt tokens + ~50K completion tokens estimated → well under cap) - Phase 2 N=400: ~$25 (subject + 3 primary judges × 2000 evals; OR MiniMax pricing vs v5's Gemini preview premium delta) - **Phase 1 cap:** $30 (brief §7) - **Phase 1 halt:** $35 - **Phase 2 cap:** $30 (separate envelope; authorized by PM-RATIFY-V6-KAPPA + subsequent brief) **Cost breakdown (expected, per phase):** - Subject (Qwen DashScope-intl): ~$2.50 (Phase 2 only) - Judge triple Opus+GPT+MiniMax: ~$22 (Phase 2) - MiniMax κ re-cal: ~$2 (Phase 1) - Kimi backup activations (per-instance failover, expected <5% trigger rate): ~$1 (Phase 2, variable) - Ollama embedding local: $0 **Wall-clock estimate (Phase 2 N=400 unchanged from v5's 2-3 hour estimate);** Phase 1 κ re-cal ≤90 min per brief §7. --- ## 15. Related artefacts ### v6 ancestry - **Manifest v5 predecessor:** anchor commit `fc16925` (audit-immutable). - **§5.2 rpm:20 edit:** anchor `ad324cc` (v5 §11 exception, retained in v6 config). - **§1.3c throttle probe PASS:** anchor `3a146ef`. - **Fold-in 3.5b sibling mirror:** anchor `d0ab680`. - **§1.3e RPD feasibility:** anchor `1d3851d`. - **§1.3f Vertex Batch INFEASIBLE:** anchor `8ad0567`. - **§1.3g Judge swap MULTI_PASS:** anchor `8a2f0e6`. - **§1.3h Stratified re-probe:** anchor `ae0d312`. - **§1.3h-C DeepSeek mt bump:** anchor `005a19a`. ### Inherited predecessors (unchanged) - **Manifest v4:** anchor `dedd698` (obsolete). - **§1.1 lock-semantics waiver:** anchor `67eb899`. - **§1.2 runner RCA:** anchor `274e987`. - **§1.3 Gate P+ probe FAIL:** anchor `66dcd5a`. - **§1.3b scope audit:** anchor `69a14708`. - **Bench-Spec LOCK v1 parent:** `PM-Waggle-OS/decisions/2026-04-22-bench-spec-locked.manifest.yaml`. - **Stage 2-Retry Gate C exit:** `PM-Waggle-OS/sessions/2026-04-24-task25-stage2-retry-complete.md`. - **Rollback tag:** `checkpoint/pre-self-evolution-2026-04-14`. ### v6-specific (this pre-registration) - **v6 Phase 1 Commit 1 (manifest emission):** THIS COMMIT. - **v6 Phase 1 Commit 2 (config amendment):** recorded at Commit 2 time. - **v6 Phase 1 Commit 3 (κ re-cal artefacts):** recorded at Commit 3 time. - **v6 brief:** `PM-Waggle-OS/briefs/2026-04-24-cc1-manifest-v6-phase1-kappa-recal-brief.md`. --- _End of Manifest v6 pre-registration. This document is the anchor for all analysis choices at Stage 3 Gate D exit under the judge-ensemble-swap path. v5 remains audit-immutable predecessor._