moving

docs/briefs/2026-04-27-substrate-integrity-audit-brief.md

@@ -0,0 +1,182 @@
+# Substrate Integrity Audit Brief (Korak 12, light scope)
+
+**Date:** 2026-04-27
+**Author:** PM
+**Status:** Authored awaiting Marko ratification + CC-3 (or CC-2 post sibling workflow) executor assignment
+**Scope corollary:** Korak 12 iz 14-step launch plan (substrate claim integrity audit)
+**Light scope rationale:** Memory Sync Repair Step 1+2 empirically verified substrate parity (zero bugs surfaced, zero API drift, all 591 mind/ tests pass against waggle-os). Korak 12 audit therefore reduces from full independent audit to standard reproducibility verification.
+
+---
+
+## §1 — Goal
+
+Pre-launch verification da paper claim 74% self-judge oracle / 33.5% trio-strict oracle (Stage 3 v6 N=400) ostaje reproducibility-anchored kroz post-pilot agent fix sprint + V2 retrieval work. **Substrate claim integrity audit je critical-path pre arxiv submission + Day 0 launch.**
+
+Light scope: 4 deliverables pre arxiv submission (estimated 1 day CC work + ~$0.50 cost).
+
+---
+
+## §2 — Light scope rationale (post 2026-04-27 Memory Sync Repair)
+
+**Originalni Korak 12 scope** (pre-Memory-Sync-Repair):
+- Manifest v6 reproducibility verification (SHA256 compare canonical)
+- Audit chain SHA verification
+- Trio judge ensemble κ recalibration
+- Apples-to-apples re-eval external verifiability
+- Mock-fallback events absence verification
+- V2 reproducibility check pre arxiv submission
+
+**Reduced scope** (post-Step 1+2 evidence):
+- Empirijski signal: zero bugs surfaced through 14 hive-mind test ports + 591 tests pass
+- Zero API drift detected on critical files (frames/search/knowledge — sve 32 cases pass)
+- Substrate parity verified through 3 lenses (empirical + structural + procedural per Memory Sync Repair closure memo §3)
+- 14 test files added → production code (Tauri desktop) sad ima coverage koje nije imao za Stage 3 v6
+
+**Implikacija:** substrate claim 74%/33.5% nije confounded by hidden bugs. Independent audit nepotreban; standard reproducibility verification dovoljan kao pre-launch prerequisite.
+
+---
+
+## §3 — Four deliverables (light scope)
+
+### 3.1 — Manifest v6 reproducibility verification
+
+**Goal:** verify pre-registered manifest v6 still anchors all paper claims at byte-identical hash.
+
+**Steps:**
+1. Compute SHA256 of `D:\Projects\waggle-os\benchmarks\preregistration\manifest-v6-preregistration.yaml`
+2. Compare to canonical SHA256 from Stage 3 v6 5-cell summary memo (`5d5c1023421cd1a79f4913bb4c0a59415e21f50797255bff7dfec8e16b68e3ed`)
+3. If MATCH: manifest reproducibility intact, paper claims pinned to known config
+4. If MISMATCH: investigate which lines changed, why, ratify with Marko whether legitimate post-stage-3 amendment or accidental drift
+
+**Expected outcome:** MATCH (manifest is in v6 §11 frozen path list, untouched by agent fix sprint).
+
+**Acceptance:** SHA256 verification documented + result recorded in Korak 12 results memo.
+
+**Cost:** $0 (local SHA computation).
+
+### 3.2 — Audit chain SHA verification
+
+**Goal:** verify Stage 3 v6 audit chain (amendment v2 + amendment v1 + cc1_brief + judge_rubric + HEAD) preserved + reachable.
+
+**Steps:**
+1. Read `decisions/2026-04-26-pilot-verdict-FAIL.md` audit chain block
+2. Verify each SHA still accessible:
+   - amendment_v2_doc_sha256: `1ab5082ff773538a26b3c3294f7fbee4e30063a8d994bdb3753bdc9dd6d6cd99`
+   - amendment_v1_doc_sha256: `3946d3e00fbb1996fb7e63096ecef51abf1e209e5ff166fd0d8758e9a3a14aad`
+   - cc1_brief_sha256: `9805adae478333178d36d71b88795afc37f8fb543c2ebccaecb7b01faf06afee`
+   - judge_rubric_sha256: `2e24826eb75e92ef1e64055bb2c632eec64ded8fedf7d5b6897ccaec9ffff2eb`
+   - head_sha: `b7e19c557fdbc42f2d0a3c3213176aa4d790f7a2`
+3. Spot-check: compute SHA256 of each referenced file, verify match
+4. Note: HEAD SHA is at execution time of Stage 3 v6 N=400 + pilot 2026-04-26; current main HEAD will differ (post agent fix commits) — that's expected. Audit chain pinning is to historical HEAD, not current.
+
+**Acceptance:** all 5 SHAs traceable + spot-check verified for amendment v1+v2 + cc1_brief + judge_rubric.
+
+**Cost:** $0 (local SHA computation).
+
+### 3.3 — Trio judge ensemble κ recalibration spot-check
+
+**Goal:** verify trio-strict κ=0.7878 (ensemble agreement on 14-instance PM-labeled subset) reproduces under current model versions.
+
+**Why needed:** Opus 4.7 / GPT-5.4 / MiniMax M2.7 may have model-version updates between Stage 3 v6 (2026-04-24) and arxiv submission. If versions changed and κ drifts substantially (>0.05 Fleiss κ), paper claim methodology bias quantification (+27.35pp self-judge bias) needs re-anchoring.
+
+**Steps:**
+1. Pull 14-instance PM-labeled subset from `benchmarks/calibration/2026-04-24-trio-strict-recal.json`
+2. Re-run trio judges (Opus 4.7 + GPT-5.4 + MiniMax M2.7 with judge max_tokens=3000 per Stage 3 v6 fix)
+3. Compute Fleiss κ on new ensemble verdicts
+4. Compare to canonical 0.7878
+5. If within ±0.05: κ stable, paper methodology bias claim intact
+6. If drift > ±0.05: investigate which judge changed behavior, ratify with Marko whether re-recalibration of larger set needed
+
+**Acceptance:** κ spot-check within ±0.05 of canonical 0.7878 OR documented drift + remediation plan.
+
+**Cost:** ~$0.20 (14 instances × 3 judges × judge cost ~$0.005).
+
+### 3.4 — V2 reproducibility check pre arxiv submission (lightweight)
+
+**Goal:** ensure V2 retrieval results (when Phase C completes) can be reproduced byte-identical from V2 manifest + dataset + seed.
+
+**Steps:**
+1. Wait for V2 Phase C completion (ETA 2-3 weeks post Korak 1 + 1.5 done — per V2 brief)
+2. Verify V2 manifest SHA + V2 results JSONL SHAs documented in V2 phase memo
+3. Smoke replay: re-issue 5 random V2 predictions using `verifyDeterministicReplay` from Phase 1.3 run-meta.ts, verify byte-identical raw responses
+4. If 5/5 byte-identical match → V2 reproducibility confirmed
+5. If < 5/5 match → investigate non-determinism source (temperature drift, model version change, randomness in retrieval ranking)
+
+**Acceptance:** V2 reproducibility verified pre arxiv submission OR documented gap + remediation.
+
+**Cost:** ~$0.30 (5 prediction replay × subject + judge cost).
+
+**Sequencing note:** 3.4 cannot start until V2 Phase C completes. 3.1 + 3.2 + 3.3 can run any time.
+
+---
+
+## §4 — Out-of-scope (what NOT in this audit)
+
+**Per light scope rationale, the following are explicitly NOT in Korak 12:**
+
+1. **Independent re-run of Stage 3 v6 N=400 trio-strict** — costly (~$30) and unnecessary; Memory Sync Repair Step 1+2 empirical verification + audit chain SHA verification are sufficient signal.
+
+2. **Apples-to-apples re-eval external verifiability** — original Apples-to-Apples self-judge re-eval anchor is already audit-ready (manifest + dataset SHA + judge prompt + raw outputs all in repo); external verifiability is community responsibility post arxiv release, not pre-launch PM scope.
+
+3. **Mock-fallback events historical absence verification** — mock-fallback would have caused massive Stage 3 v6 anomaly (deterministic-mock embedder is semantically meaningless per embedding-provider.ts:131); 74% / 33.5% oracle results impossible if mock was active; absence is implicit.
+
+4. **Per-cell substrate behavior re-verification** — Phase 2 acceptance gate dual-methodology smoke (CC-1 sprint) is the operational verification; Korak 12 doesn't duplicate that work.
+
+5. **Independent code review of mind/ substrate** — Memory Sync Repair Step 2 test ports already exercised every public API surface; coverage gap (reconcile crash-recovery, etc.) closed empirically.
+
+---
+
+## §5 — Acceptance criteria (binding)
+
+Korak 12 audit ships when:
+
+- [ ] 3.1 Manifest v6 SHA256 verification documented (MATCH expected)
+- [ ] 3.2 Audit chain 5 SHAs spot-checked + recorded
+- [ ] 3.3 Trio judge κ recalibration within ±0.05 (or documented drift)
+- [ ] 3.4 V2 reproducibility replay 5/5 byte-identical (when V2 Phase C completes)
+- [ ] Results memo: `decisions/2026-04-XX-substrate-integrity-audit-results.md` (XX = run date)
+- [ ] PM ratification + sign-off pre arxiv submission
+
+---
+
+## §6 — Sequencing
+
+**Independent of Korak 1 + 1.5 critical path:**
+- 3.1 + 3.2 + 3.3 can run **anytime** post Memory Sync Repair Step 3 sibling workflow PR opened (so workflows are ratified before integrity audit)
+- 3.4 sequenced post V2 Phase C completion
+
+**Recommended timing:**
+- 3.1 + 3.2 + 3.3 ide u Week 2-3 of remaining launch path (after agent fix Phase 5 completes — to avoid CC sesija conflict)
+- 3.4 ide post V2 Phase C (Week 5-6)
+- Korak 12 ratification kao final pre-launch gate (Week 6-9 wall-clock)
+
+---
+
+## §7 — Executor assignment
+
+CC-3 (V2 work session) može da preuzme Korak 12 jer paralelno radi V2 ablations koji touch isti benchmark infrastructure. Plus CC-3 ima context warm na manifest + judge ensemble + V2 reproducibility patterns.
+
+Alternative: CC-2 može da preuzme post sibling workflow PR completion (memory sync repair scope finished, contextually adjacent to substrate integrity work).
+
+PM rec: **CC-3 in V2 work session** as natural fit. If CC-3 sesija scope > 16h, split: assign 3.1+3.2+3.3 to CC-3 early in V2 work, 3.4 post Phase C.
+
+---
+
+## §8 — Open questions for Marko
+
+1. **Ratify light scope** (4 deliverables, ~$0.50 cost, 1 day work) over original deeper scope? PM rec yes — Memory Sync Repair empirical verification je strong substitute za independent audit.
+2. **Executor assignment** — CC-3 (preferred per V2 context warm) ili CC-2 (if sibling workflow leaves CC-2 with capacity)? PM rec CC-3.
+3. **Sequencing** — 3.1+3.2+3.3 anytime post Step 3 sibling, 3.4 post V2 Phase C? PM rec yes.
+
+---
+
+## §9 — Cross-references
+
+- 14-step launch plan: `.auto-memory/project_launch_plan_14_step_2026_04_27.md`
+- Memory Sync Repair closure memo: `decisions/2026-04-27-memory-sync-repair-CLOSED.md`
+- Pilot verdict (audit chain anchor): `decisions/2026-04-26-pilot-verdict-FAIL.md`
+- Stage 3 v6 5-cell summary (manifest SHA anchor): `D:\Projects\waggle-os\benchmarks\results\stage3-n400-v6-final-5cell-summary.md`
+- V2 brief (3.4 prerequisite): `briefs/2026-04-26-retrieval-v2-embeddings-audit-brief.md`
+- V2 pre-launch sequencing addendum: `decisions/2026-04-26-v2-pre-launch-sequencing-addendum.md`
+- arxiv paper §5 (reproducibility): `research/2026-04-26-arxiv-paper/01-paper-skeleton.md`
+- Trio judge calibration anchor: `benchmarks/calibration/2026-04-24-trio-strict-recal.json` (per project_task25_stage3_v6_phase1_pass.md memory)