waggle-os/docs/briefs/2026-04-27-substrate-integrity-audit-brief.md

Substrate Integrity Audit Brief (Korak 12, light scope)

Date: 2026-04-27 Author: PM Status: Authored awaiting Marko ratification + CC-3 (or CC-2 post sibling workflow) executor assignment Scope corollary: Korak 12 iz 14-step launch plan (substrate claim integrity audit) Light scope rationale: Memory Sync Repair Step 1+2 empirically verified substrate parity (zero bugs surfaced, zero API drift, all 591 mind/ tests pass against waggle-os). Korak 12 audit therefore reduces from full independent audit to standard reproducibility verification.

---

§1 — Goal

Pre-launch verification da paper claim 74% self-judge oracle / 33.5% trio-strict oracle (Stage 3 v6 N=400) ostaje reproducibility-anchored kroz post-pilot agent fix sprint + V2 retrieval work. Substrate claim integrity audit je critical-path pre arxiv submission + Day 0 launch.

Light scope: 4 deliverables pre arxiv submission (estimated 1 day CC work + ~$0.50 cost).

---

§2 — Light scope rationale (post 2026-04-27 Memory Sync Repair)

Originalni Korak 12 scope (pre-Memory-Sync-Repair):

• Manifest v6 reproducibility verification (SHA256 compare canonical)
• Audit chain SHA verification
• Trio judge ensemble κ recalibration
• Apples-to-apples re-eval external verifiability
• Mock-fallback events absence verification
• V2 reproducibility check pre arxiv submission

Reduced scope (post-Step 1+2 evidence):

• Empirijski signal: zero bugs surfaced through 14 hive-mind test ports + 591 tests pass
• Zero API drift detected on critical files (frames/search/knowledge — sve 32 cases pass)
• Substrate parity verified through 3 lenses (empirical + structural + procedural per Memory Sync Repair closure memo §3)
• 14 test files added → production code (Tauri desktop) sad ima coverage koje nije imao za Stage 3 v6

Implikacija: substrate claim 74%/33.5% nije confounded by hidden bugs. Independent audit nepotreban; standard reproducibility verification dovoljan kao pre-launch prerequisite.

---

§3 — Four deliverables (light scope)

3.1 — Manifest v6 reproducibility verification

Goal: verify pre-registered manifest v6 still anchors all paper claims at byte-identical hash.

Steps:

1. Compute SHA256 of D:\Projects\waggle-os\benchmarks\preregistration\manifest-v6-preregistration.yaml
2. Compare to canonical SHA256 from Stage 3 v6 5-cell summary memo (5d5c1023421cd1a79f4913bb4c0a59415e21f50797255bff7dfec8e16b68e3ed)
3. If MATCH: manifest reproducibility intact, paper claims pinned to known config
4. If MISMATCH: investigate which lines changed, why, ratify with Marko whether legitimate post-stage-3 amendment or accidental drift

Expected outcome: MATCH (manifest is in v6 §11 frozen path list, untouched by agent fix sprint).

Acceptance: SHA256 verification documented + result recorded in Korak 12 results memo.

Cost: $0 (local SHA computation).

3.2 — Audit chain SHA verification

Goal: verify Stage 3 v6 audit chain (amendment v2 + amendment v1 + cc1_brief + judge_rubric + HEAD) preserved + reachable.

Steps:

1. Read decisions/2026-04-26-pilot-verdict-FAIL.md audit chain block
2. Verify each SHA still accessible:
   • amendment_v2_doc_sha256: 1ab5082ff773538a26b3c3294f7fbee4e30063a8d994bdb3753bdc9dd6d6cd99
   • amendment_v1_doc_sha256: 3946d3e00fbb1996fb7e63096ecef51abf1e209e5ff166fd0d8758e9a3a14aad
   • cc1_brief_sha256: 9805adae478333178d36d71b88795afc37f8fb543c2ebccaecb7b01faf06afee
   • judge_rubric_sha256: 2e24826eb75e92ef1e64055bb2c632eec64ded8fedf7d5b6897ccaec9ffff2eb
   • head_sha: b7e19c557fdbc42f2d0a3c3213176aa4d790f7a2
3. Spot-check: compute SHA256 of each referenced file, verify match
4. Note: HEAD SHA is at execution time of Stage 3 v6 N=400 + pilot 2026-04-26; current main HEAD will differ (post agent fix commits) — that's expected. Audit chain pinning is to historical HEAD, not current.

Acceptance: all 5 SHAs traceable + spot-check verified for amendment v1+v2 + cc1_brief + judge_rubric.

Cost: $0 (local SHA computation).

3.3 — Trio judge ensemble κ recalibration spot-check

Goal: verify trio-strict κ=0.7878 (ensemble agreement on 14-instance PM-labeled subset) reproduces under current model versions.

Why needed: Opus 4.7 / GPT-5.4 / MiniMax M2.7 may have model-version updates between Stage 3 v6 (2026-04-24) and arxiv submission. If versions changed and κ drifts substantially (>0.05 Fleiss κ), paper claim methodology bias quantification (+27.35pp self-judge bias) needs re-anchoring.

Steps:

1. Pull 14-instance PM-labeled subset from benchmarks/calibration/2026-04-24-trio-strict-recal.json
2. Re-run trio judges (Opus 4.7 + GPT-5.4 + MiniMax M2.7 with judge max_tokens=3000 per Stage 3 v6 fix)
3. Compute Fleiss κ on new ensemble verdicts
4. Compare to canonical 0.7878
5. If within ±0.05: κ stable, paper methodology bias claim intact
6. If drift > ±0.05: investigate which judge changed behavior, ratify with Marko whether re-recalibration of larger set needed

Acceptance: κ spot-check within ±0.05 of canonical 0.7878 OR documented drift + remediation plan.

Cost: ~$0.20 (14 instances × 3 judges × judge cost ~$0.005).

3.4 — V2 reproducibility check pre arxiv submission (lightweight)

Goal: ensure V2 retrieval results (when Phase C completes) can be reproduced byte-identical from V2 manifest + dataset + seed.

Steps:

1. Wait for V2 Phase C completion (ETA 2-3 weeks post Korak 1 + 1.5 done — per V2 brief)
2. Verify V2 manifest SHA + V2 results JSONL SHAs documented in V2 phase memo
3. Smoke replay: re-issue 5 random V2 predictions using verifyDeterministicReplay from Phase 1.3 run-meta.ts, verify byte-identical raw responses
4. If 5/5 byte-identical match → V2 reproducibility confirmed
5. If < 5/5 match → investigate non-determinism source (temperature drift, model version change, randomness in retrieval ranking)

Acceptance: V2 reproducibility verified pre arxiv submission OR documented gap + remediation.

Cost: ~$0.30 (5 prediction replay × subject + judge cost).

Sequencing note: 3.4 cannot start until V2 Phase C completes. 3.1 + 3.2 + 3.3 can run any time.

---

§4 — Out-of-scope (what NOT in this audit)

Per light scope rationale, the following are explicitly NOT in Korak 12:

1. Independent re-run of Stage 3 v6 N=400 trio-strict — costly (~$30) and unnecessary; Memory Sync Repair Step 1+2 empirical verification + audit chain SHA verification are sufficient signal.

2. Apples-to-apples re-eval external verifiability — original Apples-to-Apples self-judge re-eval anchor is already audit-ready (manifest + dataset SHA + judge prompt + raw outputs all in repo); external verifiability is community responsibility post arxiv release, not pre-launch PM scope.

3. Mock-fallback events historical absence verification — mock-fallback would have caused massive Stage 3 v6 anomaly (deterministic-mock embedder is semantically meaningless per embedding-provider.ts:131); 74% / 33.5% oracle results impossible if mock was active; absence is implicit.

4. Per-cell substrate behavior re-verification — Phase 2 acceptance gate dual-methodology smoke (CC-1 sprint) is the operational verification; Korak 12 doesn't duplicate that work.

5. Independent code review of mind/ substrate — Memory Sync Repair Step 2 test ports already exercised every public API surface; coverage gap (reconcile crash-recovery, etc.) closed empirically.

---

§5 — Acceptance criteria (binding)

Korak 12 audit ships when:

• 3.1 Manifest v6 SHA256 verification documented (MATCH expected)
• 3.2 Audit chain 5 SHAs spot-checked + recorded
• 3.3 Trio judge κ recalibration within ±0.05 (or documented drift)
• 3.4 V2 reproducibility replay 5/5 byte-identical (when V2 Phase C completes)
• Results memo: decisions/2026-04-XX-substrate-integrity-audit-results.md (XX = run date)
• PM ratification + sign-off pre arxiv submission

---

§6 — Sequencing

Independent of Korak 1 + 1.5 critical path:

• 3.1 + 3.2 + 3.3 can run anytime post Memory Sync Repair Step 3 sibling workflow PR opened (so workflows are ratified before integrity audit)
• 3.4 sequenced post V2 Phase C completion

Recommended timing:

• 3.1 + 3.2 + 3.3 ide u Week 2-3 of remaining launch path (after agent fix Phase 5 completes — to avoid CC sesija conflict)
• 3.4 ide post V2 Phase C (Week 5-6)
• Korak 12 ratification kao final pre-launch gate (Week 6-9 wall-clock)

---

§7 — Executor assignment

CC-3 (V2 work session) može da preuzme Korak 12 jer paralelno radi V2 ablations koji touch isti benchmark infrastructure. Plus CC-3 ima context warm na manifest + judge ensemble + V2 reproducibility patterns.

Alternative: CC-2 može da preuzme post sibling workflow PR completion (memory sync repair scope finished, contextually adjacent to substrate integrity work).

PM rec: CC-3 in V2 work session as natural fit. If CC-3 sesija scope > 16h, split: assign 3.1+3.2+3.3 to CC-3 early in V2 work, 3.4 post Phase C.

---

§8 — Open questions for Marko

1. Ratify light scope (4 deliverables, ~$0.50 cost, 1 day work) over original deeper scope? PM rec yes — Memory Sync Repair empirical verification je strong substitute za independent audit.
2. Executor assignment — CC-3 (preferred per V2 context warm) ili CC-2 (if sibling workflow leaves CC-2 with capacity)? PM rec CC-3.
3. Sequencing — 3.1+3.2+3.3 anytime post Step 3 sibling, 3.4 post V2 Phase C? PM rec yes.

---

§9 — Cross-references

• 14-step launch plan: .auto-memory/project_launch_plan_14_step_2026_04_27.md
• Memory Sync Repair closure memo: decisions/2026-04-27-memory-sync-repair-CLOSED.md
• Pilot verdict (audit chain anchor): decisions/2026-04-26-pilot-verdict-FAIL.md
• Stage 3 v6 5-cell summary (manifest SHA anchor): D:\Projects\waggle-os\benchmarks\results\stage3-n400-v6-final-5cell-summary.md
• V2 brief (3.4 prerequisite): briefs/2026-04-26-retrieval-v2-embeddings-audit-brief.md
• V2 pre-launch sequencing addendum: decisions/2026-04-26-v2-pre-launch-sequencing-addendum.md
• arxiv paper §5 (reproducibility): research/2026-04-26-arxiv-paper/01-paper-skeleton.md
• Trio judge calibration anchor: benchmarks/calibration/2026-04-24-trio-strict-recal.json (per project_task25_stage3_v6_phase1_pass.md memory)