waggle-os/docs/decisions/2026-04-27-phase-3-acceptance-gate-results.md

decision_id, date, phase, verdict, predecessor, sprint_plan, branch_head

decision_id	date	phase	verdict	predecessor	sprint_plan	branch_head
2026-04-27-phase-3-acceptance-gate-results	2026-04-27	3 acceptance gate — H6 long-task scenario validation	H6 INCONCLUSIVE — 2 of 3 models PASS Likert criteria; Opus partial only; compression criterion FAILED by design (Phase 3.4 audit-format gap)	2026-04-27-phase-3-acceptance-gate-pre-run-halt.md	D:\Projects\waggle-os\decisions\2026-04-26-agent-fix-sprint-plan.md	8b8a940 (Phase 3.4)

Phase 3 Acceptance Gate — Results

TL;DR

H6 verdict: INCONCLUSIVE. Mixed result split cleanly along three axes:

• Replay determinism + compression-preserves-meaning Likert criteria: PASS for Qwen and GPT (both at exactly 0.3 — at the threshold). Cannot evaluate for Opus.
• Self-judge accuracy on continuous baselines: 3/3 PASS (Qwen, GPT, Opus all "Yes").
• Cross-model coverage: PARTIAL FAIL. Opus only completed 1 of 3 sub-runs (continuous baseline at $2.75 alone consumed 39% of total cap; crash-resume's first leg consumed another $2.27 + halted; compressed-context never started).
• Compression criterion: HARD FAIL across all 3 models. Zero compress events fired in any of the 8 completed sub-runs. Root cause: Phase 3.4's accumulated_context audit format is structurally too small to ever cross the 4K-token threshold within 30 turns.

The compression failure is methodologically informative, not a runtime bug. It surfaces a Phase 3.4 design gap that should be fixed before any production claim: ContextManager only compresses the audit log, but the LLM cost is dominated by the messages array (which ContextManager doesn't touch). Phase 4 must address this if compression is to provide real cost-bound value at scale.

Cumulative cost: $6.15 (subject runs $3.85 + recovery analysis $0.03 + Opus crash-resume's first-leg waste $2.27, the last being the spike that tripped the cost halt). Wall: 17 min. Both within the ratified $7 hard cap and 30 min hard cap, but very close to the $6 halt threshold (which fired as expected at the Opus crash-resume boundary).

---

Audit chain

Item	Value
Branch HEAD	8b8a940 (Phase 3.4 commit)
Subject run JSONL	tmp/phase-3-gate-2026-04-27/results/runs.jsonl (8 records)
Recovery summary	tmp/phase-3-gate-2026-04-27/results/summary.json
Per-task checkpoints	tmp/phase-3-gate-2026-04-27/results/checkpoints/<task_id>/step-NNNNNN.json
Subject corpus	tmp/phase-3-gate-2026-04-27/corpus.ts (30 events, deterministic)
Retrieval mock	tmp/phase-3-gate-2026-04-27/retrieval-mock.ts (top-K keyword)
Run script	tmp/phase-3-gate-2026-04-27/run-scenario.ts
Pre-run halt memo	2026-04-27-phase-3-acceptance-gate-pre-run-halt.md

---

Per-criterion results

(1) All 3 models complete long-task without uncaught errors

Model	continuous-baseline	crash-resume	compressed-context	Verdict
Qwen 3.6 35B-A3B	✓ 30 steps, $0.23	✓ 30 steps, $0.13	✓ 30 steps, $0.11	PASS
GPT-5.4	✓ 16 steps, $0.17	✓ 18 steps, $0.22	✓ 19 steps, $0.21	PASS
Claude Opus 4.7	✓ 23 steps, $2.75	✗ failed (cost halt during resume leg)	✗ never started	PARTIAL FAIL

GPT and Opus finalized early (16-23 steps) — agent finalized before exhausting the 30-step budget. This is correct behavior (the agent decided it had enough info to rank the themes). Qwen used the full 30 steps each run.

(2) Crash-resume Likert ≤ 0.30 from continuous baseline

Model	Likert	Pass?
Qwen	0.3	✓ at threshold
GPT-5.4	0.3	✓ at threshold
Opus	(cannot evaluate — no resume answer)	—

Both Qwen and GPT crash-resumed cleanly: process A ran to step 14, threw the simulated crash at step 15, process B (fresh runner) loaded the latest checkpoint, restored the messages array, and continued from step 15 to a clean finalize. Final answers had only minor differences (citation reordering, one extra event mention) per the judge — well-aligned with the realistic Likert ≤ 0.30 standard PM ratified for real LLMs.

This is the single most important Phase 3 result: the runRetrievalAgentLoop + CheckpointStore + cross-process resume contract (Phase 3.1 + 3.4) works end-to-end on real LLMs across sovereign + frontier-API models.

(3) Compressed Likert ≤ 0.30 from continuous baseline

Model	Likert	Pass?
Qwen	0.3	✓ at threshold
GPT-5.4	0.3	✓ at threshold
Opus	(cannot evaluate)	—

But — see criterion (4): zero compressions actually occurred. The "compressed-context" sub-runs had ContextManager configured but never triggered, so we are effectively comparing "ContextManager configured-but-inactive" vs "no ContextManager". This Likert measure doesn't validate compression preservation; it validates that having ContextManager configured doesn't break the agent loop. Useful but weaker than the pre-registered claim.

(4) Compress events fire ≥1 per model with 4K budget at 30 steps

HARD FAIL. Zero compressions across all 8 completed sub-runs.

Root-cause diagnosis

accumulated_context is built by buildAccumulatedAudit() in retrieval-agent-loop.ts — one short line per turn:

Turn 1: retrieve query="WWI"
Turn 2: retrieve query="industrial revolution"
...

Each line is ~50-70 chars. 30 turns × 60 chars = ~1.8KB raw text ≈ ~450 tokens.

ContextManager threshold = 4000 × 0.7 = 2800 tokens.

Audit log grows ~6× slower than threshold can be reached. Compression never triggers, by design.

This is a Phase 3.4 implementation gap: ContextManager only compresses the audit log, but LLM cost is dominated by the messages array (which grows by ~550 tokens per retrieval turn = 16K+ tokens by turn 30 on Opus). ContextManager doesn't touch the messages array.

What this means for cost

Opus continuous-baseline at 23 steps cost $2.75. That's ~$0.12/step average. By step 23, input was ~13K tokens × $15/M = $0.20 just for that step's input. The cost trajectory is super-linear in step count, exactly as the pre-run halt memo predicted, and ContextManager-as-implemented does nothing to bound it.

Phase 4 fix recommendations (in order of impact)

1. Apply context compression to the messages array, not just the audit log. This is the substantive fix and would actually deliver the cost-bound value the brief expected. Would integrate with the existing context-compressor.ts utilities.

2. Expand accumulated_context content to include retrieval-result snippets per turn. Would grow audit ~10× faster, making compression validation testable at 4K budget. Cheaper change but doesn't fix cost growth.

3. Add a messagesContextManager config field to runRetrievalAgentLoop that triggers compressConversation() on the messages array when needed. Most surgical; uses existing infra.

(5) Self-judge accuracy ≥ 70% on continuous baseline final answers

Model	continuous-baseline self-judge	Pass?
Qwen	Yes	✓
GPT-5.4	Yes	✓
Opus	Yes	✓

3/3 = 100% PASS. All three models produced rankings whose top-4 themes matched the ground-truth ordering (war_or_conflict → tech_breakthrough → social_movement → economic_transformation), with at least 3 supporting events cited per top-4 theme.

Note: GPT's compressed-context sub-run got a "No" verdict (the only "No" across all judged answers). Inspecting the answer: GPT reclassified scientific_discovery into a separate top-tier category, breaking the top-4 ordering. This is not evidence of compression-induced regression because no compression actually occurred — it's an example of run-to-run variation on real LLMs at the same temperature, exactly as the relaxation criterion (item #2 of pre-run halt) anticipated.

(6) No infinite compression loops, no checkpoint corruption, no data loss

✓ PASS. All 8 completed sub-runs persisted full checkpoint chains (one file per turn at checkpoints/<task_id>/step-NNNNNN.json). Crash-resume scenarios verified: process B loaded the latest pre-crash checkpoint, restored messages_snapshot + running totals, and resumed cleanly. Zero file-system corruption, zero infinite loops.

(7) tsc strict clean

✓ PASS. Verified pre-gate (packages/agent + benchmarks/harness).

(8) All 5720+ unit tests still pass

✓ PASS. Verified pre-gate (5720 passed + 1 skipped at HEAD 8b8a940).

---

Cost + wall summary

Item	Cost	Notes
Qwen 3 sub-runs	$0.47	3 × 30-step, audit-only "compression"
GPT 3 sub-runs	$0.60	3 × 16-19 steps (early finalize)
Opus continuous baseline	$2.75	23 steps; super-linear input growth
Opus crash-resume first leg	~$2.27	14 steps before simulated crash; not in result.totalCostUsd (caller-side billed via guard)
Opus crash-resume second leg	$0.00	Hit cost halt at $6.09 immediately on first call
Recovery analysis (self-judge + Likert)	$0.035	7 self-judge + 4 Likert calls via Qwen
Total cumulative	$6.15	Hard cap $7.00 / halt $6.00
Wall	17 min	Hard cap 30 min / halt 25 min

The cost halt at $6.00 fired exactly as designed at the Opus crash-resume boundary. The pre-run halt memo predicted Opus runs would dominate cost; the live trajectory confirmed it (Opus continuous alone consumed ~45% of the entire budget).

---

What this gate validated vs what it didn't

Validated

1. CheckpointStore + cross-process resume work end-to-end on real LLMs (Qwen + GPT)
2. RecoveryRunner-style retry semantics work end-to-end
3. Replay determinism is preserved at the semantic level (Likert ≤ 0.30) for Qwen and GPT
4. Self-judge methodology gives stable Yes/No verdicts on the synthesis task
5. Phase 3.4's optional-fields backwards-compat: existing tests still pass; new fields don't break the loop
6. The integrated agent loop produces valid, human-readable theme rankings on a 30-doc corpus

Did NOT validate

1. Cross-model coverage to Opus 4.7 — only the continuous baseline succeeded
2. ContextManager compression actually firing in production — design gap surfaced (audit-only, not messages)
3. Compress-vs-uncompressed Likert preservation — compression never occurred
4. Long-task semantics at >30 steps — agents finalized early on this corpus

---

H6 verdict: INCONCLUSIVE

The pre-registered H6 hypothesis was:

Long-task scenario sa checkpoints + recovery completes successfully on Opus 4.7 + Qwen 3.6 35B-A3B + GPT-5.4 bez data loss across simulated multi-hour synthesis task.

For Qwen + GPT: PASSED. Both completed all three sub-runs cleanly. Crash-resume preserves semantic answer (Likert 0.3). Self-judge accuracy 100% on baseline. No data loss.

For Opus: PARTIAL FAIL. Only the continuous baseline completed (and at $2.75 alone — way over the brief's $2.00 estimate). Crash-resume's first leg burned $2.27 before the simulated crash, then the resume leg hit cost halt and produced no answer.

For compression validation: FAILED across all 3 models due to a Phase 3.4 design gap (audit log structurally too small for 4K threshold). Not a runtime bug — the implementation does what was specified, but the specification didn't catch that ContextManager only compresses the audit log, not the cost-dominant messages array.

INCONCLUSIVE rather than FAIL because:

• 2 of 3 models passed cleanly
• The Opus shortfall is a budget issue, not a correctness issue
• The compression failure is informative (surfaces a real Phase 4 work item)

---

PM ratification asks

1. Accept H6 INCONCLUSIVE with the partial Opus result as an explicit "scope caveat" rather than a re-run requirement?

   • Pro: cheap, gets us to Phase 4 with real signal on cross-model + cross-determinism
   • Con: Opus cross-model claim weaker than full H6 spec

2. Authorize a tight Opus-only follow-up run ($3 budget) to complete cross-model coverage?

   • Scope: Opus crash-resume + Opus compressed-context only (continuous already done)
   • Estimated cost: $3-4 (no buffer)
   • Estimated wall: ~5 min
   • Total cumulative cost would reach ~$9-10 (over the original $7 cap)

3. Authorize Phase 4 kickoff with explicit Phase 4 work items for:

   • (a) Fix the audit-vs-messages compression gap. Either (i) expand accumulated_context content (cheap), (ii) add messages-array compression hook to retrieval-agent-loop (substantive), or (iii) integrate existing context-compressor.ts utilities for the messages dimension (recommended).
   • (b) Re-score 2026-04-26 pilot with new normalization + classifier (Phase 4 acceptance gate from the sprint plan).
   • (c) Add a "compression-engaged-end-to-end" assertion test that would have caught this gap before the gate.

4. Update the cost-modeling discipline for future briefs to flag any per-step cost estimate that doesn't account for both input-growth dimensions (audit log + messages array). Extension 6 from the pre-run halt memo only covered super-linear input growth in the abstract — the gate result shows that even with that flag, briefs can still under-estimate when they assume ContextManager will bound BOTH dimensions.

---

Key signals carried forward

Signal	What it tells Phase 4
Crash-resume Likert 0.3 on Qwen + GPT	The Phase 3.4 resume contract is correct end-to-end. No further work needed on resume itself.
Compressions = 0 across all runs	ContextManager-as-implemented does NOT bound real cost growth. Phase 4 must address this if cost-bound is a real production goal.
Opus continuous at $2.75 / 23 steps	Frontier-proprietary models on 30-step retrieval-loop tasks are not competitive on cost without messages-array compression. Has KVARK / sovereign-deployment implications.
GPT and Opus finalize early (16-23 steps)	Models self-terminate on this synthesis task before exhausting 30 steps. The 30-step budget is HEADROOM, not a real constraint.
All 3 models top-4 ranking matches ground truth	The agent loop produces real synthesis quality. Not a regression from anything in Phase 1-2.

---

End of Phase 3 acceptance gate. Standing AWAITING PM RATIFICATION on items 1-4 above.