Stage 2 Prep Backlog

Items that must close before Stage 2 LoCoMo full 4-cell main run kickoff (Sprint 10 or later). Each entry is scoped to produce a pre-flight stability envelope Stage 2 depends on, without touching the judge / substrate / retrieval layers once they're frozen.

Conventions:

BLOCKS Stage 2 LoCoMo full-run kickoff — item must PASS before the scaled batch is approved.
Sprint 9 orthogonal — items do not block Sprint 9 completion (judge calibration on synthesized triples is independent).
Each entry carries origin reference + acceptance pattern + budget ceiling.

Qwen3.6 thinking-mode stability matrix (pre LoCoMo full run)

Opened: 2026-04-21 Origin: PM-Waggle-OS/sessions/2026-04-21-stage-0-final-close-out.md §2.2 + §7 — Stage 0 Q2 re-run on the Task-0.5 KG produced no final answer because Qwen3.6-35B-A3B thinking-mode, at max_tokens=16000, looped in its reasoning stream ("cannot complete this thought" repeated verbatim) and never transitioned to synthesis. The substrate had everything needed; inference layer did not converge.

Why this matters for Stage 2

Stage 2 LoCoMo full run hits qwen3.6-35b-a3b-via-openrouter at ~200 per-cell × 4 cells + judge ensemble. If Qwen thinking-mode ever loops on a Stage-2 prompt shape the way it did on Stage-0 Q2, a full run will burn budget on unusable outputs AND risk corrupting the aggregate report (unconverged inference + still-charged tokens skew the cost/quality picture). We need the failure-mode envelope identified in a small, bounded test matrix BEFORE the scaled batch fires.

Acceptance pattern — systematic stability matrix

Test cells (2 × 4 × 5 = 40 configurations):

Axis	Values
thinking toggle	`on` (current default) / `off` (via `enable_thinking: false` extra_body — verify availability on DashScope + OpenRouter routes first)
max_tokens ceiling	8K / 16K / 32K / 64K
prompt shape	direct-fact / multi-anchor-enumeration / chain-of-anchor / temporal-scope / null-result-tolerant

Prompt shape definitions (worth locking before the matrix runs):

direct-fact: single factual lookup — "When did X happen?" — one retrievable datum expected.
multi-anchor-enumeration: N enumerated components requested — "List three key Y components with model + date + session per component" (this is Stage-0 Q2's shape — the one that looped).
chain-of-anchor: cross-reference across retrieved frames — "Connect A (from session X) to B (from session Y) via their common theme".
temporal-scope: date-bounded lookup — "In December 2025, what structural analysis happened for Z?" (this is Stage-0 Q1's shape).
null-result-tolerant: question whose correct answer may be "no evidence" — "Is there a Calendar event linked to session S in the same week?" (legitimate negative outcome accepted).

Matrix cell outcome categories:

Outcome	Definition	Stage 2 implication
`converged`	`content` populated, final answer parses, token count ≤ 0.9 × ceiling	Cell is safe for Stage 2
`loop`	reasoning_content repeats a phrase 3+ times in final 1K chars; content empty	Cell BLOCKS Stage 2 — avoid prompt shape or raise token ceiling, whichever is cheaper
`truncated`	content populated but ends mid-sentence; completion_tokens = ceiling	Cell WARNS — raise max_tokens for Stage 2 if used
`empty-reasoning-only`	content empty; reasoning_content populated	Cell WARNS — raise max_tokens or switch thinking off

Deliverables

A heat-map CSV at waggle-os/preflight-results/qwen-stability-matrix-<ISO>.csv with 40 rows (one per cell) columns: thinking, max_tokens, prompt_shape, outcome, completion_tokens, wall_clock_ms, cost_usd, content_preview_first_500.
A markdown summary at waggle-os/preflight-results/qwen-stability-matrix-<ISO>.md flagging which cells must be avoided for Stage 2 (cells marked loop under both thinking toggles) + recommended max_tokens ceiling per prompt shape.
If ANY cell shows converged at a lower max_tokens than 16K with thinking-off, prefer that configuration for Stage 2 baseline — materially smaller per-call cost multiplies across a 200-call batch.

Budget ceiling

$5 total for the matrix run, using these economics:

Thinking-off at 8K tokens ≈ $0.004 per call → 5 shapes × 1 call = $0.02
Thinking-on at 16K tokens ≈ $0.016 per call → 5 shapes × 1 call = $0.08
Thinking-off at 32K tokens ≈ $0.016 per call (unused thinking headroom doesn't burn)
Thinking-on at 64K tokens ≈ $0.060 per call → 5 shapes × 1 call = $0.30

40 cells × ~$0.025 average = $1.00 baseline + retry headroom → well under $5.

Dry-run the runner first with synthetic fixtures + a --cells 3 flag to cap the matrix at 3 cells during development. Real-call path only after dry-run validates that the matrix runner parses Ollama + LiteLLM responses correctly (we already have this infrastructure in scripts/stage-0-query.mjs).

Dependencies

Depends on: nothing architectural. Uses the existing scripts/stage-0-query.mjs infrastructure with an extended --backend / --model / --max-tokens / --thinking-off flag set. Minimal new code — wrap the existing script in a matrix driver.
Relationships:
- NOT blocker for Sprint 9 Tasks 4/5 (judge calibration uses Sonnet on synthesized triples — thinking-mode not relevant to Sonnet).
- NOT blocker for launch narrative (pre-LoCoMo stability is a Stage 2 operational concern, not a defensibility concern for what Sprint 9 produces).
- BLOCKS Stage 2 LoCoMo full 4-cell main run kickoff. Budget owner and PM need the stability envelope before approving the scaled batch budget.
- BLOCKS H-42 / H-43 / H-44 scaled benchmark runs if they depend on Qwen3.6-35B-A3B thinking-mode inference paths.

Estimated effort

4-6 hours wall-clock: 1-2h to extend scripts/stage-0-query.mjs into a matrix driver with cell iteration + outcome classifier, 1h for dry-run validation on synthetic prompts, 1-2h for real-call matrix execution + CSV assembly, 30 min for markdown summary + Stage-2 recommendation.

Not-in-scope (parked / deferred)

Root-cause fix for Qwen's loop behavior — that's a provider-side issue, not something the harness should try to patch. Matrix identifies the failure envelope; avoidance or budget-headroom is our mitigation, not provider debugging.
Equivalent stability matrix for Qwen3-30B-A3B or other candidate models — scoped only to the canonical Stage-2 engine (Qwen3.6-35B-A3B).
Integration with aggregate.ts Week-1 cost projection — the matrix produces static guidance, not a live cost-monitor hook.

6.6 KiB Raw Permalink Blame History Unescape Escape