--- decision_id: 2026-04-26-pilot-verdict-FAIL date: 2026-04-26 authority: PM (Marko MarkoviΔ‡) β€” ratified type: pilot close-out + halt expansion predecessors: - briefs/2026-04-26-agentic-knowledge-work-pilot/cc1-brief.md - briefs/2026-04-26-agentic-knowledge-work-pilot/cc1-brief-amendment-2026-04-26.md - briefs/2026-04-26-agentic-knowledge-work-pilot/cc1-brief-amendment-v2-2026-04-26.md manifest_anchor: pilot-2026-04-26-v1 --- # Pilot 2026-04-26 β€” Final Verdict: FAIL **Pilot:** `agentic-knowledge-work-pilot-2026-04-26` **Hypotheses tested:** H2 (Opus multiplier), H3 (Qwen multiplier), H4 (sovereignty bridge) **Sample:** N=3 tasks Γ— 4 cells = 12 candidate runs + 36 judge calls **Outcome:** **πŸ”΄ FAIL** β€” H2 1/3, H3 0/3, H4 0/3, critical_failures 0 **Total cost:** $5.5806 / $20 cap (28%) Β· **Total wall:** ~80 min over 3 sessions --- ## 1. Result table (binding) | Task | A (Opus solo) | B (Opus + harness) | C (Qwen solo) | D (Qwen + harness) | H2 (Bβˆ’A) | H3 (Dβˆ’C) | H4 (D vs A) | |------|---------------|---------------------|----------------|---------------------|----------|----------|--------------| | task-1 strategic synthesis (7 docs) | 4.611 | 4.944 | 4.583 | 4.389 | **+0.333 PASS** | βˆ’0.194 βœ— | βˆ’0.222 βœ— | | task-2 cross-thread coord (4 threads) | 4.944 | 5.000 | 4.667 | 3.944 | +0.056 βœ— | βˆ’0.722 βœ— | **βˆ’1.000 βœ—** | | task-3 decision support (3 memos) | 4.944 | 4.889 | 4.889 | 4.556 | βˆ’0.056 βœ— | βˆ’0.333 βœ— | βˆ’0.389 βœ— | | **TOTALS** | β€” | β€” | β€” | β€” | **1/3** | **0/3** | **0/3** | Pre-registered PASS criteria (cc1-brief Β§2): each hypothesis directional sign on β‰₯ 2 of 3 tasks. **All three hypotheses fail this criterion.** --- ## 2. What this pilot did and did not test **Tested:** - Multi-step agent loop with retrieval-augmented self-prompting (amendment v1 Β§1, renamed in amendment v2 Β§1) - HybridSearch (FTS5 + vec0 RRF) over per-task SessionStore corpus - Trio judge ensemble (Opus + GPT + MiniMax M2.7, ΞΊ=0.7878 from Stage 3 v6 calibration) - Two model classes (Claude Opus 4.7, Qwen 3.6 35B-A3B via DashScope direct) - Three task types (strategic synthesis, cross-thread coordination, decision support) **Did NOT test:** - True GEPA self-evolve (deferred β€” no labeled training corpus exists for open-ended N=1 synthesis tasks) - Long-task scenarios (multi-hour runs, checkpoint/recovery, context window exhaustion) - Skills/tools layer (only retrieval was exercised; agent had no tool-use beyond search) - Across-model-variance bound (H5, not pre-registered for this pilot) **Implication:** the FAIL verdict is on the multi-step harness pattern as currently implemented. GEPA self-evolve as a separate thesis remains untested and could be evaluated separately if a labeled corpus is authored. --- ## 3. Methodology audit findings (binding for future work) ### 3.1 Original smoke had 2 confounds β€” addressed via amendment v2 - **Wrong Qwen model:** original brief specified `qwen3.6-35b-a3b-via-openrouter` which bridges to Qwen 3.5 (`openrouter/qwen/qwen3.5-35b-a3b`) per litellm-config.yaml comment "one-minor regress to 3.5 until OR carries 3.6". Cells C/D Task 1 originally ran on Qwen 3.5. - **Wrong max_tokens for synthesis:** wrapper default 4096 vs Stage 3 v6 lineage 16000-64000. Reasoning headroom may have been silently constrained. Amendment v2 Β§2 corrected both: `qwen3.6-35b-a3b-via-dashscope-direct` + thinking=on + max_tokens=16000. Cells C/D Task 1 restarted under corrected config. Restart shifted Cell C 4.167 β†’ 4.583 (+0.42), Cell D 3.944 β†’ 4.389 (+0.45). H4 reversal magnitude fell from βˆ’0.555 β†’ βˆ’0.111 β€” but did not cross zero. ### 3.2 Confounds NOT eliminated by amendment v2 - **Cell B Opus loop_exhausted on Tasks 2 + 3:** the 5-step `MAX_STEPS` ceiling was binding. Task-2/B `loop_exhausted=true` (steps=3, retr=2 β€” but step budget reached without natural finalization). Task-3/B `loop_exhausted=true` (steps=4, retr=3). Opus multi-step was force-finalized on those tasks. May have degraded Cell B output β†’ contributed to H2 reading negative on Tasks 2 + 3. - **Cell A judge ceiling:** Opus solo scored 4.611 / 4.944 / 4.944. With Cell A near 5.0 on Tasks 2 + 3, there is almost no Likert headroom for Cell B to "improve" within the 1-5 scale. Multiplier hypotheses (H2) become hard to test on tasks where the solo baseline is already near judge ceiling. - **MiniMax max_tokens=1024 (initial config)** caused 3 of 12 judge calls to fail mid-JSON. Bumping 1024 β†’ 3000 (PM-ratified post-second-smoke) yielded 11/12 success across the rest of the pilot. **Recommendation:** keep max_tokens=3000 for any successor MiniMax-judge benchmark. ### 3.3 What the FAIL verdict means The harness as currently implemented does not deliver the multiplier we need on synthesis-class tasks. **Possible causes** (not yet disambiguated): 1. **5-step MAX_STEPS too tight** for Opus on longer-context tasks (Tasks 2 + 3). Raising to 8-10 steps could change H2 reading. 2. **Retrieval fragmentation hurts Qwen** β€” passing retrieved chunks back to the model splits the context vs. a single full-materials prompt. Qwen may reason better over coherent full context than over RAG-fragmented context. 3. **Judge ceiling on solo cells** β€” when solo scores ~4.94, multiplier is bounded by ~0.06 max. H2 reading on Tasks 2 + 3 is dominated by ceiling effect, not real harness capability. 4. **Single-shot prompts already include all materials** β€” for tasks with 3-7 documents fitting comfortably in context window (all 3 pilot tasks did), retrieval-augmented multi-step adds latency + token overhead without information gain. These causes are testable in a re-pilot after the harness is fixed. --- ## 4. Operational findings (binding for successor benchmarks) | Finding | Disposition | |---------|-------------| | MiniMax `max_tokens` = 1024 too tight for dense memo responses | Bump to **3000** for any future MiniMax-judge benchmark | | Loop_exhausted on Cell B/D for longer-context tasks | Raise `MAX_STEPS` to 8-10 (testable via re-pilot) | | Cell A judge ceiling on compact tasks | Use longer / more complex tasks for multiplier benchmarks; or use absolute-difficulty calibration | | Qwen 3.6 via DashScope direct + thinking=on + 16000 tokens = correct synthesis-class config | Inherit for any successor benchmark; binding via amendment v2 Β§5 | | OR-bridge `qwen3.6-35b-a3b-via-openrouter` regresses to Qwen 3.5 | Use only as documented failover; never as primary route | | `INHERITED_CONFIGS_REQUIRE_TASK_TYPE_AUDIT` rule | New binding rule per amendment v2 Β§5; PM commits to honor in future brief authoring | --- ## 5. Halt expansion β€” full N=400 multiplier benchmark NOT authorized Per cc1-brief Β§11 + amendment v2 Β§7, PM action on FAIL is one of three branches: - **Branch A β€” full benchmark NOT authorized:** Expansion would be expensive evidence collection on a known-negative direction. βœ“ **PM SELECTION** - **Branch B β€” full benchmark conditionally authorized after harness fix.** Folded into Branch A logic β€” re-pilot N=12-20 is the prerequisite (see Β§6). - **Branch C β€” pivot to retrieval V2 work.** Folded into the agent-fix sprint (see Β§6) since retrieval improvements would be evaluated in the same re-pilot as harness improvements. **PM selection: Branch A + agent-fix sprint** (per memorandum 2026-04-26 ratification). --- ## 6. Successor work β€” agent-fix sprint authorized PM has authorized a 1-2 week sprint covering: **PRIMARY (multi-step harness pattern fix in `packages/agent/`):** 1. Output normalization layer (`packages/agent/src/output-normalize.ts` β€” new) 2. Model-aware prompt shapes (`packages/agent/src/prompt-shapes/` β€” new directory) 3. Multi-step agent loop unification (consolidate `scripts/run-pilot-2026-04-26.ts` + `benchmarks/harness/src/cells.ts` proxy β†’ `packages/agent/src/agent-loop.ts` as single source of truth) 4. Long-task persistence + recovery (`packages/agent/src/long-task/{checkpoint,recovery,context-manager}.ts` β€” new) 5. Skills + tools audit (`packages/agent/src/*-tools.ts` β€” refactor to model-aware) 6. Failure taxonomy + per-cell + per-model reports (`packages/agent/src/failure-classify.ts` β€” new) 7. Run artifacts + reproducibility (`packages/agent/src/run-meta.ts` β€” new) **SECONDARY (after PRIMARY confirmed PASS via re-pilot):** 8. GEPA evaluation with proper labeled corpus (per-model + multi-model objective) β€” extends existing `packages/agent/src/{iterative-optimizer,evolution-gates,eval-dataset}.ts` **VALIDATION:** Mini re-pilot N=12-20 β€” same 4-cell structure, plus H5 (across-model variance < 0.15 Likert) + H6 (long-task scenario completes). Cost cap $30, halt $25. **Acceptance gates:** - Stage 3 v6 oracle ceiling reproduces 74% (no substrate regression) - Re-pilot H2/H3/H4 PASS on β‰₯ 2/3 tasks - Cross-model variance bounded - Long-task scenario completes on all 3 models without data loss - Zero `thinking_leakage` failures in Qwen output post-normalization - `benchmarks/harness/cells.ts` no longer proxy β€” uses `packages/agent/src/agent-loop.ts` - Pilot wrapper consolidated into `packages/agent/` - Robustness gate blocks candidates that improve one model but degrade another > 2pp --- ## 7. Substrate claim β€” paper claim #1 STANDS This pilot does NOT invalidate paper claim #1 (memory substrate quality). Stage 3 v6 LoCoMo apples-to-apples 74% (oracle ceiling) vs Mem0 published 66.9% remains the binding evidence for memory substrate. See `benchmarks/results/v6-self-judge-rebench/apples-to-apples-memo.md` and commit `b7e19c5`. The pilot was specifically scoped to paper claim #2 (multiplier on agentic knowledge work). That claim requires further work before it can be made. --- ## 8. Artefacts (binding evidence) | Path | Description | |------|-------------| | `benchmarks/results/pilot-2026-04-26/pilot-task-{1,2,3}-{A,B,C,D}.jsonl` | 12 binding cell records | | `benchmarks/results/pilot-2026-04-26/pilot-summary.json` | Re-emitted aggregate (covers all 12 cells) | | `benchmarks/results/pilot-2026-04-26/pilot-run.log` | Continuous log: smoke β†’ restart β†’ final chain | | `benchmarks/results/pilot-2026-04-26/invalidated/` | Original smoke C/D records preserved for audit | | `benchmarks/results/pilot-2026-04-26/prompts-archive/` | Per-cell prompts + multi-step traces | | `briefs/2026-04-26-agentic-knowledge-work-pilot/cc1-brief.md` | Original brief (audit-immutable) | | `briefs/2026-04-26-agentic-knowledge-work-pilot/cc1-brief-amendment-2026-04-26.md` | Amendment v1 | | `briefs/2026-04-26-agentic-knowledge-work-pilot/cc1-brief-amendment-v2-2026-04-26.md` | Amendment v2 | | `scripts/run-pilot-2026-04-26.ts` | Pilot orchestrator (to be unified into `packages/agent/`) | --- ## 9. Audit chain (per amendment v2 Β§8) ``` amendment_v2_doc_sha256 = 1ab5082ff773538a26b3c3294f7fbee4e30063a8d994bdb3753bdc9dd6d6cd99 amendment_v1_doc_sha256 = 3946d3e00fbb1996fb7e63096ecef51abf1e209e5ff166fd0d8758e9a3a14aad cc1_brief_sha256 = 9805adae478333178d36d71b88795afc37f8fb543c2ebccaecb7b01faf06afee judge_rubric_sha256 = 2e24826eb75e92ef1e64055bb2c632eec64ded8fedf7d5b6897ccaec9ffff2eb head_sha (at pilot run) = b7e19c557fdbc42f2d0a3c3213176aa4d790f7a2 manifest_anchor = pilot-2026-04-26-v1 ``` --- **End of verdict. Pilot CLOSED. Successor work scope = agent-fix sprint (PRIMARY β†’ re-pilot β†’ SECONDARY).**