10 KiB
CC-1 Brief — Agentic Knowledge Work Pilot Execution
Date authored: 2026-04-26
Execution authorization: Pending Marko ratification
Pilot ID: agentic-knowledge-work-pilot-2026-04-26
Manifest anchor: pilot-2026-04-26-v1
Estimated wall-clock: 4-6 hours
Cost ceiling: $5.00 hard cap, $4.00 halt
§0 — Substrate readiness gate
Before kickoff, confirm with grep evidence:
- hive-mind retrieval pipeline operational (must support multi-doc ingest + chunked retrieval)
- GEPA agent harness operational at HEAD (verify on commit
<HEAD_SHA>) - LiteLLM gateway reachable for both candidate models (Claude Opus 4.7 + Qwen 3.6 35B-A3B)
- LiteLLM gateway reachable for trio judge (Opus 4.7 + GPT-5.4 + MiniMax M2.7)
- HEAD commit clean working tree (no uncommitted changes that would invalidate reproducibility)
- Pilot folder readable from execution env:
D:\Projects\PM-Waggle-OS\briefs\2026-04-26-agentic-knowledge-work-pilot\
If any of the above fails, halt and ping PM with specifics. Do not proceed with workarounds.
§1 — Goal & rationale
This pilot validates the agentic knowledge work multiplier thesis with a small directional sample (N=3 tasks × 4 cells = 12 candidate runs) before authorizing a full N=400 multiplier benchmark.
The Stage 3 v6 N=400 LoCoMo benchmark proved memory substrate quality (oracle 74% > Mem0 66.9%). That is paper claim #1 — architectural pattern.
This pilot tests paper claim #2 — does adding hive-mind memory + GEPA self-evolve harness lift candidate model performance on real-world knowledge work (CEO synthesis, consultant coordination, executive decision support)?
If pilot PASSES, full N=400 multiplier benchmark is authorized for paper claim #2. If pilot FAILS, expansion halts; resources redirect to retrieval V2 work before retry.
§2 — Hypotheses (pre-registered, not modifiable post-results)
- H2 — Opus multiplier: Cell B (Opus + memory + harness) trio mean > Cell A (Opus solo) trio mean by ≥ 0.30 Likert points, on ≥ 2 of 3 tasks
- H3 — Qwen multiplier: Cell D (Qwen + memory + harness) trio mean > Cell C (Qwen solo) trio mean by ≥ 0.30 Likert points, on ≥ 2 of 3 tasks
- H4 — Sovereignty bridge: Cell D trio mean ≥ Cell A trio mean (Qwen + harness reaches Opus solo) on ≥ 2 of 3 tasks
PILOT PASS = H2 + H3 + H4 each show directional sign on ≥ 2 of 3 tasks AND no critical failures (no cell scoring < 2.0 on majority of judges) PILOT FAIL = otherwise
Anti-pattern reminder: thresholds do not shift post-hoc. Sample size is small; trust the directional sign, not absolute magnitudes.
§3 — Cell specification
| Cell | Model | Memory layer | GEPA harness | Operating mode |
|---|---|---|---|---|
| A | claude-opus-4-7 | OFF | OFF | Single-shot; full materials in context |
| B | claude-opus-4-7 | ON (hive-mind retrieval) | ON | Multi-step agent; materials ingested → retrieval → synthesize |
| C | qwen3.6-35b-a3b | OFF | OFF | Single-shot; full materials in context |
| D | qwen3.6-35b-a3b | ON (hive-mind retrieval) | ON | Multi-step agent; materials ingested → retrieval → synthesize |
Important configuration notes:
- Cell A and C (solo): All materials concatenated into a single user prompt. Single API call. No agent steps. No memory injection.
- Cell B and D (memory + harness): Materials are first ingested into hive-mind as a session corpus. GEPA agent harness then operates with retrieval over this corpus, can re-prompt itself, and produces final response after multi-step process.
- Same final question is asked across all four cells per task (verbatim from task file).
- Same temperature settings: candidate models at
temperature=0.3, top_p=0.9. Judge models attemperature=0for determinism. - Qwen primary route:
qwen3.6-35b-a3b-via-openrouter(DashScope direct) per LOCKED 2026-04-21 routing policy.
§4 — Tasks
Three tasks live in this folder:
| File | Task type | Question to answer |
|---|---|---|
task-1-strategic-synthesis.md |
Multi-document strategic synthesis | "Identify 3 most critical risks for NorthLane Q2-Q4 2026 and propose action plan" |
task-2-cross-thread-coordination.md |
Cross-thread project coordination | "Prepare me for tomorrow's emergency check-in with Diane Mercer" |
task-3-decision-support.md |
Decision support under conflict | "Formulate my CEO decision for next 6 months given three conflicting C-level memos" |
Each task file contains:
- Persona + scenario header
- Question to answer (verbatim)
- All materials (documents/threads/memos)
- Quality expectations note (NOT shown to candidate models or judges — for PM reference only)
Materials extraction for candidate prompts:
- Strip the
## End of materialsblock and everything after it (quality expectations note must NOT leak to candidate) - Concatenate persona + scenario + materials + question into final prompt
- For Cells A/C: pass entire concatenation as single user message
- For Cells B/D: chunk materials into hive-mind session per natural document boundary, then pass persona + question to agent
§5 — Judge ensemble
Judge ensemble locked: Opus 4.7 + GPT-5.4 + MiniMax M2.7
- Same trio used in Stage 3 v6 (κ_trio = 0.7878 calibrated 2026-04-24)
- Each judge scores each cell response on 6 dimensions, Likert 1-5
- Judges are blind to cell configuration (do not include "this is Opus solo" in judge prompt)
- Judges have access to: persona + scenario + question + materials + response only
Full rubric and judge prompt template in judge-rubric.md. Do not modify rubric for execution — copy verbatim into judge calls.
Total judge calls: 12 cells × 3 judges = 36 calls.
§6 — Output
Per-cell JSONL records
One record per cell per task, written to:
D:\Projects\waggle-os\benchmarks\results\pilot-2026-04-26\pilot-{task-id}-{cell-id}.jsonl
Schema in judge-rubric.md §"Output JSONL schema". 12 records total.
Aggregate summary
Single summary file:
D:\Projects\waggle-os\benchmarks\results\pilot-2026-04-26\pilot-summary.json
Schema in judge-rubric.md §"Aggregate summary file".
Run log
Append-only log of execution events to:
D:\Projects\waggle-os\benchmarks\results\pilot-2026-04-26\pilot-run.log
Include: cell start/end timestamps, candidate model latency, judge call latency, cost accumulator, errors, halt events.
§7 — Cost & halt rules
Hard cap: $5.00 cumulative spend (candidate + judge) Halt threshold: $4.00 cumulative — at this threshold, complete current cell + judges, then halt and emit partial summary
Per-call sanity check: any single API call exceeding $0.50 → halt and ping PM (likely runaway agent loop in Cells B/D)
Halt-and-ping triggers (any of these → halt, do not continue without PM):
- Single candidate call >$0.50
- Single judge call >$0.20
- Cumulative spend >$4.00
- Any cell exceeds 90 wall-clock minutes (likely agent loop)
- Any judge returns malformed JSON 3+ times in a row (judge service degraded)
- Any candidate model returns refusal / safety-block (unexpected; investigate before retry)
§8 — Reproducibility
Record at execution time:
- HEAD commit SHA of waggle-os repo
- HEAD commit SHA of hive-mind repo (if extracted by then)
- Manifest anchor string:
pilot-2026-04-26-v1 - Model versions exact (e.g.,
claude-opus-4-7@2026-03-15) - LiteLLM config snapshot
- Random seed:
seed=42for any stochastic component - Full prompt concatenations (per cell, per task) saved to
prompts-archive/subdirectory
This pilot is small enough that exact reproducibility is feasible and required.
§9 — Execution sequence
- Pre-flight (§0 substrate gate) — confirm green
- Record HEAD SHA + manifest anchor
- For each task (1, 2, 3):
- For each cell (A, B, C, D):
- Build prompt per §4 extraction rules
- Call candidate model, capture response + latency + cost
- For each judge (Opus, GPT, MiniMax):
- Build judge prompt per
judge-rubric.mdtemplate - Call judge model, capture verdict + rationale + cost
- Build judge prompt per
- Compute trio mean, strict-pass, critical-fail flags
- Write per-cell JSONL record
- Update cost accumulator; check halt rules
- For each cell (A, B, C, D):
- Compute aggregate summary per
judge-rubric.mdschema - Write summary file + final run log entry
- Ping PM with: pilot verdict (PASS/FAIL), cost, wall-clock, link to summary file
§10 — Open questions for PM ratification
Before CC-1 kicks off, PM should confirm:
- Manifest anchor freeze: Lock
pilot-2026-04-26-v1as anchor string for this pilot (no v2 mid-execution). - Qwen route confirmation: Is
qwen3.6-35b-a3b-via-openrouterstill the live primary route as of 2026-04-26? (Last LOCKED 2026-04-21.) - GEPA harness state: Is GEPA self-evolve currently passing tests at HEAD, or is there a known bug requiring workaround? (If broken, pilot blocks.)
- hive-mind ingest path: Confirm session-scoped corpus ingest is the correct pattern for materials (vs. global memory write). Pilot must not contaminate other test data.
- Judge cost reality check: Stage 3 v6 trio averaged ~$0.07 per judge call. 36 calls = ~$2.52. Plus 12 candidate calls (Opus dominates). Total estimated ~$3.50-4.50. Confirms $5 cap is realistic but tight; halt at $4 is correct buffer.
§11 — Post-execution PM actions
After CC-1 emits pilot summary:
- PM reads summary file, validates all 12 cells executed, no critical failures
- PM drafts go/no-go memo for full N=400 multiplier benchmark:
- If PASS → authorize full benchmark with cost cap, model roster, scope
- If FAIL → halt expansion, draft retrieval V2 priority brief
- Marko ratifies decision
- Memory updated with pilot result + decision
§12 — Notes
- This is a direction validator, not a paper claim. Sample size is too small for publication-grade evidence.
- Full N=400 multiplier benchmark (post-pilot, if PASS) will be the publication-grade evidence. That benchmark will use the same task design pattern but with N=400 task instances and broader model coverage (Opus + Qwen + GPT-5.4).
- Pilot results are internal-only. No external comms triggered by pilot pass/fail.
- Pilot folder lives in PM-Waggle-OS, results live in waggle-os/benchmarks/results — standard separation of brief vs. execution artifacts.