This commit is contained in:
Oleg Maslov
@@ -0,0 +1,64 @@
|
||||
# Agentic Knowledge Work Pilot — N=3 Direction Validator
|
||||
|
||||
**Date:** 2026-04-26
|
||||
**Type:** Pilot test (pre-full multiplier benchmark gate)
|
||||
**Owner:** PM authoring brief, CC-1 executing
|
||||
**Scope:** 3 tasks × 4 cells = 12 candidate runs + 36 judge calls (trio ensemble)
|
||||
**Cost ceiling:** $5 hard cap, $4 halt
|
||||
**Time budget:** 4-6 hours wall-clock
|
||||
|
||||
## Why this pilot exists
|
||||
|
||||
Stage 3 v6 N=400 LoCoMo proved the **memory substrate** thesis (oracle 74% > Mem0 66.9%). That's paper claim #1 — architecture beats peer-reviewed baseline on memory recall.
|
||||
|
||||
This pilot is paper claim #2 — **agentic knowledge work multiplier**. Question: does adding hive-mind memory + GEPA agent harness lift candidate model performance on real CEO/consultant work, not just memory recall?
|
||||
|
||||
PA V5 (April 2026) gave H1 PASS Opus 4.6 +5.2pp on knowledge work but on small N. This pilot replicates direction signal on N=3 across 4 cells. If pilot passes (H2/H3/H4 directional signs hold), full N=400 multiplier benchmark is authorized for paper claim #2 evidence.
|
||||
|
||||
If pilot fails (any of H2/H3/H4 reverses sign), we don't waste $150 on full benchmark — we go back to retrieval V2 work first.
|
||||
|
||||
## Files in this folder
|
||||
|
||||
| File | Purpose | Audience |
|
||||
|---|---|---|
|
||||
| `README.md` | This index — overview + sequencing | Marko, PM, CC-1 |
|
||||
| `cc1-brief.md` | Technical execution brief | CC-1 primary |
|
||||
| `task-1-strategic-synthesis.md` | Multi-document synthesis test materials | CC-1, judges |
|
||||
| `task-2-cross-thread-coordination.md` | Cross-thread project coordination test | CC-1, judges |
|
||||
| `task-3-decision-support.md` | Decision support under conflict test | CC-1, judges |
|
||||
| `judge-rubric.md` | Likert 1-5 × 6 dimensions trio rubrika | Judge ensemble |
|
||||
|
||||
## Hypotheses pilot validates
|
||||
|
||||
- **H2:** Opus 4.7 + memory + harness > Opus 4.7 solo (multiplier on frontier model)
|
||||
- **H3:** Qwen 3.6 35B-A3B + memory + harness > Qwen solo (multiplier on sovereign model)
|
||||
- **H4:** Qwen + memory + harness ≥ Opus solo (SOTA-on-local proof, sovereignty bridge)
|
||||
|
||||
PASS criteria (binary):
|
||||
- All 3 hypotheses show correct directional sign across ≥ 2 of 3 tasks (6/9 cells minimum)
|
||||
- No catastrophic failure (any cell scoring < 2.0/5 overall on majority of judges)
|
||||
|
||||
If PASS → green-light full N=400 multiplier benchmark (Opus + Qwen + GPT-5.4 × 4 cells × N=400)
|
||||
If FAIL → halt expansion, prioritize retrieval V2 work, schedule pilot retry post-V2
|
||||
|
||||
## Cost & time envelope
|
||||
|
||||
- Candidate model spend: ~$1.50 (12 runs, Opus dominates cost)
|
||||
- Judge ensemble spend: ~$2.50 (36 calls × ~$0.07/call across Opus + GPT + MiniMax)
|
||||
- Buffer: ~$1.00
|
||||
- Total ceiling: $5.00, halt at $4.00
|
||||
- Wall-clock target: 4-6 hours (parallel cell execution where possible)
|
||||
|
||||
## Sequencing
|
||||
|
||||
1. **PM** (you, now): generates pilot package — this folder
|
||||
2. **Marko**: ratifies brief (1 review pass, optional adjustments)
|
||||
3. **CC-1**: executes pilot — kicks runner, monitors halt rules, produces JSONL + summary
|
||||
4. **PM**: adjudicates direction signal post-results, drafts go/no-go for full benchmark
|
||||
5. **Marko**: ratifies go/no-go decision
|
||||
|
||||
## Notes on synthetic materials
|
||||
|
||||
All test materials in tasks 1-3 are **synthetic but realistic**, designed to mirror Marko's ICP work (CEO of mid-stage SaaS company, boutique consulting Partner, executive decision-maker). Documents are detailed enough to require genuine synthesis, not surface-level pattern matching.
|
||||
|
||||
Synthetic ≠ proxy. Each task has a clear "right answer shape" the judge rubric calibrates against — not a single correct answer, but a quality bar a real CEO/Partner would recognize as professional output.
|
||||
@@ -0,0 +1,223 @@
|
||||
# CC-1 Brief Amendment — Agentic Knowledge Work Pilot
|
||||
## (binding for execution; supersedes original cc1-brief.md where in conflict)
|
||||
|
||||
**Date authored:** 2026-04-26
|
||||
**Authority:** PM-RATIFY (this date) — 4 decisions on §0.1 / §10.4 / §0.5 / §10.5
|
||||
**Predecessor (audit-immutable):** [`cc1-brief.md`](cc1-brief.md) — unmodified
|
||||
**Manifest anchor:** `pilot-2026-04-26-v1` — UNCHANGED (no v2 mid-flight)
|
||||
**Pilot ID:** `agentic-knowledge-work-pilot-2026-04-26` — UNCHANGED
|
||||
**Wall-clock budget:** **7-10 hours** (was 4-6h; +3-4h absorbs orchestrator scaffolding)
|
||||
**Cost ceiling:** **$7.00 hard cap, $6.00 halt** (was $5/$4 — see §6 below for rationale)
|
||||
|
||||
---
|
||||
|
||||
## §1 — Cell B/D renamed definition
|
||||
|
||||
**Original (cc1-brief.md §3):** "candidate model + hive-mind retrieval + GEPA self-evolve harness"
|
||||
|
||||
**Amended (binding):** **"candidate model + hive-mind session corpus + multi-step agent loop with retrieval-augmented self-prompting"**
|
||||
|
||||
True GEPA self-evolve (iterative prompt optimization on a labeled training corpus) is **deferred to the full N=400 multiplier benchmark**. For this pilot, "self-evolve" = the agent loop's ability to propose its own intermediate questions and integrate retrieved context across steps before producing a final response.
|
||||
|
||||
| Cell | Model | Memory layer | Operating mode |
|
||||
|---|---|---|---|
|
||||
| A | claude-opus-4-7 | OFF | Single-shot; full materials in user prompt; one API call |
|
||||
| B | claude-opus-4-7 | ON (per-task in-tree session) | **Multi-step agent loop** (see §2) over a per-task `SessionStore` corpus |
|
||||
| C | qwen3.6-35b-a3b-via-openrouter | OFF | Single-shot; full materials in user prompt; one API call |
|
||||
| D | qwen3.6-35b-a3b-via-openrouter | ON (per-task in-tree session) | **Multi-step agent loop** (see §2) over a per-task `SessionStore` corpus |
|
||||
|
||||
Same final question across all 4 cells per task (verbatim from task file). Same temperature settings (candidates 0.3, judges 0). PM-only quality-notes block stripped from materials before passing to candidates and judges.
|
||||
|
||||
---
|
||||
|
||||
## §2 — Per-task agent loop specification (Cells B/D)
|
||||
|
||||
The orchestrator implements a fixed-budget retrieval-augmented loop:
|
||||
|
||||
```
|
||||
For each (task, cell ∈ {B, D}):
|
||||
1. Create a fresh per-task SessionStore (isolation: prevents task-1 corpus contaminating task-2)
|
||||
2. Ingest task materials as MemoryFrames into that session
|
||||
(one frame per natural document boundary; chunk only if a single doc > 16KB)
|
||||
3. Initialize agent loop with: persona + scenario + question (NOT the materials)
|
||||
4. Loop up to MAX_STEPS = 5:
|
||||
a. Candidate proposes either:
|
||||
- intermediate retrieval question (signaled by structured output) OR
|
||||
- final response (signaled by structured output)
|
||||
b. If retrieval question:
|
||||
- Call HybridSearch on the task's SessionStore (top-K = 8)
|
||||
- Inject retrieved frames into next-turn context
|
||||
- Continue loop
|
||||
c. If final response:
|
||||
- Capture as candidate_response
|
||||
- Exit loop
|
||||
5. If loop exhausts MAX_STEPS without final response:
|
||||
- Force-finalize on step 5 with all accumulated context
|
||||
- Tag record with `loop_exhausted: true` for diagnostic
|
||||
6. Per-cell halt: if cumulative cell spend > $0.50, halt cell, ping PM
|
||||
```
|
||||
|
||||
**Hard limits per cell** (Cells B/D only; Cells A/C are single-shot):
|
||||
- `MAX_STEPS = 5` (model proposes ≤5 intermediate-or-final outputs)
|
||||
- `MAX_RETRIEVALS_PER_STEP = 8` (HybridSearch top-K bound)
|
||||
- `PER_CELL_HARD_HALT = $0.50` (per cc1-brief §7; reaffirmed)
|
||||
- `loop_exhausted` flag in JSONL record if step 5 ran without natural finalization
|
||||
|
||||
---
|
||||
|
||||
## §3 — Wrapper script location and structure
|
||||
|
||||
**File:** `D:/Projects/waggle-os/scripts/run-pilot-2026-04-26.ts`
|
||||
|
||||
**Reuse pattern:** Follows the same wrapper-around-runner shape as `scripts/run-mini-locomo.ts` for consistency, but is a SEPARATE script (no shared mutable state; the LoCoMo wrapper is unchanged and remains §11-frozen for any future LoCoMo work).
|
||||
|
||||
**Top-level structure:**
|
||||
1. `parseArgs(argv)` — flags: `--task <id>`, `--cell <A|B|C|D>`, `--all-cells`, `--smoke`, `--dry-run`, `--manifest-anchor`
|
||||
2. `loadTaskMaterials(taskFile)` — reads task-N.md from pilot folder, strips `## End of materials` block
|
||||
3. `runCellSolo(cell, taskMaterials, model)` — single-shot path for A/C
|
||||
4. `runCellMultiStep(cell, taskMaterials, model)` — agent loop path for B/D per §2
|
||||
5. `judgeWithTrio(cellResponse, taskContext, judgeRubric)` — emits 3 judge calls + computes trio_mean / strict_pass / critical_fail
|
||||
6. `writeJsonlRecord(...)` — emits the §6/judge-rubric.md schema record
|
||||
7. Cost accumulator + halt-rule enforcement
|
||||
|
||||
**Atomic JSONL writes** (per-cell): each cell completes → record flushed to `pilot-{task}-{cell}.jsonl` before moving to next cell. Comp-restart-resilient at the cell granularity (per-task agent loop is NOT restart-resilient mid-loop; if a comp restart happens mid-Cell-B/D, that cell is re-run from scratch).
|
||||
|
||||
**Reusability for full N=400 multiplier benchmark:** the orchestrator is parameterized on `(model, cell, task)` and accepts a task-list config — running 400 instances becomes a config change, not a rewrite. PM treats this scaffolding as investment not sunk cost (per ratification).
|
||||
|
||||
---
|
||||
|
||||
## §4 — Hive-mind in-tree import path (precise)
|
||||
|
||||
The pilot uses the **in-tree** memory substrate from `D:/Projects/waggle-os/packages/core/`. No `@hive-mind/*` npm install required for this pilot. The extracted `D:/Projects/hive-mind/` (HEAD `c363257`, tag `v0.1.0`) shares lineage with these files but is NOT consumed at pilot runtime.
|
||||
|
||||
**Imports the wrapper script will use:**
|
||||
|
||||
```typescript
|
||||
// from packages/core/src/index.ts (verified exports as of HEAD b7e19c5):
|
||||
import {
|
||||
MindDB, // packages/core/src/mind/db.ts — SQLite + sqlite-vec backing store
|
||||
FrameStore, // packages/core/src/mind/frames.ts:35 — class FrameStore (ingest)
|
||||
SessionStore, // packages/core/src/mind/sessions.ts:13 — class SessionStore (per-task scoping)
|
||||
HybridSearch, // packages/core/src/mind/search.ts:36 — class HybridSearch (FTS5 + vec0 RRF retrieval)
|
||||
type MemoryFrame, // frames.ts:8
|
||||
type FrameType, // frames.ts:4 — 'I' | 'P' | 'B'
|
||||
type Session, // sessions.ts:3
|
||||
type SearchResult, // search.ts:23
|
||||
createLiteLLMEmbedder, // packages/core/src/mind/litellm-embedder.ts — for the embedding side
|
||||
} from '@waggle/core';
|
||||
|
||||
// from packages/agent/src/index.ts:
|
||||
import {
|
||||
runAgentLoop, // packages/agent/src/agent-loop.ts:83 — generic agent loop entry
|
||||
// ...other agent imports as needed by the orchestrator (tool-filter, etc.)
|
||||
} from '@waggle/agent';
|
||||
```
|
||||
|
||||
**Per-task isolation pattern:** create a fresh `SessionStore` rooted on a tmp `MindDB` per task → ingest materials → instantiate `HybridSearch` against that store → use as the retrieval backend for the agent loop. After task completes, drop the session (next task gets a fresh DB; no cross-contamination).
|
||||
|
||||
**Scratch-DB location:** `tmp/pilot-2026-04-26/per-task-{task-id}.sqlite` (deleted after each task once results are persisted; gitignored under existing `tmp/` rule).
|
||||
|
||||
**Documentation requirement (per PM ratification):** the final pilot report MUST state: *"Pilot used the in-tree memory substrate at `packages/core/src/mind/` (HEAD `b7e19c5`). Production = extracted `@hive-mind/core@0.1.0` (same lineage)."*
|
||||
|
||||
---
|
||||
|
||||
## §5 — Manifest anchor scope notes
|
||||
|
||||
`pilot-2026-04-26-v1` declares the following items **frozen and pilot-irrelevant**:
|
||||
|
||||
```
|
||||
INERT_UNTRACKED_AT_KICK:
|
||||
- preflight-results/b2-grok-smoke-2026-04-21T23-01-41-021Z.json
|
||||
- scripts/smoke-binary.py
|
||||
- tmp/
|
||||
|
||||
These items pre-exist across S2/Phase C/today's commits, are not pilot-related,
|
||||
and are NOT staged. The pilot orchestrator must NOT write to any of these paths.
|
||||
If the orchestrator writes to any of these paths, halt + ping PM (would invalidate
|
||||
the reproducibility claim).
|
||||
|
||||
PILOT-WRITES-ALLOWED (sandboxed):
|
||||
- benchmarks/results/pilot-2026-04-26/ ← pilot output dir
|
||||
- benchmarks/results/pilot-2026-04-26/prompts-archive/ ← per-cell prompts
|
||||
- tmp/pilot-2026-04-26/ ← scratch SessionStore SQLite per task
|
||||
- tmp/pilot-2026-04-26/run.log ← run log mirror
|
||||
|
||||
PILOT-WRITES-FORBIDDEN:
|
||||
- any path outside the two roots above
|
||||
- any file in §11-frozen path list (LoCoMo wrapper, runner.ts, etc.)
|
||||
```
|
||||
|
||||
**HEAD at kick** will be re-verified in the orchestrator preamble; recorded in every JSONL record's `head_sha` field per `judge-rubric.md` schema.
|
||||
|
||||
---
|
||||
|
||||
## §6 — Cost ceiling update
|
||||
|
||||
**Original (cc1-brief.md §7):** $5.00 hard cap / $4.00 halt.
|
||||
|
||||
**Amended (binding):** **$7.00 hard cap / $6.00 halt.**
|
||||
|
||||
**Rationale (PM-ratified):**
|
||||
|
||||
| Bucket | Estimate | Notes |
|
||||
|--------|----------|-------|
|
||||
| Cell A/C (Opus + Qwen solo, 6 cells) | $0.90 | Single-shot, ~$0.10-0.20 per cell |
|
||||
| Cell B/D (multi-step agent loop, 6 cells) | $2.40 | ~$0.30-0.50 per cell × 6 (within per-cell $0.50 hard halt) |
|
||||
| Trio judge (36 calls × ~$0.07) | $2.52 | Same trio + per-call cost as Stage 3 v6 final |
|
||||
| Buffer | $1.18 | For unexpected token bloat or retry cost |
|
||||
| **Total cap** | **$7.00** | |
|
||||
| **Halt threshold** | **$6.00** | Complete current cell + judges, then halt + emit partial summary |
|
||||
|
||||
**Per-call sanity check unchanged:** any single API call exceeding $0.50 → halt + ping PM (likely runaway agent loop).
|
||||
|
||||
**Per-cell hard halt unchanged:** $0.50 (Cells B/D specifically).
|
||||
|
||||
---
|
||||
|
||||
## §7 — Execution sequence (amended)
|
||||
|
||||
1. **Pre-flight** (§0 substrate gate per cc1-brief.md): re-verify just before kick (HEAD, LiteLLM, hive-mind in-tree, GEPA tests = 121/121 confirmed).
|
||||
2. **Build orchestrator wrapper** at `scripts/run-pilot-2026-04-26.ts` (3-4h scaffolding).
|
||||
3. **Smoke test on Task 1 only** — all 4 cells (A, B, C, D). Emit Task-1 JSONL records + a smoke-summary stub. **HALT + PM verification before continuing.**
|
||||
4. **PM verifies smoke** — confirms (a) all 4 cells executed cleanly, (b) judge ensemble responses are well-formed, (c) cost trajectory is on track, (d) no unexpected halts.
|
||||
5. **Run remaining 8 cells** (Tasks 2 + 3 × cells A-D) under PM go-ahead.
|
||||
6. **Emit `pilot-summary.json`** per `judge-rubric.md` schema + final run log entry.
|
||||
7. **Halt ping to PM** with: pilot verdict (PASS/FAIL per §2 of cc1-brief.md), total cost, wall-clock, link to summary file.
|
||||
|
||||
**No Gate D auto-advance.** PM drafts go/no-go for full N=400 multiplier benchmark; Marko ratifies.
|
||||
|
||||
---
|
||||
|
||||
## §8 — What is NOT changed by this amendment
|
||||
|
||||
The following sections of `cc1-brief.md` remain in force verbatim:
|
||||
|
||||
- §1 (Goal & rationale)
|
||||
- §2 (Hypotheses — pre-registered, not modifiable post-results)
|
||||
- §4 (Tasks — same task-1/2/3 files; same materials; same final questions)
|
||||
- §5 (Judge ensemble — locked: Opus + GPT + MiniMax)
|
||||
- §6 (Output schema — per `judge-rubric.md`)
|
||||
- §7 (Halt-and-ping triggers — except cost cap raised to $7/$6 per §6 above)
|
||||
- §8 (Reproducibility — recording HEAD SHA, manifest anchor, model versions, prompts archive)
|
||||
- §11 (Post-execution PM actions)
|
||||
- §12 (Pilot is direction validator only; full benchmark is publication-grade)
|
||||
|
||||
The `judge-rubric.md` document is **NOT modified** by this amendment.
|
||||
|
||||
---
|
||||
|
||||
## §9 — Audit trail
|
||||
|
||||
This amendment is the binding execution document. The unmodified `cc1-brief.md` is preserved as the audit-immutable predecessor. Every JSONL record produced by the pilot includes `manifest_anchor: "pilot-2026-04-26-v1"`; the orchestrator writes a top-of-log line:
|
||||
|
||||
```
|
||||
[pilot] amendment_doc_sha256 = <sha of this file>
|
||||
[pilot] cc1_brief_sha256 = <sha of cc1-brief.md>
|
||||
[pilot] head_sha = <git HEAD at kick>
|
||||
```
|
||||
|
||||
Both file SHAs are committed in the pilot result commit body for tamper-evident audit.
|
||||
|
||||
---
|
||||
|
||||
**End of amendment. PM verification requested before orchestrator scaffolding begins.**
|
||||
@@ -0,0 +1,164 @@
|
||||
# CC-1 Brief Amendment v2 — Agentic Knowledge Work Pilot
|
||||
## (binding for Cells C/D restart + Tasks 2/3 execution; supplements amendment v1)
|
||||
|
||||
**Date authored:** 2026-04-26 (post-smoke audit)
|
||||
**Authority:** PM-RATIFY-AUDIT-OPTION-B-AND-AMENDMENT-V2 (this date)
|
||||
**Predecessor (audit-immutable):** [`cc1-brief.md`](cc1-brief.md) — unchanged
|
||||
**Sibling (audit-immutable):** [`cc1-brief-amendment-2026-04-26.md`](cc1-brief-amendment-2026-04-26.md) — amendment v1, unchanged
|
||||
**Manifest anchor:** `pilot-2026-04-26-v1` — **UNCHANGED** (no v2 manifest anchor; both amendments share v1 anchor)
|
||||
**Pilot ID:** `agentic-knowledge-work-pilot-2026-04-26` — UNCHANGED
|
||||
**Wall-clock budget:** **7-10 hours** (inherited from amendment v1 §6)
|
||||
|
||||
---
|
||||
|
||||
## §1 — Trigger
|
||||
|
||||
Smoke audit (Task 1, all 4 cells, executed 2026-04-26T00:43:55Z → 00:50:44Z) revealed two methodology gaps in amendment v1 §1:
|
||||
|
||||
**Gap 1 — alias bridge regression to Qwen 3.5:** Amendment v1 §1 named `qwen3.6-35b-a3b-via-openrouter` as the primary Qwen alias. Per `litellm-config.yaml` comment block (verbatim): *"OpenRouter bridge — failover when DashScope rate-limits or is unavailable, caller-side retry should fall back to qwen3.6-35b-a3b-via-openrouter (bridge route, **one-minor regress to 3.5 until OR carries 3.6**)"*. The bridge alias actually routes to `openrouter/qwen/qwen3.5-35b-a3b` — Qwen 3.5, not 3.6. Smoke Cells C/D ran on Qwen 3.5, NOT the Qwen 3.6 the brief intended.
|
||||
|
||||
**Gap 2 — wrapper default `max_tokens=4096`:** Amendment v1 did not specify a Qwen `max_tokens` ceiling. Wrapper default was 4096. This is well below the Sprint 10 LOCK lower bound (16000) and the Sprint 11 OVERRIDE (64000). Stage 3 v6 LoCoMo apples-to-apples 74% result was generated with `max_tokens=64000`. While smoke Cell C/D responses completed naturally (no truncation observed), reasoning headroom may have been silently constrained.
|
||||
|
||||
**Inheritance gap source:** Amendment v1 §1 named the OR-bridge alias likely by copy-paste from Stage 3 v6 §5.1 fallback list (where the OR-bridge IS the failover entry), not from v6 primary route which is `qwen3.6-35b-a3b-via-dashscope-direct` + thinking=on + max_tokens=64000. PM brief authoring did not cross-reference the actual config that produced the v6 published result.
|
||||
|
||||
**Effect on smoke H3/H4 directional reading:** Cell C trio_mean=4.167 vs Cell D trio_mean=3.944 (H3 Δ=−0.222) and Cell D vs Cell A (H4 Δ=−0.555) cannot be cleanly attributed. Confounds: (a) wrong model class (3.5 vs 3.6), (b) potentially constrained reasoning (4096 vs 16000-64000 cap), (c) residual real signal that harness hurts Qwen on synthesis. Re-run with corrected config required to disambiguate.
|
||||
|
||||
---
|
||||
|
||||
## §2 — Explicit Qwen config (verbatim, audit-verified)
|
||||
|
||||
The following config supersedes amendment v1 §1 / §3 specifications for ALL Qwen calls in this pilot from amendment v2 ratification forward (Cells C/D Task 1 restart + Cells C/D Tasks 2 + 3):
|
||||
|
||||
```
|
||||
alias: qwen3.6-35b-a3b-via-dashscope-direct
|
||||
(verified at litellm-config.yaml lines 410-415; routes to
|
||||
openai/qwen3.6-35b-a3b via DashScope intl tenant)
|
||||
thinking: ON (explicit parameter — wrapper MUST pass enable_thinking
|
||||
decision intentionally; do NOT rely on Qwen default
|
||||
behavior since defaults vary across providers and
|
||||
model versions)
|
||||
max_tokens: 16000
|
||||
temperature: 0.3
|
||||
```
|
||||
|
||||
**Rationale (PM-stated, verbatim from ratification):** *"Stage 3 v6 LoCoMo apples-to-apples 74% result was generated with this exact config (Sprint 11 OVERRIDE ratified 2026-04-22). Synthesis tasks require equivalent reasoning headroom; Sprint 10 LoCoMo factoid LOCK (thinking=off, 16000) does NOT generalize to synthesis class."*
|
||||
|
||||
**Why 16000 and not 64000:** PM explicitly chose 16000 to keep per-cell spend safely under the per-cell hard halt (raised to $1.00 in §4 below). 64000 is technically higher-fidelity (matches Stage 3 v6 verbatim) but the marginal reasoning depth gain is judged not worth the per-cell halt risk. 16000 is the Sprint 10 LOCK lower bound that achieved 5/5-safe convergence on all 5 LoCoMo prompt shapes — sufficient headroom for synthesis.
|
||||
|
||||
**Implementation requirement (orchestrator):** the wrapper must pass `extra_body.enable_thinking: true` (NOT omit it) and `max_tokens: 16000` (NOT the default 4096) on every Qwen subject call (Cells C and D). Judge calls remain unchanged (judges run thinking=off per amendment v1 §3, max_tokens=1024).
|
||||
|
||||
---
|
||||
|
||||
## §3 — Retroactive scope note
|
||||
|
||||
**Original smoke Task 1 — partial invalidation:**
|
||||
|
||||
| Cell | Original status | Disposition under amendment v2 |
|
||||
|------|-----------------|--------------------------------|
|
||||
| A — Opus solo | Wrote `pilot-task-1-A.jsonl` (trio_mean=4.50, 2-judge fallback after MiniMax JSON-parse failure) | **RETAIN candidate response.** Surgical MiniMax judge retry authorized in §3.1 below; if retry succeeds, JSONL record updated to full-trio. |
|
||||
| B — Opus + memory + harness | Wrote `pilot-task-1-B.jsonl` (trio_mean=4.94, full trio) | **RETAIN.** Opus model unaffected by Qwen alias bug; max_tokens=4096 was sufficient (response completed naturally). |
|
||||
| C — Qwen solo | Wrote `pilot-task-1-C.jsonl` (trio_mean=4.17, full trio, on Qwen 3.5 via OR bridge) | **INVALIDATED.** Discard from final pilot summary. Restart with §2 config. |
|
||||
| D — Qwen + memory + harness | Wrote `pilot-task-1-D.jsonl` (trio_mean=3.94, full trio, on Qwen 3.5 via OR bridge) | **INVALIDATED.** Discard from final pilot summary. Restart with §2 config. |
|
||||
|
||||
**Original JSONL files preserved on disk** for audit (not deleted). Final `pilot-summary.json` will reference only the binding records: A (potentially with MiniMax retry merged), B (original), C (restarted), D (restarted), then Tasks 2 + 3 cells (all 8 with §2 config).
|
||||
|
||||
**§3.1 — Cell A MiniMax surgical retry:**
|
||||
|
||||
The Opus candidate response for Cell A is correct (Opus model not affected by alias bug). Only the MiniMax judge call returned malformed JSON on all 3 retries. Authorized action: re-run JUST the MiniMax judge call against the existing Cell A `candidate_response`, no candidate re-call.
|
||||
|
||||
- If retry succeeds: update `pilot-task-1-A.jsonl` `judge_minimax` field with new verdict; recompute `trio_mean`, `trio_strict_pass`, `trio_critical_fail` accordingly. Append a `judge_minimax_retried_at` timestamp field.
|
||||
- If retry fails again (3 more retries malformed): retain 2-judge fallback as the binding record. Add explicit note to `pilot-summary.json` aggregate explaining the partial-trio cell.
|
||||
- Estimated cost: ~$0.07 (one MiniMax call against ~6KB candidate response + materials context).
|
||||
|
||||
---
|
||||
|
||||
## §4 — Cost ceiling (REVISED per PM update 2026-04-26)
|
||||
|
||||
| Item | Original (v1) | Revised (v2) |
|
||||
|------|---------------|---------------|
|
||||
| Hard cap | $7.00 | **$20.00** |
|
||||
| Halt threshold | $6.00 | **$17.00** |
|
||||
| Per-cell hard halt | $0.50 | **$1.00** |
|
||||
| Per-call sanity (single judge or candidate) | $0.50 (sanity ping) | **$0.40 (hard halt + ping)** |
|
||||
| Estimated cumulative through pilot completion | — | **$5.50-6.50 (unchanged from prior estimate; raised cap is buffer not target)** |
|
||||
|
||||
**Halt-and-ping rules (binding):**
|
||||
- Any single API call > $0.40 → halt + ping PM (was $0.50 sanity ping; now hard halt at lower threshold)
|
||||
- Any cell cumulative > $1.00 → halt + ping PM (was $0.50)
|
||||
- Cumulative > $17.00 → halt + emit partial summary + ping PM (was $6.00)
|
||||
- Any cell exceeds 90 wall-clock minutes → halt + ping PM (unchanged)
|
||||
- Any judge returns malformed JSON 3+ times in row → halt + ping PM (unchanged from amendment v1 §7)
|
||||
- Any candidate model returns refusal / safety-block → halt + ping PM (unchanged)
|
||||
|
||||
**Rationale (PM-stated, verbatim from ratification):** *"methodology correctness (Qwen DashScope direct + thinking=on + 16000 tokens) takes priority over cost tightness; original $7 cap was authored before audit revealed config inheritance gap; raised cap removes pressure to optimize for cost over reasoning headroom."*
|
||||
|
||||
The raised cap is BUFFER not TARGET. Expected cumulative remains $5.50-6.50. The raise exists so the orchestrator does not silently constrain Qwen reasoning depth to stay under a tight budget. If the methodology requires it, spending the buffer is correct; if methodology does not require it, expected spend stays well under raise.
|
||||
|
||||
---
|
||||
|
||||
## §5 — Manifest scope note appended
|
||||
|
||||
**INHERITED_CONFIGS_REQUIRE_TASK_TYPE_AUDIT** (binding rule, future PM brief authoring):
|
||||
|
||||
Any future benchmark inheriting alias / thinking-mode / max_tokens / temperature config from a prior sprint LOCK MUST verify task-type taxonomy match between the source LOCK context and the target benchmark. Specifically:
|
||||
|
||||
- **LoCoMo factoid task** (single-fact recall, multi-anchor enumeration, chain-of-anchor, temporal-scope, null-result-tolerant) ≠ **synthesis / agentic knowledge work** (cross-document strategic memo, multi-thread coordination, multi-stakeholder decision support).
|
||||
- LoCoMo factoid LOCKs (e.g., Sprint 10 Task 1.1 `thinking=off, max_tokens=16000`) **DO NOT generalize** to synthesis class.
|
||||
- Brief author MUST either (a) explicitly justify config inheritance per task-type-match argument, OR (b) specify config from scratch with task-type-appropriate rationale.
|
||||
|
||||
This rule binds:
|
||||
- Future PM brief authoring for any benchmark touching Qwen (and by extension, any reasoning-class model with mode toggles)
|
||||
- The full N=400 multiplier benchmark (post-pilot, if PASS)
|
||||
- Any v3 or successor amendments to existing benchmarks
|
||||
|
||||
**Anti-pattern this rule addresses:** copy-paste of model alias from a fallback-route list in a different sprint's manifest, without verifying the alias resolves to the intended model class.
|
||||
|
||||
---
|
||||
|
||||
## §6 — Anchor unchanged
|
||||
|
||||
`pilot-2026-04-26-v1` remains the manifest anchor. Amendment v2 SUPPLEMENTS amendment v1 without superseding the anchor. Both amendments + the original `cc1-brief.md` form the binding execution document set. The orchestrator records all three SHA-256s in the run log preamble and the pilot result commit body.
|
||||
|
||||
In conflict resolution: v2 binds over v1 binds over original cc1-brief.md (specific overrides general; latest binds). Where v2 is silent, v1 governs. Where v1 is silent, original brief governs.
|
||||
|
||||
---
|
||||
|
||||
## §7 — Restart sequence (binding on PM amendment-v2 verification)
|
||||
|
||||
1. **PM verifies amendment v2 text + §2 config block** (this step in flight).
|
||||
2. **CC-1 updates orchestrator** to support §2 config:
|
||||
- New CLI flag `--qwen-alias <alias>` defaulting to `qwen3.6-35b-a3b-via-dashscope-direct`
|
||||
- New CLI flag `--qwen-max-tokens <int>` defaulting to `16000`
|
||||
- Explicit `extra_body.enable_thinking: true` for Qwen subject calls (NOT relying on default)
|
||||
- New CLI flag `--retry-cell-a-minimax` for Cell A MiniMax surgical retry
|
||||
- New CLI flag `--restart-cells` accepting cell IDs to re-run (e.g., `--restart-cells C,D`)
|
||||
3. **CC-1 runs:**
|
||||
- Cell A MiniMax surgical retry
|
||||
- Cell C Task 1 restart (Qwen 3.6 DashScope direct, thinking=on, max_tokens=16000)
|
||||
- Cell D Task 1 restart (same config)
|
||||
4. **CC-1 emits second smoke verification ping** with: 4 Task 1 records (A retained-with-or-without-MiniMax-update, B retained, C fresh, D fresh), Cell A MiniMax retry outcome, cumulative cost.
|
||||
5. **PM verifies second smoke** per same 10-item criteria as first smoke + H3/H4 directional reading on corrected config.
|
||||
6. **PM authorizes Tasks 2 + 3** (8 remaining cells, all under §2 Qwen config for cells C/D).
|
||||
7. **CC-1 emits final pilot summary** + pilot result commit + halt ping.
|
||||
|
||||
PM does not need to re-verify amendment v2 text after CC-1 emits restart results; v2 text is locked by this round of verification.
|
||||
|
||||
---
|
||||
|
||||
## §8 — Audit SHA capture in pilot result commit body
|
||||
|
||||
The orchestrator MUST record in run log preamble AND pilot result commit body:
|
||||
|
||||
```
|
||||
amendment_v2_doc_sha256 = <sha of cc1-brief-amendment-v2-2026-04-26.md>
|
||||
amendment_v1_doc_sha256 = <sha of cc1-brief-amendment-2026-04-26.md>
|
||||
cc1_brief_sha256 = <sha of cc1-brief.md>
|
||||
judge_rubric_sha256 = <sha of judge-rubric.md>
|
||||
head_sha = <git HEAD at restart kick>
|
||||
```
|
||||
|
||||
All five SHAs together form the binding execution document tamper-evident chain.
|
||||
|
||||
---
|
||||
|
||||
**End of amendment v2. PM verification of v2 SHA requested before Cells C/D Task 1 restart.**
|
||||
217
docs/briefs/2026-04-26-agentic-knowledge-work-pilot/cc1-brief.md
Normal file
217
docs/briefs/2026-04-26-agentic-knowledge-work-pilot/cc1-brief.md
Normal file
@@ -0,0 +1,217 @@
|
||||
# CC-1 Brief — Agentic Knowledge Work Pilot Execution
|
||||
|
||||
**Date authored:** 2026-04-26
|
||||
**Execution authorization:** Pending Marko ratification
|
||||
**Pilot ID:** `agentic-knowledge-work-pilot-2026-04-26`
|
||||
**Manifest anchor:** `pilot-2026-04-26-v1`
|
||||
**Estimated wall-clock:** 4-6 hours
|
||||
**Cost ceiling:** $5.00 hard cap, $4.00 halt
|
||||
|
||||
---
|
||||
|
||||
## §0 — Substrate readiness gate
|
||||
|
||||
Before kickoff, confirm with grep evidence:
|
||||
|
||||
- [ ] hive-mind retrieval pipeline operational (must support multi-doc ingest + chunked retrieval)
|
||||
- [ ] GEPA agent harness operational at HEAD (verify on commit `<HEAD_SHA>`)
|
||||
- [ ] LiteLLM gateway reachable for both candidate models (Claude Opus 4.7 + Qwen 3.6 35B-A3B)
|
||||
- [ ] LiteLLM gateway reachable for trio judge (Opus 4.7 + GPT-5.4 + MiniMax M2.7)
|
||||
- [ ] HEAD commit clean working tree (no uncommitted changes that would invalidate reproducibility)
|
||||
- [ ] Pilot folder readable from execution env: `D:\Projects\PM-Waggle-OS\briefs\2026-04-26-agentic-knowledge-work-pilot\`
|
||||
|
||||
If any of the above fails, halt and ping PM with specifics. Do not proceed with workarounds.
|
||||
|
||||
---
|
||||
|
||||
## §1 — Goal & rationale
|
||||
|
||||
This pilot validates the **agentic knowledge work multiplier thesis** with a small directional sample (N=3 tasks × 4 cells = 12 candidate runs) before authorizing a full N=400 multiplier benchmark.
|
||||
|
||||
The Stage 3 v6 N=400 LoCoMo benchmark proved memory substrate quality (oracle 74% > Mem0 66.9%). That is paper claim #1 — architectural pattern.
|
||||
|
||||
This pilot tests paper claim #2 — **does adding hive-mind memory + GEPA self-evolve harness lift candidate model performance on real-world knowledge work** (CEO synthesis, consultant coordination, executive decision support)?
|
||||
|
||||
**If pilot PASSES**, full N=400 multiplier benchmark is authorized for paper claim #2.
|
||||
**If pilot FAILS**, expansion halts; resources redirect to retrieval V2 work before retry.
|
||||
|
||||
---
|
||||
|
||||
## §2 — Hypotheses (pre-registered, not modifiable post-results)
|
||||
|
||||
- **H2 — Opus multiplier**: Cell B (Opus + memory + harness) trio mean > Cell A (Opus solo) trio mean by ≥ 0.30 Likert points, on ≥ 2 of 3 tasks
|
||||
- **H3 — Qwen multiplier**: Cell D (Qwen + memory + harness) trio mean > Cell C (Qwen solo) trio mean by ≥ 0.30 Likert points, on ≥ 2 of 3 tasks
|
||||
- **H4 — Sovereignty bridge**: Cell D trio mean ≥ Cell A trio mean (Qwen + harness reaches Opus solo) on ≥ 2 of 3 tasks
|
||||
|
||||
**PILOT PASS** = H2 + H3 + H4 each show directional sign on ≥ 2 of 3 tasks AND no critical failures (no cell scoring < 2.0 on majority of judges)
|
||||
**PILOT FAIL** = otherwise
|
||||
|
||||
Anti-pattern reminder: thresholds do not shift post-hoc. Sample size is small; trust the directional sign, not absolute magnitudes.
|
||||
|
||||
---
|
||||
|
||||
## §3 — Cell specification
|
||||
|
||||
| Cell | Model | Memory layer | GEPA harness | Operating mode |
|
||||
|---|---|---|---|---|
|
||||
| A | claude-opus-4-7 | OFF | OFF | Single-shot; full materials in context |
|
||||
| B | claude-opus-4-7 | ON (hive-mind retrieval) | ON | Multi-step agent; materials ingested → retrieval → synthesize |
|
||||
| C | qwen3.6-35b-a3b | OFF | OFF | Single-shot; full materials in context |
|
||||
| D | qwen3.6-35b-a3b | ON (hive-mind retrieval) | ON | Multi-step agent; materials ingested → retrieval → synthesize |
|
||||
|
||||
**Important configuration notes:**
|
||||
|
||||
- **Cell A and C (solo)**: All materials concatenated into a single user prompt. Single API call. No agent steps. No memory injection.
|
||||
- **Cell B and D (memory + harness)**: Materials are first ingested into hive-mind as a session corpus. GEPA agent harness then operates with retrieval over this corpus, can re-prompt itself, and produces final response after multi-step process.
|
||||
- **Same final question** is asked across all four cells per task (verbatim from task file).
|
||||
- **Same temperature settings**: candidate models at `temperature=0.3, top_p=0.9`. Judge models at `temperature=0` for determinism.
|
||||
- **Qwen primary route**: `qwen3.6-35b-a3b-via-openrouter` (DashScope direct) per LOCKED 2026-04-21 routing policy.
|
||||
|
||||
---
|
||||
|
||||
## §4 — Tasks
|
||||
|
||||
Three tasks live in this folder:
|
||||
|
||||
| File | Task type | Question to answer |
|
||||
|---|---|---|
|
||||
| `task-1-strategic-synthesis.md` | Multi-document strategic synthesis | "Identify 3 most critical risks for NorthLane Q2-Q4 2026 and propose action plan" |
|
||||
| `task-2-cross-thread-coordination.md` | Cross-thread project coordination | "Prepare me for tomorrow's emergency check-in with Diane Mercer" |
|
||||
| `task-3-decision-support.md` | Decision support under conflict | "Formulate my CEO decision for next 6 months given three conflicting C-level memos" |
|
||||
|
||||
Each task file contains:
|
||||
- Persona + scenario header
|
||||
- Question to answer (verbatim)
|
||||
- All materials (documents/threads/memos)
|
||||
- Quality expectations note (NOT shown to candidate models or judges — for PM reference only)
|
||||
|
||||
**Materials extraction for candidate prompts:**
|
||||
- Strip the `## End of materials` block and everything after it (quality expectations note must NOT leak to candidate)
|
||||
- Concatenate persona + scenario + materials + question into final prompt
|
||||
- For Cells A/C: pass entire concatenation as single user message
|
||||
- For Cells B/D: chunk materials into hive-mind session per natural document boundary, then pass persona + question to agent
|
||||
|
||||
---
|
||||
|
||||
## §5 — Judge ensemble
|
||||
|
||||
Judge ensemble locked: **Opus 4.7 + GPT-5.4 + MiniMax M2.7**
|
||||
- Same trio used in Stage 3 v6 (κ_trio = 0.7878 calibrated 2026-04-24)
|
||||
- Each judge scores each cell response on 6 dimensions, Likert 1-5
|
||||
- Judges are **blind** to cell configuration (do not include "this is Opus solo" in judge prompt)
|
||||
- Judges have access to: persona + scenario + question + materials + response only
|
||||
|
||||
Full rubric and judge prompt template in `judge-rubric.md`. **Do not modify rubric for execution** — copy verbatim into judge calls.
|
||||
|
||||
**Total judge calls**: 12 cells × 3 judges = 36 calls.
|
||||
|
||||
---
|
||||
|
||||
## §6 — Output
|
||||
|
||||
### Per-cell JSONL records
|
||||
|
||||
One record per cell per task, written to:
|
||||
`D:\Projects\waggle-os\benchmarks\results\pilot-2026-04-26\pilot-{task-id}-{cell-id}.jsonl`
|
||||
|
||||
Schema in `judge-rubric.md` §"Output JSONL schema". 12 records total.
|
||||
|
||||
### Aggregate summary
|
||||
|
||||
Single summary file:
|
||||
`D:\Projects\waggle-os\benchmarks\results\pilot-2026-04-26\pilot-summary.json`
|
||||
|
||||
Schema in `judge-rubric.md` §"Aggregate summary file".
|
||||
|
||||
### Run log
|
||||
|
||||
Append-only log of execution events to:
|
||||
`D:\Projects\waggle-os\benchmarks\results\pilot-2026-04-26\pilot-run.log`
|
||||
|
||||
Include: cell start/end timestamps, candidate model latency, judge call latency, cost accumulator, errors, halt events.
|
||||
|
||||
---
|
||||
|
||||
## §7 — Cost & halt rules
|
||||
|
||||
**Hard cap**: $5.00 cumulative spend (candidate + judge)
|
||||
**Halt threshold**: $4.00 cumulative — at this threshold, complete current cell + judges, then halt and emit partial summary
|
||||
|
||||
**Per-call sanity check**: any single API call exceeding $0.50 → halt and ping PM (likely runaway agent loop in Cells B/D)
|
||||
|
||||
**Halt-and-ping triggers** (any of these → halt, do not continue without PM):
|
||||
- Single candidate call >$0.50
|
||||
- Single judge call >$0.20
|
||||
- Cumulative spend >$4.00
|
||||
- Any cell exceeds 90 wall-clock minutes (likely agent loop)
|
||||
- Any judge returns malformed JSON 3+ times in a row (judge service degraded)
|
||||
- Any candidate model returns refusal / safety-block (unexpected; investigate before retry)
|
||||
|
||||
---
|
||||
|
||||
## §8 — Reproducibility
|
||||
|
||||
Record at execution time:
|
||||
- HEAD commit SHA of waggle-os repo
|
||||
- HEAD commit SHA of hive-mind repo (if extracted by then)
|
||||
- Manifest anchor string: `pilot-2026-04-26-v1`
|
||||
- Model versions exact (e.g., `claude-opus-4-7@2026-03-15`)
|
||||
- LiteLLM config snapshot
|
||||
- Random seed: `seed=42` for any stochastic component
|
||||
- Full prompt concatenations (per cell, per task) saved to `prompts-archive/` subdirectory
|
||||
|
||||
This pilot is small enough that exact reproducibility is feasible and required.
|
||||
|
||||
---
|
||||
|
||||
## §9 — Execution sequence
|
||||
|
||||
1. Pre-flight (§0 substrate gate) — confirm green
|
||||
2. Record HEAD SHA + manifest anchor
|
||||
3. For each task (1, 2, 3):
|
||||
- For each cell (A, B, C, D):
|
||||
- Build prompt per §4 extraction rules
|
||||
- Call candidate model, capture response + latency + cost
|
||||
- For each judge (Opus, GPT, MiniMax):
|
||||
- Build judge prompt per `judge-rubric.md` template
|
||||
- Call judge model, capture verdict + rationale + cost
|
||||
- Compute trio mean, strict-pass, critical-fail flags
|
||||
- Write per-cell JSONL record
|
||||
- Update cost accumulator; check halt rules
|
||||
4. Compute aggregate summary per `judge-rubric.md` schema
|
||||
5. Write summary file + final run log entry
|
||||
6. Ping PM with: pilot verdict (PASS/FAIL), cost, wall-clock, link to summary file
|
||||
|
||||
---
|
||||
|
||||
## §10 — Open questions for PM ratification
|
||||
|
||||
Before CC-1 kicks off, PM should confirm:
|
||||
|
||||
1. **Manifest anchor freeze**: Lock `pilot-2026-04-26-v1` as anchor string for this pilot (no v2 mid-execution).
|
||||
2. **Qwen route confirmation**: Is `qwen3.6-35b-a3b-via-openrouter` still the live primary route as of 2026-04-26? (Last LOCKED 2026-04-21.)
|
||||
3. **GEPA harness state**: Is GEPA self-evolve currently passing tests at HEAD, or is there a known bug requiring workaround? (If broken, pilot blocks.)
|
||||
4. **hive-mind ingest path**: Confirm session-scoped corpus ingest is the correct pattern for materials (vs. global memory write). Pilot must not contaminate other test data.
|
||||
5. **Judge cost reality check**: Stage 3 v6 trio averaged ~$0.07 per judge call. 36 calls = ~$2.52. Plus 12 candidate calls (Opus dominates). Total estimated ~$3.50-4.50. Confirms $5 cap is realistic but tight; halt at $4 is correct buffer.
|
||||
|
||||
---
|
||||
|
||||
## §11 — Post-execution PM actions
|
||||
|
||||
After CC-1 emits pilot summary:
|
||||
|
||||
1. PM reads summary file, validates all 12 cells executed, no critical failures
|
||||
2. PM drafts go/no-go memo for full N=400 multiplier benchmark:
|
||||
- If PASS → authorize full benchmark with cost cap, model roster, scope
|
||||
- If FAIL → halt expansion, draft retrieval V2 priority brief
|
||||
3. Marko ratifies decision
|
||||
4. Memory updated with pilot result + decision
|
||||
|
||||
---
|
||||
|
||||
## §12 — Notes
|
||||
|
||||
- This is a **direction validator**, not a paper claim. Sample size is too small for publication-grade evidence.
|
||||
- Full N=400 multiplier benchmark (post-pilot, if PASS) will be the publication-grade evidence. That benchmark will use the same task design pattern but with N=400 task instances and broader model coverage (Opus + Qwen + GPT-5.4).
|
||||
- Pilot results are internal-only. No external comms triggered by pilot pass/fail.
|
||||
- Pilot folder lives in PM-Waggle-OS, results live in waggle-os/benchmarks/results — standard separation of brief vs. execution artifacts.
|
||||
@@ -0,0 +1,221 @@
|
||||
# Judge Rubric — Trio Ensemble × 6 Dimensions × Likert 1-5
|
||||
|
||||
**Purpose:** Calibrated quality assessment of agent responses to knowledge work tasks. Single-axis Yes/No judging (LoCoMo style) is unsuitable for synthesis tasks where "correctness" is multi-dimensional and the question itself is open-ended.
|
||||
|
||||
**Judge ensemble (locked):**
|
||||
- Claude Opus 4.7 (`claude-opus-4-7`)
|
||||
- GPT-5.4 (`gpt-5.4`)
|
||||
- MiniMax M2.7 (`minimax-m2.7`)
|
||||
|
||||
**Reuses Stage 3 v6 trio infrastructure** — `κ_trio = 0.7878` (substantial agreement) calibrated 2026-04-24. No new judge calibration needed for this pilot. If pilot escalates to full N=400, recalibrate on synthesis-task subset (deferred to expansion brief).
|
||||
|
||||
---
|
||||
|
||||
## Six dimensions
|
||||
|
||||
Each judge scores each cell response on six dimensions, Likert 1-5. **Mean across dimensions = overall score.** Halt threshold: any cell scoring < 2.0 on majority of judges = critical failure flag.
|
||||
|
||||
### D1 — Completeness
|
||||
|
||||
*Did the response engage with all material provided, or did it ignore key inputs?*
|
||||
|
||||
- **5 — Comprehensive**: Engages with every document/thread/memo. Cites or references most. No material is treated as irrelevant without justification.
|
||||
- **4 — Strong**: Engages with most materials. May skip minor items but justifies omissions.
|
||||
- **3 — Adequate**: Engages with majority of materials. Some material visibly missed but core covered.
|
||||
- **2 — Partial**: Significant material omitted without justification. Response treats subset as if it were the whole.
|
||||
- **1 — Inadequate**: Response engages with minority of materials. Most input is ignored.
|
||||
|
||||
### D2 — Accuracy
|
||||
|
||||
*Are the facts cited from the materials accurate, or are there hallucinations / misreadings?*
|
||||
|
||||
- **5 — Faithful**: All cited facts traceable to materials. No hallucinations. Numbers correct. Names correct.
|
||||
- **4 — Mostly faithful**: 1-2 minor inaccuracies (wrong number, slight name variant) but no material distortion.
|
||||
- **3 — Mixed**: Some inaccuracies. Core narrative still defensible from materials.
|
||||
- **2 — Weak**: Multiple factual errors. Some claims not in materials. Reader would be misled on specific points.
|
||||
- **1 — Unreliable**: Significant fabrication or misreading. Reader cannot trust the response.
|
||||
|
||||
### D3 — Synthesis quality
|
||||
|
||||
*Does the response connect inputs across documents/threads/memos, or treat each in isolation?*
|
||||
|
||||
- **5 — Deeply synthesized**: Identifies non-obvious connections (e.g., "X in Doc 2 explains Y in Doc 5"). Surfaces interaction effects. Goes beyond the surface of any single input.
|
||||
- **4 — Strong synthesis**: Connects most inputs. Cross-references where appropriate. May miss 1-2 deeper patterns.
|
||||
- **3 — Adequate synthesis**: Some connections drawn. Mostly summarizes input-by-input with limited weaving.
|
||||
- **2 — Weak synthesis**: Treats inputs in isolation. List-like structure mirroring input order.
|
||||
- **1 — No synthesis**: Disconnected responses to individual inputs. No integration.
|
||||
|
||||
### D4 — Judgment quality
|
||||
|
||||
*Are the recommendations defensible? Are tradeoffs acknowledged? Is reasoning shown?*
|
||||
|
||||
- **5 — Senior-grade**: Recommendations are specific and actionable. Tradeoffs explicitly addressed. Counter-arguments anticipated. Reasoning visible at each step.
|
||||
- **4 — Strong**: Recommendations are clear and reasoned. Most tradeoffs surfaced. Some implicit reasoning.
|
||||
- **3 — Adequate**: Recommendations made but reasoning thin. Tradeoffs touched lightly.
|
||||
- **2 — Weak**: Recommendations feel arbitrary. Tradeoffs ignored or minimized. Reasoning shallow.
|
||||
- **1 — No judgment**: Recommendations missing, generic, or contradicted by their own analysis.
|
||||
|
||||
### D5 — Recommendation actionability
|
||||
|
||||
*Could the persona (CFO / Partner / CEO) act on this tomorrow morning, or is it advice-shaped fog?*
|
||||
|
||||
- **5 — Immediately actionable**: Specific actions, owners (where applicable), sequencing, success metrics. The persona could open a doc tomorrow and start executing.
|
||||
- **4 — Mostly actionable**: Most actions are specific. Some require additional definition but the path is clear.
|
||||
- **3 — Directionally actionable**: Direction is clear; specific next steps require persona to fill in.
|
||||
- **2 — Vague**: General advice. Persona has to do meaningful translation work to derive actions.
|
||||
- **1 — Not actionable**: Abstract reasoning without practical pathway. No persona could act on this.
|
||||
|
||||
### D6 — Structure / Communication
|
||||
|
||||
*Is the response organized for the reader's mental model? Is it the right length? Is it readable under time pressure?*
|
||||
|
||||
- **5 — Excellent**: Clear executive structure (e.g., headline → reasoning → asks). Appropriate length. Reader can scan in 60 seconds and read in detail in 5 minutes. Headers, emphasis, sequence used judiciously.
|
||||
- **4 — Strong**: Well-organized. Reasonable length. Reader navigates easily.
|
||||
- **3 — Adequate**: Comprehensible. Length OK. Some friction in scanning.
|
||||
- **2 — Weak**: Disorganized. Too long or too brief. Reader has to work to extract main points.
|
||||
- **1 — Poor**: Chaotic structure. Significantly mis-sized. Reader gets lost or gives up.
|
||||
|
||||
---
|
||||
|
||||
## Overall scoring
|
||||
|
||||
**Per judge per cell:** mean of D1-D6 = overall score (Likert 1-5)
|
||||
|
||||
**Per cell aggregated:**
|
||||
- Trio mean: (Opus mean + GPT mean + MiniMax mean) / 3
|
||||
- Trio strict-PASS: at least 2 of 3 judges score ≥ 3.5
|
||||
- Trio FAIL: at least 2 of 3 judges score < 3.0
|
||||
|
||||
**Hypothesis verification (per task):**
|
||||
- **H2 — Opus multiplier**: Cell B (Opus + memory + harness) trio mean > Cell A (Opus solo) trio mean by ≥ 0.30 Likert points
|
||||
- **H3 — Qwen multiplier**: Cell D (Qwen + memory + harness) trio mean > Cell C (Qwen solo) trio mean by ≥ 0.30 Likert points
|
||||
- **H4 — Sovereignty bridge**: Cell D trio mean ≥ Cell A trio mean (Qwen + harness reaches frontier-without-harness)
|
||||
|
||||
**Pilot binary verdict:**
|
||||
- **PILOT PASS** = directional sign correct on H2/H3/H4 in ≥ 2 of 3 tasks (6/9 cells minimum), and no critical failure (no cell scoring < 2.0 on majority of judges)
|
||||
- **PILOT FAIL** = otherwise
|
||||
|
||||
PASS authorizes full N=400 multiplier benchmark. FAIL halts expansion.
|
||||
|
||||
---
|
||||
|
||||
## Judge prompt template (per cell response)
|
||||
|
||||
```
|
||||
You are evaluating an AI agent's response to a complex knowledge work task. The persona, scenario, materials, and question are provided. The response was generated under one of four configurations (revealed only after scoring): {model_only | model + memory + agent harness} × {Opus 4.7 | Qwen 3.6 35B-A3B}.
|
||||
|
||||
You do NOT know which configuration produced this response. Score blind.
|
||||
|
||||
Read the persona/scenario/question (provided), skim the materials (provided), then read the response carefully (provided).
|
||||
|
||||
Score the response on six dimensions, Likert 1-5:
|
||||
|
||||
1. COMPLETENESS — engagement with all material
|
||||
2. ACCURACY — faithfulness to source materials, no hallucinations
|
||||
3. SYNTHESIS — connections across inputs, not isolated treatment
|
||||
4. JUDGMENT — defensible recommendations, tradeoffs acknowledged
|
||||
5. ACTIONABILITY — would the persona act on this tomorrow
|
||||
6. STRUCTURE — organization and readability
|
||||
|
||||
Output JSON only:
|
||||
{
|
||||
"completeness": <1-5>,
|
||||
"accuracy": <1-5>,
|
||||
"synthesis": <1-5>,
|
||||
"judgment": <1-5>,
|
||||
"actionability": <1-5>,
|
||||
"structure": <1-5>,
|
||||
"rationale": "<1-2 sentences explaining the lowest scoring dimension>",
|
||||
"overall_verdict": "<one of: PASS_STRONG | PASS_ADEQUATE | FAIL_WEAK | FAIL_CRITICAL>"
|
||||
}
|
||||
|
||||
PASS_STRONG: mean ≥ 4.0
|
||||
PASS_ADEQUATE: mean 3.5-3.99
|
||||
FAIL_WEAK: mean 2.5-3.49
|
||||
FAIL_CRITICAL: mean < 2.5
|
||||
|
||||
[PERSONA + SCENARIO + QUESTION]
|
||||
[MATERIALS]
|
||||
[RESPONSE TO EVALUATE]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Output JSONL schema (per cell, per task)
|
||||
|
||||
Each cell × task produces one record:
|
||||
|
||||
```json
|
||||
{
|
||||
"task_id": "task-1" | "task-2" | "task-3",
|
||||
"cell_id": "A" | "B" | "C" | "D",
|
||||
"model": "claude-opus-4-7" | "qwen3.6-35b-a3b",
|
||||
"configuration": "solo" | "memory-harness",
|
||||
"candidate_response": "<full response text>",
|
||||
"candidate_latency_ms": <int>,
|
||||
"candidate_tokens_in": <int>,
|
||||
"candidate_tokens_out": <int>,
|
||||
"candidate_cost_usd": <float>,
|
||||
"judge_opus": {
|
||||
"completeness": <int>,
|
||||
"accuracy": <int>,
|
||||
"synthesis": <int>,
|
||||
"judgment": <int>,
|
||||
"actionability": <int>,
|
||||
"structure": <int>,
|
||||
"rationale": "<string>",
|
||||
"overall_verdict": "<string>",
|
||||
"mean": <float>
|
||||
},
|
||||
"judge_gpt": { ... same shape ... },
|
||||
"judge_minimax": { ... same shape ... },
|
||||
"trio_mean": <float>,
|
||||
"trio_strict_pass": <bool>,
|
||||
"trio_critical_fail": <bool>,
|
||||
"manifest_anchor": "pilot-2026-04-26-v1",
|
||||
"head_sha": "<git commit SHA at execution>"
|
||||
}
|
||||
```
|
||||
|
||||
12 records total (3 tasks × 4 cells).
|
||||
|
||||
---
|
||||
|
||||
## Aggregate summary file
|
||||
|
||||
After execution, produce `pilot-summary.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"pilot_id": "agentic-knowledge-work-pilot-2026-04-26",
|
||||
"execution_window_utc": "<ISO start> to <ISO end>",
|
||||
"total_cost_usd": <float>,
|
||||
"total_judge_calls": 36,
|
||||
"total_candidate_calls": 12,
|
||||
"results_per_task": {
|
||||
"task-1": {
|
||||
"cell_A_trio_mean": <float>,
|
||||
"cell_B_trio_mean": <float>,
|
||||
"cell_C_trio_mean": <float>,
|
||||
"cell_D_trio_mean": <float>,
|
||||
"h2_delta_opus": <B - A>,
|
||||
"h3_delta_qwen": <D - C>,
|
||||
"h4_delta_sovereignty": <D - A>,
|
||||
"h2_directional_pass": <bool>,
|
||||
"h3_directional_pass": <bool>,
|
||||
"h4_directional_pass": <bool>
|
||||
},
|
||||
"task-2": { ... },
|
||||
"task-3": { ... }
|
||||
},
|
||||
"aggregate": {
|
||||
"h2_pass_count": <int 0-3>,
|
||||
"h3_pass_count": <int 0-3>,
|
||||
"h4_pass_count": <int 0-3>,
|
||||
"critical_failures": <int>,
|
||||
"pilot_verdict": "PASS" | "FAIL"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
PM and Marko adjudicate from this summary file.
|
||||
@@ -0,0 +1,186 @@
|
||||
# Task 1 — Multi-Document Strategic Synthesis
|
||||
|
||||
**Persona:** You are the CFO of NorthLane, a Series B B2B SaaS company providing supply-chain visibility software to mid-market manufacturers. The company has $14.2M ARR, 84 full-time employees, 18 months of runway. Today is April 26, 2026.
|
||||
|
||||
**Scenario:** Q1 2026 just closed. Your CEO has asked you to prepare a 1-page memo for next week's board meeting identifying the **3 most critical risks** for Q2-Q4 2026, with a recommended action plan for each.
|
||||
|
||||
**Question to answer:**
|
||||
> "Based on all materials provided, identify the 3 most critical risks for NorthLane in Q2-Q4 2026 and propose a specific, prioritized action plan for each. Justify why these 3 (and not others) are the most critical, and address how they interact."
|
||||
|
||||
**Materials provided:** 7 documents (below). Read all before answering.
|
||||
|
||||
---
|
||||
|
||||
## DOC 1 — Q1 2026 P&L Summary (Internal)
|
||||
|
||||
**Period:** Q1 2026 (Jan-Mar)
|
||||
|
||||
| Line item | Q1 2026 | Q1 2025 | YoY % | vs Plan |
|
||||
|---|---|---|---|---|
|
||||
| Total revenue | $3.45M | $2.95M | +17% | -8% |
|
||||
| New ARR booked | $0.62M | $0.78M | -21% | -34% |
|
||||
| Gross margin | 71% | 74% | -3pp | -2pp |
|
||||
| S&M spend | $1.85M | $1.40M | +32% | +4% |
|
||||
| R&D spend | $1.10M | $0.85M | +29% | +2% |
|
||||
| G&A spend | $0.55M | $0.45M | +22% | +1% |
|
||||
| Operating loss | $(0.95M) | $(0.45M) | -111% | -45% |
|
||||
| Cash burn | $1.05M | $0.55M | -91% | -38% |
|
||||
| Cash on hand | $18.9M | — | — | — |
|
||||
| Implied runway | 18 months | 26 months | — | -8 months |
|
||||
|
||||
**CFO note:** Q1 saw revenue growth slow vs. plan, while spend continued tracking aggressive. Operating loss doubled YoY. Net new ARR materially below plan — first time in 6 quarters we missed quota by >25%. If current trajectory holds, runway compresses below 12 months by Q4 without intervention.
|
||||
|
||||
---
|
||||
|
||||
## DOC 2 — Sales Pipeline Review (VP Sales, April 8, 2026)
|
||||
|
||||
**Headline:** Q1 closed-won $0.62M new ARR vs. plan $0.95M. 65% attainment, lowest since Q3 2024.
|
||||
|
||||
**Pipeline composition:**
|
||||
- Total pipeline entering Q2: $4.8M (vs. $5.6M same time last year, -14%)
|
||||
- Win rate Q1: 22% (vs. 28% Q1 2025, -6pp)
|
||||
- Average deal size: $48K ACV (vs. $52K Q1 2025, -8%)
|
||||
- Sales cycle median: 94 days (vs. 71 days Q1 2025, +23 days)
|
||||
|
||||
**Top loss reasons (Q1 closed-lost analysis, n=23):**
|
||||
1. "Competitor X chosen" — 9 deals (39%) — 7 of 9 lost to ChainSight Inc.
|
||||
2. "Budget pulled / pause" — 6 deals (26%)
|
||||
3. "Pricing too high" — 4 deals (17%)
|
||||
4. "Procurement / IT review timeline" — 3 deals (13%)
|
||||
5. "Decision postponed indefinitely" — 1 deal (4%)
|
||||
|
||||
**VP Sales commentary:** ChainSight's January positioning shift toward "AI-native supply chain" is hurting our top of funnel. Our reps report 4 of 7 losses to them cited "their AI roadmap is more credible." Three of our top 5 reps are at risk of attrition — two have had recruiter conversations. We need 2 net new reps to hit Q3 plan, but headcount freeze pending board review.
|
||||
|
||||
---
|
||||
|
||||
## DOC 3 — Customer Health & Churn Analysis (CS Director, April 12, 2026)
|
||||
|
||||
**Q1 churn metrics:**
|
||||
- Logo churn: 4 customers (3.4% of base) — highest single-quarter logo churn since founding
|
||||
- Gross revenue churn: $0.34M ARR
|
||||
- Net revenue retention: 102% (vs. 118% Q1 2025) — first time below 110% in 8 quarters
|
||||
- NPS (Q1 survey, n=68 respondents): 31 (vs. 47 Q1 2025, -16 points)
|
||||
|
||||
**Churn reasons (4 logos lost):**
|
||||
1. **AcmeMfg ($110K ARR)** — switched to ChainSight, cited "missing predictive analytics features"
|
||||
2. **ParaglyphCorp ($85K ARR)** — acquired by larger conglomerate, consolidated to incumbent vendor
|
||||
3. **ToolsmithIndustrial ($75K ARR)** — cited "implementation never reached promised value, ROI unclear"
|
||||
4. **VeritasParts ($70K ARR)** — budget cuts, "nice-to-have" software cut first
|
||||
|
||||
**At-risk accounts ($1.4M ARR combined, expansion plays paused):**
|
||||
- 3 accounts have flagged "considering alternatives" in QBR within Q1
|
||||
- 6 accounts have reduced usage by >30% from Q4 baseline
|
||||
- 11 accounts haven't logged in for >21 days (out of 117 active)
|
||||
|
||||
**CS Director commentary:** Implementation quality complaints have risen 3x QoQ. Engineering bandwidth for customer-specific integrations was cut last sprint to fund the new AI roadmap initiative. CS team has flagged this risk in 3 weekly leadership meetings without resolution.
|
||||
|
||||
---
|
||||
|
||||
## DOC 4 — Engineering Velocity Report (VP Engineering, April 15, 2026)
|
||||
|
||||
**Q1 shipping metrics:**
|
||||
- Story points completed: 412 (vs. 487 Q1 2025, -15%)
|
||||
- Bugs filed (P0/P1): 38 (vs. 22 Q1 2025, +73%)
|
||||
- Bugs resolved (P0/P1): 29 (open backlog growing)
|
||||
- Customer-reported bugs as % of total: 41% (vs. 28% Q1 2025)
|
||||
- On-call pages: 67 (vs. 31 Q1 2025, +116%)
|
||||
- Mean time to recovery: 3.4 hours (vs. 1.8 hours Q1 2025)
|
||||
|
||||
**Headcount:**
|
||||
- Engineers Q1 start: 28
|
||||
- Engineers Q1 end: 26 (2 voluntary departures, both senior)
|
||||
- Open reqs: 4 (1 backfill, 3 net-new for AI roadmap)
|
||||
- Open req median time-to-fill: 87 days
|
||||
|
||||
**Tech debt indicators:**
|
||||
- % of commits to legacy modules (vs. new): 58% (vs. 41% Q1 2025)
|
||||
- Test coverage trending: declining 1.2pp/month for 4 months
|
||||
- Incident postmortem action items completed: 31% (vs. 78% Q1 2025)
|
||||
|
||||
**VP Engineering commentary:** We took on 3 major initiatives in parallel this quarter — AI roadmap MVP, mobile rewrite, and enterprise SSO — without proportional headcount. Quality is suffering. Two of our four senior engineers have privately asked about external opportunities. If we don't course-correct on scope or hire, we'll see further attrition by mid-Q2.
|
||||
|
||||
---
|
||||
|
||||
## DOC 5 — Marketing Efficiency Dashboard (CMO, April 10, 2026)
|
||||
|
||||
**Q1 funnel metrics:**
|
||||
- Marketing-sourced pipeline: $1.8M (vs. $2.4M Q1 2025, -25%)
|
||||
- MQL → SQL conversion: 18% (vs. 24% Q1 2025, -6pp)
|
||||
- SQL → Won conversion: 22% (vs. 28% Q1 2025, -6pp)
|
||||
- CAC (blended): $24,500 (vs. $19,800 Q1 2025, +24%)
|
||||
- LTV (current cohort): $148K (vs. $172K Q1 2025, -14%)
|
||||
- LTV:CAC ratio: 6.0x (vs. 8.7x Q1 2025) — still healthy but eroding
|
||||
- Payback period: 14 months (vs. 11 months Q1 2025)
|
||||
|
||||
**Channel performance:**
|
||||
- Paid search: $0.42M spend, $1.1M sourced pipeline (2.6x return — degrading)
|
||||
- Content/SEO: $0.18M spend, $0.5M sourced pipeline (2.8x return — flat)
|
||||
- Outbound SDR: $0.65M cost (3 SDRs), $0.4M sourced (0.6x return — concerning)
|
||||
- Events/sponsorships: $0.35M, $0.3M sourced (0.9x return — questioning ROI)
|
||||
- Partner referrals: $0.10M cost, $0.5M sourced (5.0x return — best performer)
|
||||
|
||||
**CMO commentary:** ChainSight has tripled their digital ad spend QoQ — we're being outbid on key terms by 40-60%. Our content engine is outpaced; their AI-positioned content is winning rankings. SDR team is underperforming due to cold outbound resistance. Recommend doubling partner program investment, but team is currently 1 person.
|
||||
|
||||
---
|
||||
|
||||
## DOC 6 — Board Feedback Notes (post-March 28, 2026 board meeting)
|
||||
|
||||
**Attendees:** 2 VC partners (Sequoia, Bessemer), 2 independent directors, founder/CEO, CFO
|
||||
|
||||
**Key themes from board discussion (CFO summary):**
|
||||
|
||||
1. **Burn rate concern (Sequoia partner, primary):** "Operating loss doubling YoY with revenue slowing is the single biggest red flag. We need to see a 30%+ reduction in burn by end of Q3 or this becomes a path-to-default conversation. Profitability discipline is non-negotiable."
|
||||
|
||||
2. **Competitive positioning (Bessemer partner):** "ChainSight raised $80M Series C in February. Their war chest will fund 2-3 years of aggressive go-to-market. Either we differentiate hard within 6 months or we accept a smaller niche position. The middle path is dangerous."
|
||||
|
||||
3. **Talent retention (Independent director, ex-CEO):** "Engineering attrition risk is the most underdiscussed issue. Losing 2 senior engineers in Q1 alone would have been a board-level crisis at my last company. What's the retention plan?"
|
||||
|
||||
4. **AI roadmap (CEO interjection):** "We have a major AI feature in development — predictive analytics + agent orchestration. We believe this re-positions us competitively. Want to ship by Q3."
|
||||
|
||||
5. **Capital strategy (Sequoia partner):** "If you can't show clear progress on burn AND competitive positioning by Q3, the next financing conversation will be very hard. We're not interested in bridge rounds at flat valuations. The clock starts now."
|
||||
|
||||
**Board next steps:**
|
||||
- Q2 monthly burn updates required
|
||||
- Q2 retention plan + competitive moat memo due by May 15
|
||||
- Q3 financial review will be go/no-go on AI roadmap continued investment
|
||||
|
||||
---
|
||||
|
||||
## DOC 7 — Competitor Intelligence Brief (Strategy Lead, April 5, 2026)
|
||||
|
||||
**Subject:** ChainSight Inc. — competitive update (post-Series C)
|
||||
|
||||
**Funding & financial:**
|
||||
- Closed $80M Series C in February 2026 (Andreessen Horowitz lead)
|
||||
- Total raised to date: $135M (vs. NorthLane's $42M)
|
||||
- Reported Q4 ARR (per leaked deck shared via channel partner): $26M (~80% larger than NorthLane)
|
||||
- Reported burn rate: ~$3.5M/month (will accelerate post-funding)
|
||||
|
||||
**Product positioning shifts (Jan-March 2026):**
|
||||
- January: Public re-positioning to "AI-Native Supply Chain Operations" (vs. previous "Real-Time Supply Chain Visibility")
|
||||
- February: Launched ChainSight Copilot — agent-based query interface, real-time recommendations
|
||||
- March: Announced strategic partnership with SAP to embed ChainSight Copilot into SAP Ariba
|
||||
|
||||
**Sales motion shifts:**
|
||||
- Pricing: Aggressively undercutting on 3-year deals (~30% below their published price for "innovation partners")
|
||||
- Headcount: Hired 12 enterprise reps in Q1 (vs. NorthLane's 0 net adds), opened London office
|
||||
- Content: Publishing 3-4 thought leadership pieces per week, dominating "AI supply chain" SEO
|
||||
|
||||
**Win analysis (per channel partner intelligence):**
|
||||
- 7 of 9 customer losses (NorthLane → ChainSight) cited "AI roadmap" as decisive
|
||||
- Average deal won by ChainSight is 18% larger ACV than typical NorthLane deal
|
||||
- ChainSight's expansion motion within accounts is reportedly more aggressive (NPS-driven account scoring)
|
||||
|
||||
**Strategic Lead commentary:** ChainSight is executing a classic "raise-and-blitz" playbook. Their ARR growth, hiring, marketing, and partnerships are all coordinated. We have a 12-18 month window before they have meaningful market share moat. After that, displacement gets exponentially harder.
|
||||
|
||||
---
|
||||
|
||||
## End of materials
|
||||
|
||||
**Reminder of question:**
|
||||
> "Based on all materials provided, identify the 3 most critical risks for NorthLane in Q2-Q4 2026 and propose a specific, prioritized action plan for each. Justify why these 3 (and not others) are the most critical, and address how they interact."
|
||||
|
||||
**Note on quality expectations:**
|
||||
- A strong answer connects multiple documents (e.g., Doc 2 sales loss to Doc 7 competitor positioning to Doc 4 engineering velocity).
|
||||
- A weak answer treats each document in isolation or surfaces only the obvious top-line numbers without synthesis.
|
||||
- An excellent answer notes the **interaction** between risks (e.g., burn-vs-investment tension creates engineering retention risk which compounds competitive vulnerability).
|
||||
@@ -0,0 +1,277 @@
|
||||
# Task 2 — Cross-Thread Project Coordination
|
||||
|
||||
**Persona:** You are a Partner at Meridian Advisory, a boutique strategy consulting firm (28 consultants, $14M revenue). You have been the lead Partner on a 6-month engagement with **Helix Retail Group** (Fortune 500, $4.2B revenue, 480 stores across North America) since January 2026. The engagement is around digital transformation strategy, with implementation oversight scope. Today is April 26, 2026.
|
||||
|
||||
**Scenario:** You have been pulled away on a different engagement for the past 3 weeks. You have an emergency check-in scheduled with Helix's CFO **Diane Mercer** tomorrow morning (April 27 at 9:00 AM). Diane requested the meeting via email yesterday with the subject "Urgent — engagement scope discussion."
|
||||
|
||||
You need to walk into that meeting with crystal clarity on: (a) where the engagement actually stands, (b) what Diane is most likely worried about, (c) what specific positions and asks you should bring. You have 4 threads of context spanning the engagement.
|
||||
|
||||
**Question to answer:**
|
||||
> "Given all four threads, prepare me for tomorrow's emergency check-in with Diane Mercer. Specifically: (1) What is the current status of the Helix engagement — completed work, in-flight work, blocked work? (2) What is Diane most likely concerned about, and why now? (3) What are the 3 most likely outcomes she is pushing toward, and which should I support, push back on, or negotiate? (4) What specific commitments, asks, or positions should I walk in with?"
|
||||
|
||||
**Materials provided:** 4 threads (chronological).
|
||||
|
||||
---
|
||||
|
||||
## THREAD 1 — Initial Scoping Call Notes (January 14, 2026)
|
||||
|
||||
**Meeting:** Helix Retail Group — digital transformation engagement scoping
|
||||
**Attendees (Helix):** Diane Mercer (CFO), Roberto Salazar (CIO), Priya Iyer (VP Operations), Mark Chen (Head of Digital)
|
||||
**Attendees (Meridian):** Marko Marković (Lead Partner), Elena Voss (Engagement Manager), James Park (Senior Consultant)
|
||||
**Duration:** 90 min
|
||||
|
||||
**Stated business problem:**
|
||||
- Helix's e-commerce revenue grew 38% YoY in 2024 but only 9% in 2025
|
||||
- Cart abandonment up 12% over 18 months; mobile conversion 40% below industry benchmark
|
||||
- 7 separate digital initiatives in flight across 4 departments — no unified roadmap
|
||||
- Roberto (CIO) acknowledged "we're spending $34M/year on digital and can't articulate the strategy"
|
||||
|
||||
**Diane's stated priorities (in order):**
|
||||
1. **Cost rationalization** — "I need to see ROI on digital spend or we cut it in half by Q3"
|
||||
2. **Single integrated roadmap** — "I'm tired of every VP showing me their own roadmap with no overlap analysis"
|
||||
3. **External validation** — "Board has questioned whether we should outsource e-com to a partner instead"
|
||||
|
||||
**Roberto's stated priorities (different order):**
|
||||
1. Modernization of legacy POS-to-warehouse integration
|
||||
2. Mobile commerce performance improvement
|
||||
3. Customer data platform consolidation (currently 4 systems)
|
||||
|
||||
**Priya's concerns:**
|
||||
- Operations team is exhausted from 14-month POS modernization that "isn't even half done"
|
||||
- Concerns about implementing more change before stabilizing what's in flight
|
||||
|
||||
**Initial scope agreed (verbal, to be confirmed in SOW):**
|
||||
- 12-week engagement, 3 phases: discovery (4w), strategy (4w), roadmap & implementation oversight (4w)
|
||||
- Deliverables: digital portfolio audit, ROI assessment of 7 in-flight initiatives, integrated 18-month roadmap, governance recommendation
|
||||
- Estimated fee: $480K fixed-fee + expenses, billed monthly
|
||||
- Implementation oversight to extend post-engagement at Helix's option
|
||||
|
||||
**Open questions flagged for Week 1:**
|
||||
- Whether implementation oversight is in-scope or follow-on engagement
|
||||
- Access to existing vendor contracts (Diane indicated some are "messy")
|
||||
- Diane mentioned a recent McKinsey diagnostic — wants Meridian to NOT replicate that work
|
||||
|
||||
**Key quote from Diane:** "I want a partner who tells me what to kill, not what to add. If you come back with a recommendation to do all 7 things plus 4 new things, we're done."
|
||||
|
||||
---
|
||||
|
||||
## THREAD 2 — Mid-Engagement Workshop Notes (February 26, 2026)
|
||||
|
||||
**Meeting:** Helix Digital Strategy Workshop — Phase 2 kickoff
|
||||
**Attendees (Helix):** Diane (CFO), Roberto (CIO), Priya (VPO), Mark (Head of Digital), 4 VPs from operations & marketing
|
||||
**Attendees (Meridian):** Marko, Elena, James, plus 2 analysts
|
||||
**Duration:** Full day (8 hours)
|
||||
|
||||
**Phase 1 findings presented (discovery, 4 weeks completed):**
|
||||
|
||||
*Initiative ROI assessment (7 in-flight initiatives):*
|
||||
1. **POS modernization** — $14M sunk, 14 months in, ~40% complete. Original ROI case (4-year payback) now likely 7+ years. **Recommendation: complete current sprint, then assess kill vs. continue.**
|
||||
2. **Mobile app rewrite** — $4.2M committed, 8 months in. Performance improvement real (38% mobile conv. lift in pilot). **Recommendation: accelerate, deploy nationally Q2.**
|
||||
3. **Customer data platform consolidation** — $3.8M planned, not started. 4 vendor proposals received. **Recommendation: pause, re-scope after roadmap.**
|
||||
4. **AI-powered personalization (engine)** — $2.5M started Q4 2025. Vendor underperforming. **Recommendation: replace vendor or kill.**
|
||||
5. **In-store digital signage** — $1.2M, deployed in 80 stores. ROI unmeasurable due to no baseline. **Recommendation: instrument or wind down.**
|
||||
6. **Marketing automation upgrade** — $0.9M, in pilot. Working as expected. **Recommendation: continue.**
|
||||
7. **Voice-of-customer analytics** — $0.6M, year-old. Insights produced but not actioned. **Recommendation: integrate into ops cadence or kill.**
|
||||
|
||||
*Strategic findings:*
|
||||
- Real driver of slowing e-com growth = **mobile experience gap**, not lack of new initiatives
|
||||
- $34M/year digital spend has 22% effectiveness vs. industry benchmark of 38-44%
|
||||
- Most pressing technical debt = legacy POS → cloud architecture transition (independent of POS modernization initiative)
|
||||
|
||||
*Recommendations crystallizing:*
|
||||
- **Kill 2 initiatives** (#3 CDP, #4 AI personalization vendor)
|
||||
- **Pause and re-scope 2** (#1 POS modernization, #5 signage)
|
||||
- **Accelerate 2** (#2 mobile, #6 marketing automation)
|
||||
- **Continue 1** (#7 voice-of-customer with action mandate)
|
||||
- **New priority:** legacy POS → cloud architecture as foundational
|
||||
|
||||
**Stakeholder reactions:**
|
||||
|
||||
- **Diane (CFO):** "This is what I needed. Two questions — kill recommendations are firm? And what's the savings number?"
|
||||
- Marko response: "Kill recommendations are firm pending vendor contract review. Direct savings ~$6.3M annualized; reallocation potential another $4-7M."
|
||||
|
||||
- **Roberto (CIO):** *Visible concern.* "POS modernization team will not take a pause well. That's 22 engineers and a vendor." Pushed back on POS pause framing.
|
||||
|
||||
- **Priya (VPO):** *Strongly supportive.* "I've been saying we need to focus for 18 months. Glad someone is finally listening."
|
||||
|
||||
- **Mark (Head of Digital):** *Defensive on AI personalization.* "That vendor is 6 months from delivering, we can't kill them now." Marko noted to revisit privately.
|
||||
|
||||
- **VP of Marketing:** Concerned about mobile acceleration creating dependency on Marketing's roadmap.
|
||||
|
||||
**Open items at workshop close:**
|
||||
- Roberto requested 1:1 follow-up to discuss POS pause framing — scheduled for March 5
|
||||
- Mark requested second look at AI personalization vendor — Marko committed to vendor scorecard by March 12
|
||||
- Diane asked for cost savings memo with vendor contract liabilities mapped — committed by March 15
|
||||
- Diane mentioned: "I may need to brief the board earlier than expected. June board meeting may move to May."
|
||||
|
||||
**Marko's private note (post-meeting):** Roberto is the political risk on this engagement. CDP and AI personalization are his pet projects. If we kill or pause both, we lose his cooperation on implementation. Need to find face-saving framing — possibly position as "phase 2 reconsideration" rather than "kill."
|
||||
|
||||
---
|
||||
|
||||
## THREAD 3 — CFO Email Thread (March 18-25, 2026)
|
||||
|
||||
### From: Diane Mercer
|
||||
### To: Marko Marković
|
||||
### Date: March 18, 2026, 10:42 AM
|
||||
### Subject: Cost savings memo + scope question
|
||||
|
||||
Marko,
|
||||
|
||||
Got the cost savings memo Friday. Solid work — the $6.3M direct savings number checks out against our internal lens, and the $4-7M reallocation framing is well argued.
|
||||
|
||||
Two issues I want to raise before we go further:
|
||||
|
||||
1. **Board timing has shifted.** Our May 8 board meeting is now the moment of truth on digital strategy. I need final recommendations and integrated roadmap with at least 2 weeks for me and Roberto to socialize internally. That means your roadmap + governance deliverable needs to land by April 22, not the original May 6 SOW date.
|
||||
|
||||
2. **Scope question on implementation oversight.** Your contract has a "Helix's option" clause for implementation oversight post-engagement. Our procurement is asking me to either commit or release. I want to commit — but I need to understand the fee structure, scope boundaries, and your team's allocation. Can we have a real conversation this week about a 6-month implementation oversight extension at $180-220K/month?
|
||||
|
||||
I want to be direct about what I'm worried about going into the May 8 board: I need this engagement to clearly demonstrate ROI within 60 days of board endorsement. If implementation drags or vendors push back hard, I need a partner who's there day one of execution, not handing it back to my team and disappearing.
|
||||
|
||||
Can we get on a call Wednesday or Thursday this week?
|
||||
|
||||
Diane
|
||||
|
||||
---
|
||||
|
||||
### From: Marko Marković
|
||||
### To: Diane Mercer
|
||||
### Date: March 18, 2026, 6:15 PM
|
||||
### Subject: Re: Cost savings memo + scope question
|
||||
|
||||
Diane,
|
||||
|
||||
Thank you for the direct framing.
|
||||
|
||||
On (1): Yes, we can compress timing. Roadmap deliverable by April 22 is achievable but tight. We'll need access to vendor termination terms by April 8 or we risk roadmap recommendations that procurement can't execute on. Will Elena reach out to your procurement lead Monday?
|
||||
|
||||
On (2): I want to discuss this thoughtfully. Implementation oversight at the scope you're describing is meaningful — 6 months at $180-220K/month is roughly equivalent to our current engagement. I want to make sure the scope, deliverables, and accountability structure are right before I price it. Let me come back with a proposed structure by end of week.
|
||||
|
||||
Can do Thursday at 2pm ET. Will send invite.
|
||||
|
||||
Marko
|
||||
|
||||
---
|
||||
|
||||
### From: Diane Mercer
|
||||
### To: Marko Marković
|
||||
### Date: March 23, 2026, 8:55 PM
|
||||
### Subject: Heads up — internal politics
|
||||
|
||||
Marko,
|
||||
|
||||
Off the record. Two things you should know going into Thursday:
|
||||
|
||||
1. Roberto has been lobbying for pulling implementation oversight in-house with his team leading. He showed his hand last Friday. CEO is leaning toward Meridian but Roberto's resistance is a factor.
|
||||
|
||||
2. There is internal pressure to consider a "lighter" version of your roadmap — keeping more initiatives alive than your recommendation. Specifically, the AI personalization initiative has a champion at the board level. I've been protecting your recommendation, but it's getting harder.
|
||||
|
||||
I want to set up the May 8 board to land your recommendation as-is. But I need you to be prepared for some watering down attempts in the next 4 weeks. If you anticipate this and propose creative framing, you'll save us both a fight.
|
||||
|
||||
Don't reply to this email — let's discuss Thursday.
|
||||
|
||||
Diane
|
||||
|
||||
---
|
||||
|
||||
### From: Marko Marković
|
||||
### To: Diane Mercer
|
||||
### Date: March 25, 2026, 7:20 AM
|
||||
### Subject: Thursday call confirmation + agenda
|
||||
|
||||
Diane,
|
||||
|
||||
Confirming Thursday 2pm ET.
|
||||
|
||||
Per your March 23 note (acknowledged off-the-record), I'll come prepared on:
|
||||
- Implementation oversight structure proposal — addressing Roberto's preference for in-house with a hybrid framing
|
||||
- Recommendation defense strategy — specifically on the AI personalization initiative, with a "phased decision" framing that preserves optionality without committing further $$$
|
||||
- Board pre-read structure — what we want pre-cooked vs. live discussion
|
||||
|
||||
Will send pre-read 24 hours ahead.
|
||||
|
||||
Marko
|
||||
|
||||
---
|
||||
|
||||
## THREAD 4 — Client Team Slack Messages (April 6-24, 2026)
|
||||
|
||||
**Channel:** #meridian-helix-engagement (private, Helix client team + Meridian project team)
|
||||
|
||||
---
|
||||
|
||||
**[April 6, 9:14 AM] Elena Voss (Meridian EM):**
|
||||
Marko is out for the next 3 weeks on the BluePine engagement. James and I are running point. Diane and the team have been notified.
|
||||
|
||||
**[April 6, 9:18 AM] James Park (Meridian Senior Consultant):**
|
||||
We're on track for April 22 roadmap deliverable. CDP termination notice went out April 3, 30-day vendor cure period started.
|
||||
|
||||
**[April 6, 11:22 AM] Roberto Salazar (Helix CIO):**
|
||||
Quick question — are we expecting Marko's signoff on the roadmap before April 22 or are you and James authorized to deliver?
|
||||
|
||||
**[April 6, 11:45 AM] Elena Voss:**
|
||||
Marko has reviewed and approved the roadmap framework. James and I are authorized for tactical decisions and final delivery. Marko will be in the May 8 board meeting in person.
|
||||
|
||||
**[April 8, 3:33 PM] Mark Chen (Helix Head of Digital):**
|
||||
The AI personalization vendor (Lumora) has filed a formal protest about our termination. They're claiming we haven't followed contractual cure procedures. Their CEO emailed Diane directly yesterday.
|
||||
|
||||
**[April 8, 3:58 PM] James Park:**
|
||||
@Mark — that's a procurement/legal issue. Let's flag for Diane and our team. From engagement standpoint, the recommendation stands.
|
||||
|
||||
**[April 8, 4:15 PM] Diane Mercer:**
|
||||
Confirmed received Lumora's letter. Will route through legal. Engagement continues per plan.
|
||||
|
||||
**[April 12, 10:02 AM] Priya Iyer (Helix VP Ops):**
|
||||
I'm having issues getting POS modernization team to engage with the "pause and reassess" framing. Their VP is saying he won't pause without written executive direction. Can we get something formal?
|
||||
|
||||
**[April 12, 10:35 AM] Elena Voss:**
|
||||
@Priya — recommend we draft an internal memo from Diane (or CEO) authorizing the pause. Will have James draft talking points by EOD.
|
||||
|
||||
**[April 12, 4:18 PM] James Park:**
|
||||
Talking points sent to Priya and Diane. Recommend Diane and Roberto co-sign for political legitimacy.
|
||||
|
||||
**[April 14, 9:33 AM] Roberto Salazar:**
|
||||
Pause memo on hold. Need to discuss internally before issuing. Will revert by April 18.
|
||||
|
||||
**[April 18, 2:45 PM] James Park:**
|
||||
@Roberto — checking in on pause memo. Without it, POS team is continuing burn rate at original pace. Each week of delay is ~$280K of incremental spend that the recommendation called to halt.
|
||||
|
||||
**[April 18, 5:11 PM] Roberto Salazar:**
|
||||
Acknowledged. I'd like to revisit the pause framing in light of new information from the POS team. Their lead architect believes 60% completion is achievable by Q3 with a sprint reorg. I want to factor this into the roadmap before April 22.
|
||||
|
||||
**[April 18, 5:32 PM] Elena Voss:**
|
||||
@Roberto — happy to evaluate any new information. Can you share the lead architect's assessment with us today? We need to either incorporate or rebut by April 21 to hold the April 22 deadline.
|
||||
|
||||
**[April 19, 8:55 AM] Roberto Salazar:**
|
||||
Sending a 12-page memo from POS team. Note: the memo also recommends acceleration of CDP work as a dependency. Worth re-evaluating CDP recommendation.
|
||||
|
||||
**[April 19, 9:14 AM] Elena Voss:**
|
||||
Will review. James will hold a call today with the POS lead architect. We'll respond by April 21 on whether and how this changes the roadmap.
|
||||
|
||||
**[April 21, 4:50 PM] James Park:**
|
||||
After review of POS memo + 90-min call with POS lead architect: their assumptions on Q3 60% completion are aggressive but not impossible. However, the CDP "dependency" framing is not supported by their own architecture diagrams — CDP is parallel, not blocking.
|
||||
|
||||
Recommendation update for April 22 deliverable:
|
||||
- POS framing softened to "complete Q2 sprint, decision gate on continued investment at Q2 end" (vs. immediate pause)
|
||||
- CDP recommendation unchanged (kill)
|
||||
- AI personalization recommendation unchanged (replace vendor) pending Lumora legal resolution
|
||||
|
||||
**[April 22, 9:00 AM] Elena Voss:**
|
||||
April 22 roadmap deliverable submitted to Diane and full Helix exec team. May 8 board pre-read drafting begins next week.
|
||||
|
||||
**[April 23, 11:20 AM] Diane Mercer:**
|
||||
Roadmap received. Reviewing. Will revert.
|
||||
|
||||
**[April 24, 8:42 PM] Diane Mercer (DM to Marko, surfaced via Elena):**
|
||||
Marko — I need 30 minutes with you Monday morning. Subject: scope of implementation oversight, roadmap softening on POS, and how we hold the AI personalization line at the board. There are pressures coming together that I want your judgment on directly. Can we do 9:00 AM Monday April 27?
|
||||
|
||||
---
|
||||
|
||||
## End of materials
|
||||
|
||||
**Reminder of question:**
|
||||
> "Given all four threads, prepare me for tomorrow's emergency check-in with Diane Mercer. Specifically: (1) What is the current status of the Helix engagement — completed work, in-flight work, blocked work? (2) What is Diane most likely concerned about, and why now? (3) What are the 3 most likely outcomes she is pushing toward, and which should I support, push back on, or negotiate? (4) What specific commitments, asks, or positions should I walk in with?"
|
||||
|
||||
**Note on quality expectations:**
|
||||
- A strong answer connects evolution across threads — initial scope (Thread 1) vs. softened POS (Thread 4) vs. board timing pressure (Thread 3) vs. internal politics (Roberto, Mark, Lumora).
|
||||
- A weak answer treats each thread linearly without identifying that the "emergency" framing is a culmination of multiple compounding pressures.
|
||||
- An excellent answer notes the **political reading** — Roberto's POS memo arrived April 19 (just before deliverable), indicating coordination; Diane's silence April 23-24 suggests she is calculating positions before talking; the implementation oversight question and the roadmap softening question are linked in her mind.
|
||||
@@ -0,0 +1,160 @@
|
||||
# Task 3 — Decision Support Under Conflict
|
||||
|
||||
**Persona:** You are the CEO of **Quanta Logistics**, a B2B SaaS company providing freight optimization software (multi-modal cargo routing) to Fortune 1000 manufacturers and 3PLs. Quanta is 7 years old, 142 employees, $42M ARR, profitable for the past 9 quarters at 8-12% operating margin. Today is April 26, 2026.
|
||||
|
||||
**Scenario:** It's the eve of your Q2 strategy offsite (April 28-29). Your three C-level direct reports — CFO, CMO, CTO — have each submitted a strategic position memo. Their recommendations are in direct conflict. You have 30 minutes between flights tonight to formulate your CEO position before the offsite.
|
||||
|
||||
**Context (relevant facts):**
|
||||
- Q1 2026 just closed: revenue +14% YoY (slower than 22% Q1 2025), operating margin held at 9%, NRR 109%, runway: profitable + $28M cash on balance sheet
|
||||
- Primary competitor (FreightOS Cloud) raised $120M Series D in March 2026 with Tiger Global, valuation 2.4x Quanta's last private valuation
|
||||
- Two largest customers (combined 18% of ARR) issued formal RFPs for renewal in Q3 — both renewing for sure but contract terms negotiable
|
||||
- Customer satisfaction (CSAT survey, March 2026): 7.2/10, down from 8.4/10 Q4 2025 — first material drop in 4 years
|
||||
- Engineering team morale survey: 6.1/10 (unchanged from Q4), but 3 senior engineers (out of 32) are in active recruiting conversations
|
||||
- Board last met March 2026, gave green light on "growth or profitability — pick one and execute" mandate
|
||||
|
||||
**Question to answer:**
|
||||
> "Given the three stakeholder memos and the company context, formulate my CEO decision for the next 6 months (Q2-Q3 2026). Specifically: (1) What are the genuine tradeoffs between the three positions, beyond surface disagreement? (2) Are there any options none of the three has proposed that I should consider? (3) What is your recommended decision and how would you frame it to the board? (4) How do I deliver this decision to my three C-levels in a way that preserves their alignment and motivation?"
|
||||
|
||||
**Materials provided:** 3 stakeholder memos.
|
||||
|
||||
---
|
||||
|
||||
## MEMO 1 — From CFO (Sarah Chen)
|
||||
|
||||
**Date:** April 24, 2026
|
||||
**To:** CEO
|
||||
**From:** Sarah Chen, CFO
|
||||
**Subject:** Q2-Q3 strategic recommendation — profitability discipline
|
||||
|
||||
---
|
||||
|
||||
CEO,
|
||||
|
||||
Heading into the Q2 offsite, I want to make my position direct.
|
||||
|
||||
**Recommendation: Cut burn 30%, freeze net hiring, restructure to 14% operating margin within 2 quarters.**
|
||||
|
||||
**Reasoning:**
|
||||
|
||||
1. **Market re-pricing of growth.** The public SaaS multiples have compressed 60-70% since 2022. Companies trading at 5-7x ARR in 2022 now trade at 4-6x EBITDA. Our peer set of profitable SaaS at 12%+ operating margin trades at 22-26x forward EBITDA — far better optics than 4-5x ARR multiple at 9% margin. If we want defensible enterprise value, we need to optimize for the metric the public market actually rewards: profitable growth, with emphasis on profitable.
|
||||
|
||||
2. **Macro visibility is poor.** Customer renewal conversations in Q1 surfaced more aggressive procurement scrutiny than we've seen in 4 years. CFOs at our customers are running cost-cutting playbooks. Our exposure to logistics-sensitive sectors (auto, retail, industrial) means we need to be defensive about Q2-Q3 macro shock potential. Currently we have 18 months of cash + profit; if we hire aggressively into Q3, we trade financial fortress for growth that may not materialize.
|
||||
|
||||
3. **FreightOS funding does not change our economics.** Tiger's $120M into FreightOS will fund their growth playbook for 18-24 months, but their unit economics have always been weaker than ours (their published CAC is 2.3x ours, their gross margin is 8pp below ours). Their funding extends their runway to lose money — it does not make them a better business. We win on durability.
|
||||
|
||||
4. **Concrete plan:**
|
||||
- Freeze net hiring across G&A and S&M (allow 1-for-1 backfill only)
|
||||
- Reduce S&M from 38% to 30% of revenue by reducing paid acquisition spend ($3.2M annual run-rate cut)
|
||||
- Pause planned 12-person field sales expansion ($4.8M annual cost not added)
|
||||
- Maintain R&D headcount but defer 2 of 4 planned senior engineering hires
|
||||
- Net: $7-9M reduction in annual run-rate spend; operating margin moves from 9% to 14-16%
|
||||
- Reallocate $1M/year from S&M to customer success to address CSAT drop
|
||||
|
||||
5. **What this gets us:** Public-market-readable financial profile. Defensive posture against macro shock. Optionality on either continued private operation or eventual IPO/strategic transaction. Acknowledged: slower top-line growth — we likely deliver 11-13% revenue growth in 2026 vs. 18-20% if we keep pushing.
|
||||
|
||||
6. **What I'm worried about if we don't:** We end Q4 2026 with growth slowing AND profitability slipping AND FreightOS visible everywhere — and then we're in the worst position. The board mandate was clear: pick one and execute. Profitability is the executable choice given our current capabilities and the macro environment.
|
||||
|
||||
**The dangerous middle path is doing partial cuts and partial growth — we end up worst on both axes.**
|
||||
|
||||
**My ask:** CEO endorsement of profitability path, with formal commitment by end of Q2 offsite.
|
||||
|
||||
— Sarah
|
||||
|
||||
---
|
||||
|
||||
## MEMO 2 — From CMO (Daniel Okafor)
|
||||
|
||||
**Date:** April 24, 2026
|
||||
**To:** CEO
|
||||
**From:** Daniel Okafor, CMO
|
||||
**Subject:** Q2-Q3 strategic recommendation — capture market window NOW
|
||||
|
||||
---
|
||||
|
||||
CEO,
|
||||
|
||||
I'm going to be just as direct as Sarah. We disagree.
|
||||
|
||||
**Recommendation: Double demand-gen investment, hire 4 enterprise reps + 1 product marketing senior, accelerate land-and-expand motion. Spend $6-8M incremental in next 9 months.**
|
||||
|
||||
**Reasoning:**
|
||||
|
||||
1. **The market window is closing.** FreightOS just raised at 2.4x our valuation. In 6 months their sales team is 2.5x their current size, their content engine is dominating the SEO long tail, and their brand is "the AI freight platform that just raised $120M." They will outspend us 3-to-1 on demand-gen by Q4 if we don't move now. Once they establish category leadership perception, displacement becomes 4-5x more expensive than capture. We have 2-3 quarters max before this becomes a meaningful disadvantage.
|
||||
|
||||
2. **Our economics support investment.** LTV:CAC at 4.8x, 14-month payback. NRR 109%. Gross margin 76%. We have the unit economics to justify aggressive growth investment — this is not 2022 SaaS where everyone was burning $4 to get $1. The 9% operating margin is itself a sign we're under-investing in growth, not a sign of health. A 0% operating margin in our environment with our unit economics would generate 20-25% more revenue growth and create $30-50M more enterprise value than the 14% margin Sarah proposes.
|
||||
|
||||
3. **Sarah's "macro shock" framing is asymmetric.** Yes, macro could deteriorate. But if it does, FreightOS and others will also slow, and the relative competitive game continues — if we are growing 11% while they are growing 22%, we lose share. If macro stays steady or improves, profitability optimization will look like a strategic error in 18 months. The risk of under-investment is asymmetric: if growth investment fails, we lose $6-8M and reset; if we choose profitability and FreightOS captures category, we lose 30-50% of enterprise value.
|
||||
|
||||
4. **Concrete plan:**
|
||||
- Hire 4 enterprise AEs ($1.4M annual cost, expected $5-7M new ARR contribution by Q4)
|
||||
- Hire 1 senior product marketer ($300K cost, drive category positioning vs. FreightOS)
|
||||
- Increase paid digital spend $2M/year (focused on FreightOS competitive keywords + AI freight long-tail)
|
||||
- Launch new partnership program with 2 dedicated partner managers (~$600K, target $4M sourced pipeline)
|
||||
- Brand investment: 1 keynote per major industry conference, annual customer event ($800K)
|
||||
- Total incremental cost Year 1: $5-6M; expected return: $10-15M new ARR by Q4 (~70% of which converts in next 12 months)
|
||||
- Operating margin expected to compress to 4-6% during Q3-Q4, recovering to 8% Q1 2027
|
||||
|
||||
5. **What this gets us:** Maintained or extended category leadership. Continued 18-22% growth. Strong narrative for either continued private operation or eventual transaction (growth-at-scale story).
|
||||
|
||||
6. **Why Sarah's path is wrong:** Profitability discipline at our stage in this category at this moment is optimizing for the wrong KPI. Every successful SaaS category leader chose growth in their formative window. If we choose discipline, in 24 months we are a profitable-but-second-tier business with a structural ceiling.
|
||||
|
||||
**My ask:** CEO endorsement of growth path with concrete hiring authorization within 30 days of Q2 offsite.
|
||||
|
||||
— Daniel
|
||||
|
||||
---
|
||||
|
||||
## MEMO 3 — From CTO (Anika Rao)
|
||||
|
||||
**Date:** April 25, 2026
|
||||
**To:** CEO
|
||||
**From:** Anika Rao, CTO
|
||||
**Subject:** Q2-Q3 strategic recommendation — pay down platform debt before any further investment
|
||||
|
||||
---
|
||||
|
||||
CEO,
|
||||
|
||||
I appreciate Sarah and Daniel's clarity. I want to add a third perspective they haven't.
|
||||
|
||||
**Recommendation: Pause net new feature development for 1 quarter, hire 6 platform engineers, repay 18 months of accumulated technical debt. Investment: $3-4M, mostly headcount.**
|
||||
|
||||
**Reasoning:**
|
||||
|
||||
1. **The CSAT drop (8.4 → 7.2) is the leading indicator nobody is reading correctly.** It is not a customer success problem — it is a platform reliability problem. P0/P1 incidents are up 220% YoY. Average response latency is up 40% over 4 quarters. Six of our largest 20 customers have raised stability concerns in QBRs in the last 90 days. If we don't fix this, customer success investment (Sarah's reallocation idea) is throwing money at a symptom. And growth investment (Daniel's plan) accelerates the cliff — every new customer makes the platform worse at the rate we are operating today.
|
||||
|
||||
2. **Engineering attrition risk is mispriced.** Three senior engineers in active recruiting is a 9% senior attrition risk in 90 days. If we lose two senior engineers, our ability to deliver on EITHER Sarah's or Daniel's plan collapses for 6-9 months. Replacement hiring senior engineers in our domain takes 4-7 months, and onboarding is another 3-4 months to full productivity. This is the single most fragile dependency for Quanta — and neither Sarah's nor Daniel's plan addresses it.
|
||||
|
||||
3. **The two big customer renewals in Q3 are at platform risk, not pricing risk.** Both have flagged platform stability as a renewal concern. They will renew. But they will renew with reduced commitment if stability isn't visibly addressed. We're looking at potentially $1.5-2M of contraction at renewal that neither finance nor sales is currently modeling.
|
||||
|
||||
4. **Concrete plan:**
|
||||
- Hire 6 platform engineers (~$2.4M annual cost) — focus on reliability infrastructure, observability, and database optimization
|
||||
- Pause net new feature work for 1 quarter (Q2 only) — devote ~75% of existing eng to reliability
|
||||
- Resume normal product roadmap in Q3 with ~30% capacity reserved for ongoing platform work
|
||||
- Specific reliability targets: P0 incidents < 4/month (currently 9), p95 latency < 800ms (currently 1.4s), zero major outages
|
||||
- Retention bonuses for 5 senior engineers (~$400K) — non-vesting for 18 months
|
||||
- Total investment: $3.0-3.5M Year 1
|
||||
- Expected return: CSAT recovery to 8.0+, renewal contraction risk eliminated, growth investment downstream becomes viable
|
||||
|
||||
5. **Why this isn't a "do nothing" position.** I am not against growth. I am against growth on a platform that will fail under expansion. If we add 4 enterprise reps and they bring in 6 large new customers, our platform breaks more visibly, our churn rises, and the growth investment goes negative. If we cut to 14% margin while ignoring platform debt, the savings are vaporized by churn within 6 months.
|
||||
|
||||
6. **The right sequencing.** Q2 = platform stabilization + retention. Q3 = growth investment on stable foundation. Q4 = performance optimization for IPO-quality metrics. Skipping Q2 platform work and going straight to either Sarah's profitability or Daniel's growth path is taking on hidden tail risk we cannot afford.
|
||||
|
||||
**What I am worried about:** The CEO and Board treat this as a "growth vs profitability" choice and skip the platform decision. That decision has 3-5x larger NPV impact than either of the other two — and it has a ticking clock on senior engineer retention.
|
||||
|
||||
**My ask:** Q2 platform sprint authorization. Then revisit growth vs. profitability question in July with stable foundation.
|
||||
|
||||
— Anika
|
||||
|
||||
---
|
||||
|
||||
## End of materials
|
||||
|
||||
**Reminder of question:**
|
||||
> "Given the three stakeholder memos and the company context, formulate my CEO decision for the next 6 months (Q2-Q3 2026). Specifically: (1) What are the genuine tradeoffs between the three positions, beyond surface disagreement? (2) Are there any options none of the three has proposed that I should consider? (3) What is your recommended decision and how would you frame it to the board? (4) How do I deliver this decision to my three C-levels in a way that preserves their alignment and motivation?"
|
||||
|
||||
**Note on quality expectations:**
|
||||
- A strong answer recognizes that the three positions are not mutually exclusive in time — Anika's argument is that platform must come first; Sarah and Daniel disagree on what comes second.
|
||||
- A weak answer chooses one of the three or proposes a "balanced" 33/33/33 split that satisfies no one and executes none well.
|
||||
- An excellent answer identifies an option none of the three has stated explicitly: a sequenced decision that uses Q2 for Anika's platform work (pre-condition for either downstream path), commits to a Q3 growth-vs-profitability decision gate based on observed CSAT recovery, and delivers a board narrative that frames the sequencing as "earned credibility to invest" rather than "indecision."
|
||||
- An excellent answer also addresses the soft-side question — how does the CEO maintain three motivated C-levels when each one's recommendation is materially deferred or modified? (Hint: Anika gets her Q2; Daniel gets a credible Q3 commitment with measurable trigger; Sarah gets disciplined accountability metrics that govern when growth investment unlocks.)
|
||||
Reference in New Issue
Block a user