Files
waggle-os/benchmarks/results/pilot-2026-04-26/pilot-task-1-B.jsonl
Oleg Maslov 0c3e2ead3b
Some checks failed
Installer Smoke / installer-smoke (push) Has been cancelled
moving
2026-09-02 10:10:29 +02:00

2 lines
7.9 KiB
JSON
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
{"task_id":"task-1","cell_id":"B","model":"claude-opus-4-7","configuration":"memory-harness","candidate_response":"# MEMO: Q2Q4 2026 Critical Risks & Action Plan\n\n**To:** Board of Directors\n**From:** CFO, NorthLane\n**Date:** April 26, 2026\n**Re:** Top 3 Risks for Q2Q4 2026 and Recommended Actions\n\n---\n\n## Framing\n\nNorthLane enters Q2 with $18.9M cash, 18 months of runway, and three converging pressures: a doubling YoY operating loss, a well-funded competitor (ChainSight, $80M Series C) defining the \"AI-native\" category we sell into, and the first quarter of declining customer health in 8 quarters (NRR 102%, NPS -16pp, logo churn at an all-time high). These are not independent issues — they are a single feedback loop. The three risks below are prioritized because each is both an existential threat *and* an accelerant of the others. Other concerns (marketing CAC inflation, SDR ROI, mobile rewrite, SSO scope) are real but downstream of these three.\n\n---\n\n## Risk 1 — Burn / Runway Compression (Highest Priority)\n\n**Why critical:** Q1 burn was $1.05M (vs. $0.55M Q1'25, +91%); operating loss doubled YoY while new ARR was 65% of plan. Sequoia has explicitly framed Q3 as a path-to-default test and ruled out flat bridge rounds. At current trajectory runway falls below 12 months by Q4, which collapses negotiating leverage on every other decision.\n\n**Action plan (own: CFO; milestone: end of Q2):**\n1. **Restructure to a 30%+ burn reduction by Q3 exit** (board ask). Target ~$0.7M/mo burn run-rate.\n2. **Reallocate, don't just cut.** Kill or pause: Events/sponsorships ($0.35M, 0.9x ROI), 2 of 3 SDR seats (0.6x ROI), the mobile rewrite, and enterprise SSO unless tied to a signed deal. Estimated savings: $2.83.4M annualized.\n3. **Redirect ~30% of savings** into the two highest-ROI areas: partner program (5.0x return, currently 1 person) and engineering retention/hiring for AI roadmap.\n4. **Institute monthly burn reviews** with board (already required) plus a tripwire: if Q2 net new ARR < $0.75M, trigger a deeper Q3 RIF plan pre-approved now.\n\n---\n\n## Risk 2 — Competitive Displacement by ChainSight (1218 Month Window)\n\n**Why critical:** 7 of 9 Q1 losses went to ChainSight; \"AI roadmap credibility\" was the cited reason in 4 of 7. Win rate fell 6pp, sales cycle stretched 23 days, ACV down 8%, and one churned logo (AcmeMfg, $110K) explicitly switched to them. ChainSight's SAP/Ariba partnership and Copilot launch are creating category-defining gravity. Strategy lead's 1218 month displacement window is consistent with the funnel data.\n\n**Action plan (own: CEO + VP Product; milestone: May 15 moat memo, Q3 ship):**\n1. **Narrow the AI roadmap to one defensible wedge** rather than \"predictive analytics + agent orchestration.\" Recommend: predictive analytics tied to mid-market manufacturer-specific data (our ICP advantage; ChainSight is moving upmarket via SAP). Ship a credible Q3 GA, not a demo.\n2. **Re-position around ICP, not platform parity.** Stop competing on \"AI-native\" generality; own \"the supply-chain visibility platform purpose-built for mid-market manufacturers.\" Refresh website, sales deck, and analyst briefings by end of Q2.\n3. **Double the partner channel** (currently 5.0x ROI, 1 FTE). Add 12 partner managers funded from the SDR/events cuts above; pursue 2 ERP/MRP integration partnerships as a counter to ChainSightSAP.\n4. **Competitive deal SWAT:** named win-back program for the 9 Q1 losses; pricing flexibility approved up to -15% on multi-year deals where ChainSight is finalist.\n\n---\n\n## Risk 3 — Engineering Attrition + Customer Health Doom Loop\n\n**Why critical:** This is the hidden risk the board independent director flagged. Two senior engineers left in Q1; two more are looking. P0/P1 bugs +73%, MTTR nearly doubled, postmortem completion fell from 78% → 31%. Engineering bandwidth was *taken from CS integrations to fund the AI roadmap* — directly producing the implementation-quality complaints that drove ToolsmithIndustrial's churn and put $1.4M ARR of at-risk accounts in play. NRR fell from 118% → 102%. If this loop continues, we lose the AI ship date *and* the installed base that funds it — simultaneously undermining Risks 1 and 2.\n\n**Action plan (own: CEO + VP Eng + CS Director; milestone: stabilize by end of Q2):**\n1. **Retention package for top 6 engineers within 2 weeks:** refreshed equity grants (2-year cliff-extending), targeted comp adjustments to market, and explicit scope relief. Budget: ~$400600K equity, modest cash. Cheaper than one senior backfill at 87-day time-to-fill.\n2. **Cut engineering scope to two tracks:** AI wedge (Risk 2) and customer integration/quality. Pause mobile rewrite and SSO. This also funds Risk 1.\n3. **Stand up a customer-save program for the $1.4M at-risk ARR:** executive sponsors on the 3 \"considering alternatives\" accounts, usage-recovery plays on the 6 declining accounts, re-engagement on the 11 dormant. Target: hold NRR ≥105% for Q2.\n4. **Re-establish quality bar:** postmortem completion back to >70%, P0/P1 backlog burned down by end of Q2 — these are leading indicators of the next churn cohort.\n\n---\n\n## How These Interact (and Why Sequencing Matters)\n\n- **Risk 3 feeds Risk 2:** without engineering stability we cannot ship the Q3 AI wedge that closes the competitive gap.\n- **Risk 2 feeds Risk 1:** without competitive differentiation, win rates and NRR keep eroding, and burn cuts alone won't extend runway because the numerator (revenue) keeps shrinking.\n- **Risk 1 constrains Risks 2 and 3:** we cannot hire or invest our way out; every dollar must be reallocated, not added.\n\n**The integrated bet:** Cut ~$3M of low-ROI spend (Risk 1), redeploy a portion into engineering retention and partner channel (Risks 2 & 3), and narrow product scope to one defensible AI wedge shipped by Q3. This is the only path that satisfies the board's Q3 go/no-go on both burn *and* competitive positioning.\n\n**Decisions requested from the board:** (1) approval of $400600K retention equity pool; (2) endorsement of scope cuts (mobile, SSO, events, 2 SDRs); (3) confirmation of Q2 tripwire for deeper RIF if ARR <$0.75M.","candidate_latency_ms":52093,"candidate_tokens_in":13870,"candidate_tokens_out":2684,"candidate_cost_usd":0.40935,"loop_exhausted":false,"steps_taken":3,"retrieval_calls":2,"judge_opus":{"completeness":5,"accuracy":5,"synthesis":5,"judgment":5,"actionability":5,"structure":5,"rationale":"The memo integrates all seven documents, cites specific figures accurately, explicitly addresses risk interactions, and ends with concrete board asks including dollar amounts, owners, and tripwires. No notable weakness across dimensions.","overall_verdict":"PASS_STRONG","mean":5},"judge_gpt":{"completeness":5,"accuracy":4,"synthesis":5,"judgment":5,"actionability":5,"structure":5,"rationale":"Accuracy is the lowest score because while the memo is largely faithful to the materials, a few recommendations introduce specifics not directly grounded in the source documents (e.g., exact savings estimates, reducing 2 of 3 SDRs, and the suggested retention grant mechanics). These are reasonable inferences, but they go beyond explicit evidence provided.","overall_verdict":"PASS_STRONG","mean":4.833333333333333},"judge_minimax":{"completeness":5,"accuracy":5,"synthesis":5,"judgment":5,"actionability":5,"structure":5,"rationale":"All six dimensions score at maximum; the memo demonstrates textbook CFO-quality analysis that correctly identifies the three systemic risks, grounds every claim in material data, shows how they interact as a feedback loop, and concludes with exactly three board decisions it can act on next week.","overall_verdict":"PASS_STRONG","mean":5},"trio_mean":4.944444444444444,"trio_strict_pass":true,"trio_critical_fail":false,"manifest_anchor":"pilot-2026-04-26-v1","head_sha":"b7e19c557fdbc42f2d0a3c3213176aa4d790f7a2","ts_iso":"2026-04-26T00:48:10.266Z","cell_cost_usd":0.5669907}