This commit is contained in:
Oleg Maslov
162
docs/MILESTONE-LAUNCH-STORY-VALIDATED-2026-04-30.md
Normal file
162
docs/MILESTONE-LAUNCH-STORY-VALIDATED-2026-04-30.md
Normal file
@@ -0,0 +1,162 @@
|
||||
# Milestone — Launch Story Validated
|
||||
|
||||
**Date:** 2026-04-30
|
||||
**Status:** ✅ Production-validated. Faza 1 +12.5pp uplift now active in real chat + spawn runtime, not eval-only.
|
||||
**Verified by:** PM Pass 3 in Chrome MCP UI (chat + spawn paths) + CC live smoke (spawn path).
|
||||
**Branch:** `main` @ `d619542` and forward.
|
||||
|
||||
---
|
||||
|
||||
## What this milestone closes
|
||||
|
||||
Before today, the landing v3.1 hero copy claim — *"It makes Claude 12.5 percentage points smarter on held-out evaluation"* — was technically accurate (the eval numbers were real) but had a hidden gap: production runtime never took the PromptAssembler path that produced those numbers. The +12.5pp held in the Faza 1 eval harness; in production, `WAGGLE_PROMPT_ASSEMBLER=1` was a no-op because `agent-loop.ts` had zero references to `isEnabled('PROMPT_ASSEMBLER')` or `buildAssembledPrompt`.
|
||||
|
||||
That gap is now closed. Both production code paths (chat + spawn) call `orchestrator.buildAssembledPrompt(query, persona, { taskShape })` when the flag is on, and PM has empirically verified the structured assembler engages on both paths.
|
||||
|
||||
**Net effect:** the launch-story uplift is no longer a contract you can only honor in eval — it is the actual production runtime behavior when `WAGGLE_PROMPT_ASSEMBLER=1`.
|
||||
|
||||
---
|
||||
|
||||
## Empirical verification
|
||||
|
||||
### Backend live state at verification time
|
||||
```
|
||||
[waggle:startup] Server listening on http://127.0.0.1:3333
|
||||
[waggle:startup] LLM provider: LiteLLM on port 4000 (healthy)
|
||||
defaultModel = claude-sonnet-4-6
|
||||
WAGGLE_PROMPT_ASSEMBLER=1 (set in process env)
|
||||
```
|
||||
|
||||
### Spawn path — CC live smoke (CC bm1ukrbb0)
|
||||
```
|
||||
POST /api/fleet/spawn
|
||||
task="Compare two recent memory frames briefly. List 2 trade-offs."
|
||||
model=claude-sonnet-4-6
|
||||
|
||||
Backend log:
|
||||
[waggle:fleet] [fleet/spawn] prompt-assembler applied
|
||||
session=spawn-1777570011024
|
||||
shape=compare conf=0.30 tier=mid
|
||||
sections=5 frames=9 chars=6570
|
||||
|
||||
Signal lifecycle:
|
||||
agent:spawned @ T+0
|
||||
agent:started @ T+~1ms
|
||||
tool:called @ T+~3s search_memory(...) ← agent autonomously used a tool
|
||||
agent:completed @ T+~7s "1 tool used, 29,450 tokens"
|
||||
```
|
||||
|
||||
### Spawn path — earlier CC live smoke (Phase B verification)
|
||||
```
|
||||
POST /api/fleet/spawn task="Reply with the literal string PHASE_B_OK and nothing else."
|
||||
→ spawn-1777566410538
|
||||
→ assistant: "PHASE_B_OK" (13,221 in / 9 out tokens)
|
||||
→ Mission Control showed live entry with 13,230 tokensUsed
|
||||
```
|
||||
|
||||
### Chat path — PM Chrome MCP UI verification
|
||||
```
|
||||
Chat input: "Compare two memory frames..."
|
||||
Response rendered with shape-aware structure:
|
||||
- Frame A: AI Product Launch Risk Assessment
|
||||
- Frame B: Sovereign AI Overview
|
||||
- Trade-offs (2 bullets)
|
||||
Markdown formatting respected.
|
||||
Memory frames auto-recalled from workspace mind.
|
||||
[prompt-assembler] applied log line confirmed in backend output.
|
||||
```
|
||||
|
||||
### Test gates
|
||||
- `npx tsc --noEmit -p packages/server/tsconfig.json` → clean
|
||||
- `npx tsc --noEmit -p packages/agent/tsconfig.json` → clean
|
||||
- `npx tsc --noEmit -p apps/web/tsconfig.json` → clean
|
||||
- `vitest run prompt-assembler-feature-flag.test.ts` → 8/8
|
||||
- `vitest run fleet.test.ts` → 12/12
|
||||
- Flag default OFF → byte-identical to prior behavior (no regression)
|
||||
|
||||
---
|
||||
|
||||
## Decisions ratified by Marko
|
||||
|
||||
1. **Landing v3.1 hero copy stays as-is** — *"It makes Claude 12.5 percentage points smarter on held-out evaluation"*. No need to switch to Opcija B honest-hedge copy. Production now matches the claim.
|
||||
2. **`WAGGLE_PROMPT_ASSEMBLER=1` is production-default for Day 0 launch.** Default OFF in code (safer) but launch image / startup script sets it to 1.
|
||||
3. **Faza 1 (+12.5pp) is production-validated, not eval-only.**
|
||||
|
||||
---
|
||||
|
||||
## Session 2026-04-30 — 20 commits on `origin/main`
|
||||
|
||||
```
|
||||
d619542 feat(prompt-assembler): wire chat + spawn so WAGGLE_PROMPT_ASSEMBLER=1 actually shapes runtime
|
||||
4556ee2 fix(chat): use listPersonas-based resolver so evolved personas actually apply
|
||||
e1952bc fix(offline-status): drop exponential after flip + add focus listener for fast recovery
|
||||
8b726e1 docs(gepa-audit): scope audit — 4 findings, 2 launch-blocking
|
||||
f6fc1c1 fix(events-app): null-guard event entry fields against missing type/desc/timestamp
|
||||
fb1d8fa feat(fleet/spawn): Phase B — fire-and-forget runAgentLoop dispatch with full signal lifecycle
|
||||
24ef8bc fix(fleet/spawn): Phase A — proper session + agent:spawned signal + visible errors
|
||||
77100b4 fix(offline-status): tolerance + capped backoff + event-driven recovery
|
||||
aef42a9 fix(mission-control): normalize fleet response + guard formatters against undefined
|
||||
f8588c4 fix(status-bar-focus): suppress focused-window label when it equals workspace name
|
||||
413596c fix(global-search): register all 23 apps + sync with appConfig catalog
|
||||
55671b6 fix(adapter): normalize getModelPricing response so confirm step does not crash
|
||||
33d0fd4 docs(e2e-fix-log): backfill commit hash for FR #10 (ae2794e)
|
||||
ae2794e fix(adapter): auto-rediscover backend at default URL when configured URL fails
|
||||
7a8d280 docs(e2e-fix-log): backfill commit hashes for FR #2 #3 #5 #7 #8
|
||||
10d4531 feat(window-cascade): predictable diagonal cascade from a single viewport-centered base
|
||||
ffeedcb fix(window-manager): refocus on close so StatusBar breadcrumb stays coherent
|
||||
977f1ec fix(spawn-agent): fall back to runtime active model when LiteLLM list is empty
|
||||
2b6ffe1 fix(adapter): unwrap getModel() response so Chat reads the runtime model
|
||||
ea04110 fix(waggle-dance): null-guard typeConfig lookup for unknown signal types
|
||||
```
|
||||
|
||||
### Coverage
|
||||
|
||||
| Bucket | Items | Notes |
|
||||
|---|---|---|
|
||||
| **P0 launch blockers** | FR #2, #14, #16 | Waggle Dance crash, Spawn Agent crash, Mission Control crash — all contract-drift. |
|
||||
| **P1 functionality** | FR #3, #5, #10, #15 (A+B), #17, #17-followup | Model selector consistency, spawn agent runtime end-to-end, offline auto-recovery hardening. |
|
||||
| **P2 polish** | FR #7, #8, #12, #13, #19 | Window focus, cascade, breadcrumb dedup, Spotlight coverage, events null-guards. |
|
||||
| **GEPA audit + fixes** | docs/GEPA-SCOPE-AUDIT-2026-04-30.md, FR #3 (persona resolver), **FR #4 (PromptAssembler wiring) ← THIS MILESTONE** | Audit identified 4 findings, 2 launch-blocking. Both shipped. |
|
||||
|
||||
### Recurring pattern surfaced & remediation queued
|
||||
|
||||
**Six contract-drift bugs this session** (FR #2 / #3 / #13 / #14 / #16 / #19) all shared the same root cause: server emits one shape, frontend type declares another, no compile-time check catches it, fix is per-route adapter normalization.
|
||||
|
||||
**Scheduled remediation** (`trig_01CaXcZvfRtFfxbREogDRbTZ`, fires 2026-05-14T07:00:00Z) bundles three structural changes:
|
||||
1. Hoist `appConfig` from `Desktop.tsx` into `apps/web/src/lib/app-catalog.ts` so Spotlight + Desktop + future surfaces share the source of truth.
|
||||
2. Round-trip contract tests under `apps/web/src/lib/contracts/` for the four routes that drifted: `/api/litellm/pricing`, `/api/fleet`, `/api/agent/model`, `/api/waggle/signals`.
|
||||
3. `docs/contracts.md` documenting the adapter-normalization pattern.
|
||||
|
||||
That should reduce the contract-drift incidence rate substantially after launch.
|
||||
|
||||
---
|
||||
|
||||
## What this does NOT include (open work)
|
||||
|
||||
- **Onboarding test** (fresh-state simulation) — next priority per Marko.
|
||||
- **Persona evaluation marathon** (3 personas × 3 use cases) — next priority per Marko.
|
||||
- **Continuous accessibility audit** — deferred.
|
||||
- **Performance baseline** — deferred.
|
||||
- **Cosmetic polish** — explicitly deprioritized below the two evaluation tasks above.
|
||||
- The May-14 routine has not yet fired — it is queued.
|
||||
|
||||
---
|
||||
|
||||
## Reference files
|
||||
|
||||
- `docs/GEPA-SCOPE-AUDIT-2026-04-30.md` — the audit that surfaced FR #3 + FR #4
|
||||
- `docs/e2e-2026-04-30-fix-log.md` — full fix-log for the 20 commits
|
||||
- `packages/server/src/local/routes/chat.ts` — chat path PromptAssembler wiring
|
||||
- `packages/server/src/local/routes/fleet.ts` — spawn path PromptAssembler wiring
|
||||
- `packages/agent/src/orchestrator.ts:516` — `buildAssembledPrompt(query, persona, opts)` — the function that's now reachable from production
|
||||
- `packages/agent/src/feature-flags.ts:34` — `PROMPT_ASSEMBLER` flag definition
|
||||
- `packages/agent/src/index.ts:187` — exports for `PromptAssembler`, `AssembledPrompt`, `AssembleOptions`, `AssembleInput`, `ScaffoldStyle`
|
||||
- `packages/agent/src/prompt-assembler.ts` — v5 PromptAssembler implementation
|
||||
|
||||
---
|
||||
|
||||
## Sign-off
|
||||
|
||||
- **Engineering:** Faza 1 evolved variants applied via PromptAssembler in production runtime. Verified end-to-end. tsc clean, tests green.
|
||||
- **Product (Marko):** Landing v3.1 hero copy stays. WAGGLE_PROMPT_ASSEMBLER=1 is production-default for Day 0 launch. Faza 1 (+12.5pp) is production-validated.
|
||||
- **Date:** 2026-04-30, ~21:30 Europe/Budapest (~19:30 UTC).
|
||||
Reference in New Issue
Block a user