This commit is contained in:
Oleg Maslov
162
docs/audits/2026-05-29-prod-readiness/REPORT.md
Normal file
162
docs/audits/2026-05-29-prod-readiness/REPORT.md
Normal file
@@ -0,0 +1,162 @@
|
||||
# Waggle OS — Production-Readiness Hardening Audit
|
||||
|
||||
**Date:** 2026-05-29
|
||||
**Scope:** Shippable product surface — sidecar (packages/server), frontend (apps/web), Tauri glue (app/), and shipped dependencies (packages/shared, packages/core / hive-mind-core, packages/agent).
|
||||
**Method:** Build/test/lint gates + multi-perspective verified findings (refuted findings already dropped upstream). Verdicts: confirmed (verified end-to-end), unverified (plausible, evidence cited, not independently re-walked in this synthesis pass).
|
||||
|
||||
---
|
||||
|
||||
## 1. Executive Summary
|
||||
|
||||
The product builds and the unit suite is green (6627/6629; the 2 non-passes are one network-timeout infra flake and one skip). However, the audit surfaces a systemic input-validation gap on the sidecar HTTP boundary and a LAN-exposure-by-default posture that together undermine the project security contract (CLAUDE.md section 7).
|
||||
|
||||
The single most serious issue (R1-001) is a complete LAN auth-bypass: the sidecar binds to 0.0.0.0 by default and the unauthenticated /health endpoint returns the very bearer token used to authenticate every other route. Any co-located host can read the token and drive the full authenticated API (backup exfiltration, chat, file r/w, data erase).
|
||||
|
||||
Three themes dominate beyond that:
|
||||
|
||||
- Boundary validation is patchy by construction: a full set of Zod schemas exists in packages/shared/src/schemas.ts but is never wired in (R6-007), so every fs-write route (chat, tasks, ingest, documents) hand-rolls ad-hoc checks and several omit the existing assertSafeSegment guard, yielding path-traversal write sinks (R6-001, R6-002, R1-004, R2-005).
|
||||
- Billing path has revenue/entitlement defects: /api/stripe/sync upgrades tier without verifying payment (R1-002), and checkout ignores billingPeriod and the 4-var price contract entirely (R1-003).
|
||||
- Desktop lifecycle + resilience gaps: orphaned sidecar on exit (R7-002), inert watchdog (R7-003), wrong updater repo slug so auto-update 404s forever (R7-004), no top-level React error boundary (R4-001), and a memory-moat regression where harvest cognify indexes with a MOCK embedder, poisoning semantic recall (R3-001).
|
||||
|
||||
Gate blockers for ship: lint is non-functional (no root flat config) and tauri-tsc fails (empty app/src/). Neither is a source defect, but both mean two of the five quality gates currently provide zero signal.
|
||||
|
||||
Counts (post-dedup): 15 critical/high, 30 medium, 17 low. Recommended fix campaign is 6 phases, front-loaded on the network-exposure + traversal cluster.
|
||||
|
||||
---
|
||||
|
||||
## 2. Gate Results
|
||||
|
||||
| Gate | Status | Summary |
|
||||
|---|---|---|
|
||||
| build-web (npm run build) | PASS | Vite v5.4.21, 2373 modules, 28.51s, exit 0. Advisory warnings only (CSS @import order, dynamic/static import chunking, 1.68 MB main chunk over 500 kB). Does NOT typecheck the sidecar. |
|
||||
| build-packages (npm run build:packages) | PASS | tsc --build chain (shared, core, agent, server), exit 0, zero diagnostics. Verified with --force clean rebuild + forced server rebuild (covers dirty telegram.ts). Genuine green. |
|
||||
| lint (npm run lint) | FAIL (config) | ESLint 9.39.4 aborts (exit 2): no root eslint.config flat file and no .eslintrc. eslint . lints zero files. Only apps/web/eslint.config.js exists. Broken gate wiring, not a code defect. |
|
||||
| test (npm run test -- --run) | PARTIAL | Vitest 3.2.4, exit 1. 6627 passed / 1 failed / 1 skipped (462 files). Sole failure = marketplace-sync.test.ts network timeout, infra flake. |
|
||||
| tauri-tsc (tsc -p app/tsconfig.json) | FAIL (config) | TS18003 No inputs were found: app/src/ has zero .ts/.tsx files. Contradicts CLAUDE.md section 2. No Tauri TS glue to typecheck. |
|
||||
|
||||
---
|
||||
|
||||
## 3. Findings (severity-sorted, post-dedup)
|
||||
|
||||
Merges performed (kept highest severity + strongest verdict, unioned evidence):
|
||||
- R1-001 superset of R2-001 (0.0.0.0 bind + /health token leak)
|
||||
- R1-004 superset of R6-003 (ingest workspaceId traversal)
|
||||
- R2-003 superset of R1-006 (CORS startsWith bypass)
|
||||
- R2-006 superset of R1-013 (/api/debug/logs leak)
|
||||
- R2-002 superset of R1-007 + R6-004 (OAuth reflected XSS; CSP mitigates to low)
|
||||
|
||||
| ID | Sev | Surface | File:line | Issue | Fix | Verdict |
|
||||
|---|---|---|---|---|---|---|
|
||||
| R1-001 (+R2-001) | critical | sidecar bootstrap + /health | service.ts:202, index.ts:278,2329, security-middleware.ts:236,278-283 | Default 0.0.0.0 bind + auth-exempt /health returns wsSessionToken = full LAN auth bypass | Bind 127.0.0.1 default (widen only on WAGGLE_HOST); stop returning wsToken from /health | confirmed |
|
||||
| R1-002 | high | stripe/sync.ts:37-74 | POST /api/stripe/sync | Upgrades tier without checking payment_status; metadata fallback unlocks PRO/TEAMS for unpaid session | Require payment_status paid (or complete + active sub) before updateUserTier | confirmed |
|
||||
| R1-003 | high | stripe/checkout.ts:29-50 | POST checkout | Ignores billingPeriod; always single legacy price so annual billed at monthly; or NO_PRICE_CONFIGURED under 4-var contract | Select price by billingPeriod from 4-var env w/ legacy fallback (mirror tierFromPriceId) | confirmed |
|
||||
| R1-004 (+R6-003) | high | ingest.ts:149-154,420-433 | POST /api/ingest | addToFileRegistry builds fs path from unvalidated body workspaceId so traversal write | Resolve-and-confirm under dataDir/workspaces (mirror allowedRoot guard) | confirmed |
|
||||
| R2-003 (+R1-006) | high | index.ts:1904 | CORS plugin | origin.startsWith() so attacker prefix host passes; localhost auth-exempt returns responses | ALLOWED_ORIGINS.includes(origin) exact match | confirmed |
|
||||
| R3-001 | high | harvest.ts:409-419 | harvest cognify | Hard-codes mock embedder; vector-indexes harvested frames with meaningless vectors so unretrievable by semantic search | Reuse fastify.embeddingProvider; skip indexing when provider is mock | confirmed |
|
||||
| R4-001 | high | App.tsx:13-29 | app/route shell | No top-level error boundary; any render throw white-screens whole app | Wrap Routes/Index + Desktop chrome+overlays in recoverable boundary | confirmed |
|
||||
| R6-001 | high | chat.ts:530-546, chat-persistence.ts:21-46 | POST /api/chat | Unvalidated workspace/session from body so sessions jsonl traversal write/read on hot path | assertSafeSegment(workspace/session) before persist, 400 | confirmed |
|
||||
| R6-002 | high | tasks.ts:26-50,92-137 | /api/workspaces/:id/tasks | Unvalidated :id in tasksPath/writeTasks so mkdir+write traversal | assertSafeSegment(id) at top of each handler | confirmed |
|
||||
| R7-002 | high | lib.rs:132-140 | Tauri lifecycle | No RunEvent::Exit handler; tray Quit unwired so sidecar orphaned, holds port 3333, next launch fails | RunEvent::Exit handler kills child regardless of frontend | confirmed |
|
||||
| R7-003 | high | service.rs:228-231 | watchdog | Detects downed sidecar, emits restart event, never respawns; listener unwired so permanent backend disconnect | Self-healing watchdog: clear dead Child + spawn_service_sync | confirmed |
|
||||
| R7-004 | high | tauri.conf.json:56-59 | updater | Endpoint points to marolinik/waggle (actual repo waggle-os) so latest.json 404s every check | Correct slug in tauri.conf.json + 3 manifest URLs in release.yml | confirmed |
|
||||
| R8-001 | high | LauncherApp.tsx:365,447-462; tool-launcher.ts:275-291 | AI-OS hooks | LAUNCH_COHORT routes cursor/claude-desktop hook actions to binless stub packages so npx always fails | HOOKS_COHORT = claude-code only; gate runHookCommand + buttons on it | confirmed |
|
||||
| R9-002 | high | protobufjs@7.5.4 | hive-mind-core + apps/web | Below 7.5.5 fix for critical RCE GHSA-xq3m-2v4x-88gg (9.8); ships via transformers + posthog OTLP | overrides protobufjs >=7.5.5 (targeted) | confirmed |
|
||||
| R9-005 | high | js-cookie@3.0.5 via @clerk/shared | apps/www auth + server clerk | Prototype-hijack cookie-attr injection GHSA-qjx8-664m-686j (7.5) on auth surface | Bump @clerk so @clerk/shared resolves js-cookie >3.0.5 | confirmed |
|
||||
| R1-005 | medium | backup.ts:377-381 | restore | startsWith(dataDir) w/o sep so sibling-dir escape; write uses unvalidated targetPath | Guard resolved===root or startsWith(root+sep); write to resolved | confirmed |
|
||||
| R2-006 (+R1-013) | medium | index.ts:1920-1937 | /api/debug/logs | Returns vault key NAMES + 500 audit rows; no same-origin gate (recon, esp. w/ R1-001) | isLocalOrigin guard; drop providerKeys; global setErrorHandler | confirmed |
|
||||
| R3-002 | medium | subagent-orchestrator.ts:154-169 | subagent | Circular-dep failure mints NEW worker id so duplicate orphan + permanent pending ghost | Reuse stepWorkerIds.get(name); update in place | unverified |
|
||||
| R3-003 | medium | retry-policy.ts:72-81 | 429 retry | HTTP-date Retry-After so parseInt NaN so setTimeout(0) so hammers endpoint, burns retries | Number.isFinite guard + default fallback | unverified |
|
||||
| R3-004 | medium | frames.ts:251-267 | harvest dedup | findDuplicate SELECTs + SHA-256-hashes 500 rows per insert so O(n*500) blocks event loop | Indexed content_hash column; or per-batch hash set + skipDedup | unverified |
|
||||
| R4-002 | medium | BackupApp.tsx:93 | backup UI | Restore button has no onClick/handler, dead control on data-safety app | Wire to file-picker POST /api/restore, or remove | confirmed |
|
||||
| R4-003 | medium | useChat.ts:130-133 | chat stream | msgs last role unguarded; empty array mid-stream so render crash | Guard empty array return early | unverified |
|
||||
| R4-004 | medium | HarvestTab.tsx:523-538 | harvest UI | async onClick no try/catch so unhandled rejection, stale row, no feedback | try/catch + toast; fetchSources after success | unverified |
|
||||
| R4-005 | medium | ChatApp.tsx:520-534,1102 | chat pin | handlePin async no error handling fire-and-forget so silent pin failure | try/catch + toast | unverified |
|
||||
| R4-006 | medium | BackupApp.tsx:18-22 | backup metadata | No response.ok before json(); non-2xx so misleading No backups yet | Check r.ok; error+retry panel | unverified |
|
||||
| R4-007 | medium | ConnectorsApp.tsx:66-67,87-102 | connectors | Single shared tokenInput across connectors (wrong-cred footgun); connect failure console-only | Reset inputs on expanded change; toast in catch | unverified |
|
||||
| R4-008 | medium | ChatApp.tsx:644-651 | chat file-drop | Ingest failures swallowed to console; user believes file ingested | Accumulate failures, success/failure toast | unverified |
|
||||
| R6-005 | medium | browse.ts:26-94 | /api/browse/local + mkdir | Resolves any absolute path; enumerates host FS + mkdir anywhere; no confinement | Same-origin gate; rate-limit mkdir; no extension origins | unverified |
|
||||
| R6-006 | medium | workspace-context.ts:254,313,350 | context helper | Session-dir path from unvalidated workspaceId so existence/count probe | assertSafeSegment(workspaceId) at callers/helper | unverified |
|
||||
| R6-008 | medium | server tests (chat/tasks/ingest) | tests | No traversal test coverage for unguarded fs-write routes; green unit test gives false assurance | Per-route 400-on-traversal tests + assert no out-of-root file | unverified |
|
||||
| R5-001 | medium | ContextMenu.tsx:32,43-82 | context menu | Enter indexes actionItems (skips disabled) but render index counts disabled so wrong item fires | Compute currentActionIndex with same filter | unverified |
|
||||
| R5-002 | medium | TelegramDigestCard.tsx:136,160 | light-mode | Hardcoded text-emerald-400 saved label fails contrast on cream surface | Semantic success/status-healthy token | unverified |
|
||||
| R5-003 | medium | LauncherApp.tsx:289-392 | light-mode | Dark-only bg-950 banners + pale text-300 so black islands on cream | Semantic adaptive tokens (bg-destructive/10 etc.) | unverified |
|
||||
| R1-008 | medium | cron.ts:20-33,73-76 | GET /api/cron | Unguarded JSON.parse(job_config); one corrupt row so 500 breaks ENTIRE list | try/catch fallback to empty obj + log | unverified |
|
||||
| R1-009 | medium | connectors.ts:16-24 | health probe | No try/catch around registry.healthCheck() so unhandled 500 + raw error leak | try/catch to status error or 502 | unverified |
|
||||
| R1-010 | medium | agent-run.ts:38-40,85-92 | /api/agent/run | Module-level LiteLLM URL/key snapshot ignores built-in-proxy fallback so broken for Anthropic-only default | Read litellmUrl/key from server state at request time | unverified |
|
||||
| R1-011 | medium | webhook.ts:22-32,73-124 | config write | Non-atomic read-modify-write of config.json + processed-events; concurrent so lost tier / double-process | Atomic temp+rename + mutex; idempotency in SQLite | unverified |
|
||||
| R1-012 | medium | files.ts:117-120,312-319 | upload | getRawBody buffers entire body before size check so multi-GB OOM | Abort stream over MAX_UPLOAD_SIZE in data handler; or fastify multipart | unverified |
|
||||
| R2-004 | medium | security-middleware.ts:276-294 | auth | No Host-header validation; DNS rebinding defeats localhost-trust exemption | Host allowlist; pair with 127.0.0.1 bind | unverified |
|
||||
| R2-005 | medium | documents.ts:36,68,88 | documents | Unvalidated :id so workspaces/<id>/documents.json out-of-root write/read | assertSafeSegment(id) + (name) | unverified |
|
||||
| R9-001 | medium | drizzle-orm@0.44.7 | sidecar/worker/launcher | GHSA-gpj5-g38j-94v9 SQLi via identifiers (<0.45.2); no sql.identifier sink, Postgres-only | Bump >=0.45.2 (major; re-tsc) | confirmed |
|
||||
| R9-003 | medium | fastify@5.8.4 | sidecar HTTP | GHSA-247c body-schema bypass via Content-Type space (<=5.8.4); few routes use Fastify schema | Bump >5.8.4 (clears fast-uri); re-tsc + tests | confirmed |
|
||||
| R9-004 | medium | clerk/fastify@3.1.5 | team auth | GHSA-w24r authz bypass org/billing/reverification (<=3.1.15); affected helpers not invoked | Bump >3.1.15 | confirmed |
|
||||
| R9-006 | medium | lodash@4.17.23 | apps/web recharts + sidecar archiver | code-injection via template (<=4.17.23); template not reachable (hygiene) | overrides lodash >=4.17.24 | confirmed |
|
||||
| R9-007 | medium | xmldom@0.8.11 via mammoth | sidecar docx ingest | XML injection via CDATA serialization (<0.8.12) | overrides xmldom >=0.8.13; verify mammoth | unverified |
|
||||
| R9-008 | medium | tmp@0.2.5 via exceljs | sidecar xlsx export | Path traversal via prefix/postfix (<0.2.6) | overrides tmp >=0.2.6 | unverified |
|
||||
| R9-009 | medium | fastify/static@9.0.0 | sidecar static | dir-listing traversal + route-guard bypass via encoded sep (<=9.1.0) | Bump >9.1.0; verify assets served | unverified |
|
||||
| R2-002 (+R1-007,R6-004) | low | oauth.ts:190-194,268-286 | OAuth callback | Reflects untrusted query + upstream body into unescaped HTML; CSP script-src self blocks exec so markup/phishing only | HTML-escape (escapeXml exists); or return JSON | confirmed |
|
||||
| R4-009 | low | useAgentStatus.ts:15-40 | hook | Initial poll() setState after unmount (no cancelled guard) | cancelled flag checked after await | unverified |
|
||||
| R4-010 | low | ChatWindowInstance.tsx:194-203 | hook | fetchTeam lacks cancelled guard its siblings have so setState after unmount | if cancelled return after getTeamMembers | unverified |
|
||||
| R3-005 | low | search.ts:172-178 | keyword search | Comment promises LIKE fallback that does not exist; FTS5 parse error so 0 hits, false no-memory | Implement LIKE fallback OR fix comment | unverified |
|
||||
| R3-006 | low | sse-parser.ts:108-122 | streaming | Tool-call deltas missing index collapse to 0 so parallel tool args concatenated/corrupt | Synthetic index per distinct tc.id | unverified |
|
||||
| R3-007 | low | knowledge.ts:131-135 | entity search | searchEntities does not escape LIKE metachars so percent/underscore wildcard, literal percent unfindable | Escape metachars + ESCAPE clause | unverified |
|
||||
| R3-008 | low | agent-loop.ts:256-308 | abort | Signal checked only between turns; fetch + reader do not forward so in-flight stream runs to completion | Pass signal to fetch + check in read loop | unverified |
|
||||
| R5-004 | low | UpgradeModal/TrialExpiredModal/EraseDataDialog | a11y | Modals lack role=dialog/aria-modal, Escape, focus trap | Add role/aria-modal + Escape + focus (reuse pattern) | unverified |
|
||||
| R5-005 | low | AppWindow.tsx:283-300 | window chrome | Minimize + Maximize identical bg-primary/40 dots, indistinguishable without hover | Distinct colors or lucide icons | unverified |
|
||||
| R5-006 | low | WorkspaceBriefing.tsx + 60 files | light-mode | 324 hardcoded Tailwind palette colors never respond to light theme; heading contrast borderline | Theme-aware tokens; convert load-bearing text first | unverified |
|
||||
| R7-005 | low | lib.rs:83 | shortcut | register(shortcut) propagates error in setup() so Ctrl+Shift+W collision crashes on launch | Log + continue on Err | unverified |
|
||||
| R7-006 | low | tauri.conf.json:4 / Cargo.toml:3 | version | Drift: tauri.conf 0.2.0 vs Cargo 0.1.0 | Sync Cargo.toml | unverified |
|
||||
| R7-007 | low | tauri.conf.json:41 | CSP | img-src self data https so any-HTTPS image exfil channel | Scope img-src to icon CDN + self + data | unverified |
|
||||
| R7-008 | low | tauri.build-override.conf.json:5-9 | signing | macOS ad-hoc sign so Gatekeeper block / updater cannot verify (needs macOS check) | Developer ID + notarization before GA | unverified |
|
||||
| R8-002 | low | tool-launcher.test.ts / tools-routes-launch.test.ts | test-gap | Hook tests assert npx SHAPE but mock execution so binless-stub failure invisible to CI | Static cohort/bin test | unverified |
|
||||
| R8-003 | low | tool-launcher.ts:36-38 | doc-drift | Module doc claims cursor/claude-desktop hooks supported; only claude-code functional | Amend comment | unverified |
|
||||
|
||||
---
|
||||
|
||||
## 4. Themes
|
||||
|
||||
1. T1 - Network exposure & auth boundary (headline risk): R1-001, R2-003, R2-004, R2-006, R6-005.
|
||||
2. T2 - Sidecar input-validation gap at fs boundary: R1-004, R1-005, R6-001, R6-002, R6-006, R6-007, R6-008, R2-005, R2-002.
|
||||
3. T3 - Billing correctness & revenue integrity: R1-002, R1-003, R1-011.
|
||||
4. T4 - Memory/agent core correctness: R3-001..R3-008.
|
||||
5. T5 - Frontend resilience & error feedback: R4-001..R4-010.
|
||||
6. T6 - Desktop packaging & lifecycle: R7-002..R7-008.
|
||||
7. T7 - AI-OS hook cohort mismatch: R8-001, R8-002, R8-003.
|
||||
8. T8 - Backend error-handling robustness: R1-008, R1-009, R1-012.
|
||||
9. T9 - Dependency supply-chain hygiene: R9-001..R9-009.
|
||||
10. T10 - Light-mode finish & a11y polish: R5-001..R5-006.
|
||||
|
||||
---
|
||||
|
||||
## 5. Proposed Remediation Phases
|
||||
|
||||
### Phase 1 - Network exposure & auth boundary (CRITICAL/HIGH)
|
||||
Closes: R1-001, R2-003, R2-006, R2-004, R6-005
|
||||
Cluster: local/index.ts + security-middleware.ts + cors-config.ts. Default-bind 127.0.0.1, remove wsToken from /health, exact-match CORS, Host-header allowlist, gate /api/browse/* + /api/debug/logs to local origin.
|
||||
Verify: tsc -p packages/server; new tests (/health no wsToken, non-local origin rejected, traversal-prefixed origin rejected); manual LAN curl shows no token.
|
||||
|
||||
### Phase 2 - fs-boundary input validation (HIGH/MEDIUM)
|
||||
Closes: R6-001, R6-002, R1-004, R1-005, R2-005, R6-006, R6-007, R6-008
|
||||
Cluster: wire assertSafeSegment / resolve-and-confirm + existing Zod schemas across chat, tasks, ingest, documents, backup restore, workspace-context.
|
||||
Verify: new per-route traversal tests (R6-008) asserting 400 + no out-of-root write; packages/server Vitest green; tsc -p packages/server.
|
||||
|
||||
### Phase 3 - Billing correctness (HIGH)
|
||||
Closes: R1-002, R1-003, R1-011
|
||||
Cluster: packages/server/src/stripe/. Payment-status gate on sync, billingPeriod-aware 4-var price selection, atomic + locked config writes.
|
||||
Verify: sync rejects unpaid (402); checkout selects annual price; NO_PRICE_CONFIGURED only when truly unset; webhook.test.ts green; tsc.
|
||||
|
||||
### Phase 4 - Memory/agent core + AI-OS hooks (HIGH/MEDIUM)
|
||||
Closes: R3-001, R8-001, R8-002, R8-003, R3-002, R3-003, R3-004
|
||||
Verify: harvest cognify skips/real-provider test; static cohort/bin test (R8-002) red to green; packages/agent Vitest; tsc -p packages/agent.
|
||||
|
||||
### Phase 5 - Desktop lifecycle, updater & frontend resilience (HIGH/MEDIUM)
|
||||
Closes: R7-002, R7-003, R7-004, R7-005, R7-006, R4-001, R4-002, R4-003, R4-004, R4-005, R4-006, R4-007, R4-008, R1-008, R1-009, R1-010, R1-012
|
||||
Verify: cargo build (app/src-tauri); manual kill so no orphaned node.exe on 3333, updater hits waggle-os URL; npm run build + Playwright (error boundary catches forced throw, Restore works).
|
||||
|
||||
### Phase 6 - Dependency hygiene, security polish & light-mode/a11y (MEDIUM/LOW)
|
||||
Closes: R9-002, R9-005, R9-001, R9-003, R9-004, R9-006, R9-007, R9-008, R9-009, R2-002, R7-007, R7-008, R5-001, R5-002, R5-003, R5-004, R5-005, R5-006, R3-005, R3-006, R3-007, R3-008, R4-009, R4-010
|
||||
Prefer targeted root overrides for transitive advisories (avoid blanket npm audit fix). Semantic-token swaps for light-mode; a11y modal pattern reuse.
|
||||
Verify: npm audit clears protobufjs/js-cookie/lodash; npm run build:packages + npm run build green after bumps; tsc -p packages/server after drizzle/fastify majors; light-mode spot-check.
|
||||
|
||||
### Cross-cutting gate repair (alongside Phase 1)
|
||||
lint and tauri-tsc gates are non-functional. Add root eslint.config.js (or scope lint to apps/web) and populate/point app/tsconfig.json at real Tauri TS or remove the dead gate, so future phases get real verification signal.
|
||||
75
docs/audits/2026-06-01-full-repo-verification-sweep.md
Normal file
75
docs/audits/2026-06-01-full-repo-verification-sweep.md
Normal file
@@ -0,0 +1,75 @@
|
||||
# Full-Repo Verification Sweep — 2026-06-01
|
||||
|
||||
**HEAD:** `f72cda5` (main, pushed) · **Baseline:** `ebf1bc0` (S4 handoff, last known-green)
|
||||
**Method:** 3 race-safe parallel dimensions (full vitest · repo lint · serialized tsc/build) → per-dimension failure triage. Workflow `full-repo-verification-sweep`, 5 agents, ~8.5 min.
|
||||
|
||||
## Verdict: ✅ GREEN for all session work — **0 session-induced failures**
|
||||
|
||||
The 4 merges since baseline — litellm credential-pool (`..8cdc929`), Wave 2/3 hook ports (`..1219e14`), OQ-6 dedup (`..b1c633b`), OQ-4 hermes compact-on-stop (`..f72cda5`) — introduced **no regressions**. Every failure observed is either missing local infra or a pre-existing install-tree quirk; none touch code changed this cycle. This closes the verification-scope gap that let the `placeholder-audit` regression slip last session (the full repo suite + all tsconfigs + repo lint were checked, not just affected packages).
|
||||
|
||||
| Dimension | Raw result | Triage verdict | Session-induced |
|
||||
|---|---|---|---|
|
||||
| **lint** | `npm run lint` exit 0, **0 errors / 0 warnings** repo-wide | CLEAN | 0 |
|
||||
| **unit-tests** | **7034/7035 non-skipped pass** (506/526 files); 19 files + 1 test fail | INFRA | 0 |
|
||||
| **typecheck-build** | `build:packages` OK · `build` OK · **29/30 tsconfigs clean** | PRE_EXISTING | 0 |
|
||||
|
||||
## Dimension detail
|
||||
|
||||
### lint — CLEAN
|
||||
`eslint .` exits 0 with 0 errors and 0 warnings. Confirms the S4 lint-debt burndown (`no-explicit-any` → 0 repo-wide, all rules ratcheted to ERROR) is holding.
|
||||
|
||||
### unit-tests — INFRA (no code fix)
|
||||
The pure-unit surface (7034 tests, 506 files) is fully green and exercises the session code. The 19 failed files + 1 failed test + 2 unhandled rejections are **all** deterministic missing-infra failures — this verifier has no PostgreSQL (host `5434`) and no Redis (host `6381`), the docker-compose services that every server/worker/integration suite hard-requires. Re-confirmed via `Get-NetTCPConnection` (nothing on 5434/6379/6381) and isolated re-run (identical `ECONNREFUSED:6381` — **not flaky**).
|
||||
|
||||
Root cause for the server suites: `buildServer()` registers `redisPlugin` (eager `new Redis()` ×2) before `wsGateway`; with Redis absent the register-chain stalls on ioredis retries and Fastify's plugin-load timeout fires, mis-reporting `wsGateway` as "did not start" in `server.test.ts`. All other server/worker failures are `beforeAll(buildServer)` timeouts or direct Postgres/Redis `ECONNREFUSED`.
|
||||
|
||||
Affected (all infra, all `touched=0` or type-only edits in range): `packages/server/tests/{server,auth,audit,cron,proactive,db/schema}`, `…/daemons/{hive-mind,scout,subconscious}`, `…/routes/{agents,analytics,resources,tasks,teams,knowledge,messages}`, `…/ws/gateway` (integration block), `packages/worker/tests/job-processor` (worker package untouched all session).
|
||||
|
||||
**To make this dimension pass:** `docker-compose up -d postgres redis` (+ drizzle migrate) before the sweep, OR scope the no-Docker green-gate to the pure-unit surface and exclude the documented infra-dependent suites. This matches how the landing commits were verified (server suite 1745 pass / 1 skip *with* infra running).
|
||||
|
||||
### typecheck-build — PRE_EXISTING (no code fix)
|
||||
`npm run build:packages` (tsc --build shared→core→agent→server) and `npm run build` (apps/web) both pass; `dist/` emits clean. 29 of 30 tsconfigs typecheck clean.
|
||||
|
||||
The lone failure: `tsc --noEmit -p apps/web/tsconfig.node.json` → `vite.config.ts(7,29) TS2769` (defineConfig overload mismatch). Root cause is a **dual-vite install** — root `vite@8.0.14` (hoisted from tailwindcss/vite + plugin-react + vitest) vs `apps/web` `vite@5.4.21` — producing two incompatible vite type trees. Provenance proves pre-existing: the failing line dates to the 2025-01-01 Lovable scaffold (`1bc8809`); `vite.config.ts`, `tsconfig.node.json`, and `apps/web/package.json` are **byte-identical baseline→HEAD**; no vite version changed in range; deterministic (not flaky). **The real build is unaffected** — `npm run build` uses `tsconfig.app.json` and runs `vite.config.ts` via esbuild, never tsc. `tsconfig.node.json` is an extra exhaustive check the sweep ran; it is not in CLAUDE.md's verification commands nor any CI/build path.
|
||||
|
||||
## Informational (surfaced during triage — not failures)
|
||||
|
||||
1. **Clerk auth fix is sound** (`83edcf5`, this cycle): `(clerk as any).verifyToken(token)` → standalone `verifyToken(token, { secretKey })` in `ws/gateway.ts` + `plugins/auth.ts`. This is the production bug fix noted in the S4 handoff — `verifyToken` is a standalone `@clerk/fastify` export, not a `ClerkClient` method (the old cast would `TypeError` at runtime). On the request-time auth path (not plugin-startup), strict-tsc-clean, no test impact (tests swap `_authHandler` / `setWsTokenVerifier`). Confirmed beneficial.
|
||||
|
||||
## Optional future housekeeping (NOT blocking, NOT session work)
|
||||
- **Dedupe vite to one major** across root + `apps/web` so the extra `tsconfig.node.json` check passes (dependency-tree maintenance).
|
||||
- **No-Docker CI gate:** formalize a vitest project/exclude split so the pure-unit surface gates green without Postgres/Redis, and the infra suites run only in a Docker-provisioned lane.
|
||||
|
||||
## Conclusion
|
||||
Main at `f72cda5` is green for everything shipped this cycle. The only red is environmental (no local Docker infra) or pre-existing (dual-vite), with documented evidence and zero attribution to session commits. No code changes required.
|
||||
|
||||
---
|
||||
|
||||
## Addendum — CI health (GitHub Actions `ci.yml`), discovered 2026-06-01
|
||||
|
||||
Investigating the no-Docker test gate surfaced that **CI's `test` job has been RED on every push** (and is a multi-layer breakage, all pre-existing):
|
||||
|
||||
- **L1 — `npm install` → `EBADPLATFORM`** *(FIXED — PR #5, branch `fix/ci-cross-platform-install-and-unit-gate`)*. Root `package.json` pinned 4 Windows-only native binaries (`@rolldown/binding-win32-x64-msvc`, `@swc/core-win32-x64-msvc`, `lightningcss-win32-x64-msvc`, `sqlite-vec-windows-x64`) as **hard** deps, so `npm install` failed on Linux/macOS. Fix: moved them to `optionalDependencies` (npm skips os-mismatched optional deps). Verified on CI: install + tsc + lint + app-tsc now PASS on Linux.
|
||||
- **L2 — bare `npx tsc --noEmit`**: FINE (root `tsconfig.json` is a near-noop; exit 0).
|
||||
- **L3 — `npm run lint`**: FINE (0/0).
|
||||
- **L4 — `npm test` (full vitest, no Docker)** *(unit-gate split landed in PR #5)*: default `npm test` now excludes the 19 Postgres/Redis suites so the gate runs without Docker.
|
||||
|
||||
**Remaining CI-debt (pre-existing, NOT caused by PR #5 — revealed because L1 let tests run for the first time in a while). Full CI-green needs all three:**
|
||||
|
||||
1. **Workspace packages not built before tests** (~23 failures): `ci.yml` runs `npm test` with no prior build, so `@waggle/{shared,hive-mind-core,hive-mind-shim-core,hive-mind-hooks-core,hive-mind-hooks-codex}` fail to resolve their entry (no `dist/`). Locally these resolve only because `dist/` exists from prior builds. Fix options: add a build step in CI, OR add vitest `resolve.alias` → `src` for these packages (mirrors the existing `@waggle/marketplace` alias). Note `build:packages` alone is insufficient — it only builds shared/core/agent/server, not the `hive-mind-*` packages.
|
||||
2. **Uncommitted seed DB** (~80 failures): the marketplace sync suites `copyfile` `packages/marketplace/marketplace.db`, a gitignored/uncommitted file absent on a fresh CI checkout. Fix: generate the seed in test setup, commit a fixture, or gate these tests.
|
||||
3. **Tests asserting on local working-tree state** (~4): assertions that `.planning/` exists at repo root and a hive-950 hex allow-list — both depend on gitignored/local-only state absent on CI. Fix: make these robust to a clean checkout, or scope them out of CI.
|
||||
|
||||
### Resolution (PR #5, `fix/ci-cross-platform-install-and-unit-gate`)
|
||||
|
||||
The `test` job is now **GREEN on CI Linux** (workflow conclusion `success`). The 228 pre-existing failures were resolved in layers, all verified on CI:
|
||||
|
||||
1. **Install** — 4 Windows-only natives → `optionalDependencies`.
|
||||
2. **Workspace resolution** (~23 + cascades) — `vitest.aliases.ts` maps every `@waggle/*` (with a `src/index.ts`) to its `src/` dir in both vitest configs (subpath-safe).
|
||||
3. **Seed/env fixtures** — excluded `sync-verification` (gitignored 13MB `marketplace.db`); skip the `marketplace.db exists` assertions when absent; fixed the hive-950 backslash allow-list; `.planning` guard tolerates clean checkout.
|
||||
4. **The final 9** (parallel root-cause) — **2 product bugs** (`backup.ts` excludes the `models/` ONNX cache from backups; `trust-wiring` reads audit via the writer connection, not a fresh WAL reader) + determinism (`EMBEDDING_PROVIDER=mock` test pin; dead-port Ollama; benchmark `emitPreregistrationEvent:false`; codex `skipIf(!BIN_BUILT)`).
|
||||
|
||||
**Residual (non-blocking, pre-existing, follow-up):**
|
||||
- **`e2e` job "Build frontend"** — `apps/web`'s real tsc build can't resolve dist-exporting `@waggle/*` deps (`@waggle/hive-mind-core`, …) because the e2e job builds only `build:packages` (shared/core/agent/server), not the `hive-mind-*` packages. `e2e` is `continue-on-error` so it does NOT block CI. Fix options: build all imported `@waggle` packages, or add `@waggle/*`→`src` `paths` to `apps/web/tsconfig.app.json` (mirror the vitest aliases) — but validate against the deploy's `build:all` first.
|
||||
- **Docker infra test lane** (the 19 Postgres/Redis suites) — still local-only via `npm run test:infra`; a Docker-services CI job needs a verified migrate step.
|
||||
- **dual-vite `tsconfig.node.json`** — pre-existing, not in any build path.
|
||||
88
docs/audits/2026-06-01-memory-overclaim-investigation.md
Normal file
88
docs/audits/2026-06-01-memory-overclaim-investigation.md
Normal file
@@ -0,0 +1,88 @@
|
||||
# Memory Over-Claim Investigation — 2026-06-01
|
||||
|
||||
**Trigger:** The 5-persona human E2E found the agent, on a *fresh* session, claimed
|
||||
*"I have this from our last session / you're back in context"* and asserted specifics
|
||||
the persona never stated (Ivan, LoCoMo, 4-month runway, OpenClaw, "227 entities").
|
||||
Chen (the careful skeptic) scored trust 1/10 over it. Question: **workspace-memory
|
||||
framed as session-history, or true confabulation?**
|
||||
|
||||
**Verdict: BOTH — and neither is cross-user data bleed.** The personas ran inside
|
||||
Marko's own populated `Default / Researcher` workspace, so all recalled data is
|
||||
legitimately Marko's. The problems are (1) a prompt instruction that frames
|
||||
workspace memory as *this speaker's* prior conversation, and (2) the LLM
|
||||
embellishing real recall with invented specifics that the prompt never forbids.
|
||||
|
||||
## Evidence (live workspace on :3333, the exact memory the agent used)
|
||||
|
||||
Dumped all **11 frames** + the **179-entity** knowledge graph and tested every
|
||||
disputed claim for presence in real memory:
|
||||
|
||||
| Claim the agent made | In real memory? | |
|
||||
|---|---|---|
|
||||
| Ivan (owns GPU/H200) | **PRESENT** (frame 6 + entity "Ask Ivan") | real recall |
|
||||
| Mihail (owns architecture) | **PRESENT** (frame text) | real recall |
|
||||
| LoCoMo / Mem0 | **PRESENT** (frame text) | real recall |
|
||||
| H200 / GPU | **PRESENT** | real recall |
|
||||
| Egzakta, Hermes | entities present | real recall |
|
||||
| **"4 months runway"** | **ABSENT** from all frames | **confabulated** |
|
||||
| **"227 entities tracked"** | real count is **179** | **confabulated number** |
|
||||
| **"OpenClaw + Hermes competitive analysis"** | OpenClaw **ABSENT** in frames | **confabulated** |
|
||||
| **"our last session" / "you're back in context"** | no *this-speaker* session; prior sessions exist but are the owner's | **framing over-claim** |
|
||||
|
||||
So the recall substrate **works** (it retrieved Marko's real frames). The trust
|
||||
damage comes from framing + embellishment, not from a broken retriever and not
|
||||
from one user's memory leaking into another's.
|
||||
|
||||
## Root cause (code)
|
||||
|
||||
`packages/agent/src/orchestrator.ts` → `recallMemory()`, lines ~529-533, injected
|
||||
into the system prompt every turn:
|
||||
|
||||
```
|
||||
# Recalled Memories
|
||||
These memories were automatically retrieved for the user's current message.
|
||||
IMPORTANT: Use these to ground your response. Cite them naturally:
|
||||
"From our previous discussion...", "You mentioned that...", "Based on your workspace context..."
|
||||
Do NOT ignore relevant memories. Do NOT present memory content as your own reasoning — attribute it.
|
||||
```
|
||||
|
||||
Two defects:
|
||||
1. **Framing:** it instructs the model to cite *workspace* memory as *"From our
|
||||
previous discussion…" / "You mentioned that…"* — asserting a shared history
|
||||
with the current speaker that may not exist (first contact, or the memory is
|
||||
the workspace owner's, not this speaker's). This directly seeds
|
||||
"welcome back / our last session."
|
||||
2. **No anti-confabulation guard:** it says "attribute it" but never "state ONLY
|
||||
what the memories say; don't invent specifics not present." So the model fills
|
||||
gaps with plausible numbers/names (runway, 227, OpenClaw) and presents them as
|
||||
recall.
|
||||
|
||||
## Proposed fix (surgical — same block)
|
||||
|
||||
```
|
||||
# Recalled Memories
|
||||
These are facts saved in this WORKSPACE'S memory, retrieved for the user's current
|
||||
message. They may come from earlier sessions, other sessions, or imported sources —
|
||||
NOT necessarily from this conversation.
|
||||
IMPORTANT — ground your response in them, but attribute provenance HONESTLY:
|
||||
- Say "your saved memory shows…" / "from your workspace notes…". Do NOT say
|
||||
"from our previous discussion" or "you just mentioned" unless it was actually
|
||||
said earlier in THIS conversation.
|
||||
- On the user's first message, do NOT claim continuity ("welcome back",
|
||||
"as we discussed", "you're back in context") — you have no prior turn yet.
|
||||
- State ONLY what the memories below actually say. Do NOT invent specifics
|
||||
(numbers, names, dates, competitors) that are not present — if unsure, ask
|
||||
rather than assert.
|
||||
- Do NOT present memory content as your own reasoning — attribute it.
|
||||
```
|
||||
|
||||
Expected effect: flips Chen (the fabricated-history failure), de-risks Maya/Sam/Leo
|
||||
(unverifiable specifics), and keeps the genuine recall that bonded them. Pairs with
|
||||
the report's fix #1 (auditable memory) and #3 (demote the "Recalled N / Auto-saved N"
|
||||
chrome).
|
||||
|
||||
## Not a data-bleed (scope note)
|
||||
|
||||
Single-tenant workspace; all data is the owner's. The cross-*user* bleed risk only
|
||||
arises in shared/team workspaces and was NOT exercised here — flag for a separate
|
||||
multi-tenant test, but it is not what this run found.
|
||||
117
docs/audits/2026-06-01-production-readiness-assessment.md
Normal file
117
docs/audits/2026-06-01-production-readiness-assessment.md
Normal file
@@ -0,0 +1,117 @@
|
||||
# Waggle OS — Production-Readiness Assessment
|
||||
|
||||
**Date:** 2026-06-01
|
||||
**Commit:** `839d4ce` (main, tree clean except this report + the vision-E2E design doc)
|
||||
**Method:** 5 parallel auditors (build/tsc, CI/deploy, open-work residuals, test-infra/local-run, vision-E2E design) + independent re-verification of every load-bearing claim against the live repo and the GitHub Actions API.
|
||||
|
||||
---
|
||||
|
||||
## 1. Bottom Line
|
||||
|
||||
**Waggle OS is NOT production-ready for the desktop-binary / containerized-deploy path.** A single dependency-ordering gap — `build:packages` never builds `@waggle/hive-mind-core` before the packages that hard-depend on it — red-lines the CI e2e job, BOTH Tauri verify jobs (Windows + macOS), and every deploy artifact, while the green CI checkmark on main hides it (the unit-test gate passes only because vitest aliases `@waggle/*` to `src/`). The unit-test suite, tsc gates, lint, and the local web build are genuinely green, and the open-work residuals (§10 #1/#2/#3, OQ-4, OQ-5) are code-complete and test-green — but **the release/deploy plumbing has 6 hard blockers** that must be fixed before any binary or server ship. The gates that remain are: fix the package-build order, make the Dockerfile/render.yaml buildable + add a DB-migration step, then runtime-verify on a real binary.
|
||||
|
||||
---
|
||||
|
||||
## 2. Production Blockers (must-fix-before-launch)
|
||||
|
||||
> Each blocker re-verified independently. The first is the root cause of four downstream failures.
|
||||
|
||||
### B1 — `build:packages` omits `@waggle/hive-mind-core` → breaks CI e2e + both Tauri verifies + release + deploy *(ROOT CAUSE)*
|
||||
- **Owner type:** code
|
||||
- **Evidence:** `package.json` `build:packages = shared→core→agent→server`. `packages/core/package.json` declares `"@waggle/hive-mind-core": "*"`. `packages/hive-mind-core/package.json` exports ONLY `dist/index.js` + `dist/index.d.ts` (no `src` export), its `dist/` is **gitignored** (`.gitignore:11:dist`) and **NOT tracked** (`git ls-files packages/hive-mind-core/dist/` → empty), and it is **never built** by `build:packages`. On a fresh checkout its `dist/` is absent → `tsc --build` of `core` emits `TS2307: Cannot find module '@waggle/hive-mind-core'` (16 errors). **Confirmed live:** Tauri `verify-macos` run `26762292125` on the current HEAD `839d4ce` failed at the `Build packages (shared → core → agent → server)` step with exactly these TS2307 errors (`src/config.ts(4,69)`, `src/compliance/*`, `src/index.ts(81,8)`, etc.). The CI `e2e` job fails identically at its `Build packages` step. *(This is why the BUILD/TSC auditor saw "GREEN locally" — its machine had a stale pre-built `dist/`; on fresh checkout it is RED, which the CI logs prove.)*
|
||||
- **Why CI looks green anyway:** the unit `test` job passes only because `vitest.aliases.ts` remaps `@waggle/*` → `src/`, sidestepping the missing dist. The `e2e` job is `continue-on-error: true` (`ci.yml:46`), so the workflow reports `success` even though e2e never runs.
|
||||
- **Fix (verified):** prepend `cd packages/hive-mind-core && npx tsc --build &&` to the `build:packages` script. Re-verified the full corrected chain (`hive-mind-core → shared → core → agent → server`) exits 0 from a clean `dist`. This one change un-blocks e2e, both Tauri verifies, `release.yml`, and any deploy that runs `build:all`.
|
||||
|
||||
### B2 — Tauri `verify-windows` + `verify-macos` both RED (desktop release path broken)
|
||||
- **Owner type:** code (resolved by B1)
|
||||
- **Evidence:** `gh run list` (Tauri Build Verification / main / `839d4ce`): `verify-windows=failure`, `verify-macos=failure`. Both die at the `Build packages` step with the TS2307 above — **NOT** at Rust compile / signing / native-deps (the workflow header comment's diagnosis is wrong; it never reaches Rust). `release.yml` (tag-triggered) shares the same `sidecar→core→hive-mind-core` dependency and will fail the same way on a real `v*` tag.
|
||||
- **Fix:** B1's fix. After it lands, re-run the Tauri verify workflow to confirm it now reaches (and passes) the Rust/Vite/sidecar stages.
|
||||
|
||||
### B3 — Dockerfile is not buildable (three independent breakages)
|
||||
- **Owner type:** code
|
||||
- **Evidence:** (1) `Dockerfile:20,57,84` `COPY packages/ui/package.json packages/ui/` — but `packages/ui/package.json` **does not exist** (CLAUDE.md §2: `ui` is not a workspace; confirmed `ls` → no such file) → COPY of a literal missing file fails the build. (2) Root `npm run build` = `cd apps/web && tsc && vite build`, but the Dockerfile only copies `app/` (`:22,:29`), **never `apps/`** → `RUN npm run build` fails (`cd apps/web` not found). (3) No `hive-mind-*` package source/manifest is copied, yet `CMD npx tsx packages/server/src/index.ts` resolves `@waggle/core → @waggle/hive-mind-core` at runtime.
|
||||
- **Fix:** remove the `packages/ui` COPY lines; copy `apps/` (not just `app/`); copy the `hive-mind-*` packages needed for runtime resolution; build packages (with B1's fix) before `npm run build`.
|
||||
|
||||
### B4 — `render.yaml` builds the wrong directory and serves an empty/stale shell
|
||||
- **Owner type:** code
|
||||
- **Evidence:** `render.yaml:16` `buildCommand: npm install && cd app && npm run build` runs `vite build` in `app/`, but **`app/` has no `src/`** (confirmed `ls app/src` → no such file) and `app/index.html` references `/src/main.tsx`. The real UI is `apps/web`. `app/dist` is gitignored and the committed copy is a stale Apr-3 brand shell. `WAGGLE_FRONTEND_DIR=./app/dist` (`render.yaml:28`) → server serves nothing usable. The build also never runs `build:packages`, so the runtime entrypoint hits the same hive-mind-core gap.
|
||||
- **Fix:** point the build at the repo root `npm run build:all` (which builds packages + `apps/web` → root `/dist`) and set `WAGGLE_FRONTEND_DIR=./dist`.
|
||||
|
||||
### B5 — No DB-migration step in any deploy artifact
|
||||
- **Owner type:** code
|
||||
- **Evidence:** `packages/server/src/db/migrate.ts` runs drizzle migrations from `./drizzle` (migrations present: `0000_wild_glorian.sql`, `0001_redundant_sauron.sql`). Grep of `Dockerfile`, `render.yaml`, `docker-compose.production.yml` for `migrat|seed` → only a code-comment match; no `startCommand`/`CMD`/entrypoint runs `migrate`. A fresh Postgres (render-provisioned or compose) starts with no schema → team-server queries fail at runtime.
|
||||
- **Fix:** add a migrate step to the container entrypoint / render `startCommand` (e.g. `tsx packages/server/src/db/migrate.ts && <server start>`).
|
||||
|
||||
### B6 — `render.yaml` provisions Postgres+Redis but runs the local SQLite sidecar entrypoint (infra mismatch + CORS fail-closed gap)
|
||||
- **Owner type:** decision (which deployment target?) then code
|
||||
- **Evidence:** `render.yaml:17` `startCommand: npx tsx packages/server/src/local/start.ts --skip-litellm` → the **desktop/SQLite single-user sidecar**, not the team Postgres server (`packages/server/src/index.ts`, what the Dockerfile `CMD` runs). render injects/provisions managed Postgres+Redis (`render.yaml:32-40`) that the chosen entrypoint largely bypasses; team features (Clerk-gated, Postgres-backed) are not actually served. Separately, `config.ts:17-35` throws `'CORS_ORIGIN ... required in production'` when `NODE_ENV=production` (set in `render.yaml:21`) and unset — and **render.yaml defines no `CORS_ORIGIN`** (docker-compose.production.yml correctly enforces it at `:47`), so the team-server path would crash on boot.
|
||||
- **Fix:** decide the render target. If it is the team server, switch `startCommand` to the Postgres entrypoint, add `CORS_ORIGIN`, and wire migrate (B5). If render is meant to host the local sidecar demo, drop the managed Postgres/Redis to stop paying for bypassed infra.
|
||||
|
||||
---
|
||||
|
||||
## 3. E2E Prerequisite Status
|
||||
|
||||
**The E2E vision harness depends on `npm run build` (apps/web → `/dist` on :3333) succeeding so the Playwright `webServer` can boot.**
|
||||
|
||||
- **Local status: GREEN.** `npm run build` (`cd apps/web && tsc --noEmit -p tsconfig.app.json && vite build --outDir ../../dist --emptyOutDir`) exits 0; `dist/index.html` is freshly written (Jun 1 17:00). `apps/web` imports only `@waggle/shared` (grep: 4 hits, zero `@waggle/hive-mind-core` — the only hive-mind references are comments in `LauncherApp.tsx`), and `@waggle/shared/dist` exists, so the apps/web build itself is not blocked by B1.
|
||||
- **CI status: the e2e job's `npm run build` is currently UNREACHABLE** because the step before it — `npm run build:packages` (`ci.yml:66`) — fails at B1 (TS2307). So in CI today, the frontend never builds and Playwright never runs (the job is `continue-on-error`, so this is silently masked).
|
||||
- **Net:** the E2E prerequisite is **green on a machine with a pre-built `hive-mind-core/dist`, but red on a clean checkout / in CI** until B1 is fixed. The handoff's "e2e frontend build blocked" note is real for CI; it just localizes to `build:packages` (B1), not to `apps/web` tsc.
|
||||
- **Exact fix:** apply **B1** (build `hive-mind-core` first in `build:packages`). After that, the e2e job reaches `npm run build` (already green) and Playwright can boot the :3333 server. No change to `apps/web` tsconfig is needed.
|
||||
|
||||
---
|
||||
|
||||
## 4. Non-Blocking Residuals
|
||||
|
||||
**Open-work items (all code-complete + test-green; remaining work is platform/binary-blocked or doc-only):**
|
||||
- **OQ-4 hermes compact-on-stop — DONE.** `compact-on-stop.ts` (opt-in `WAGGLE_HERMES_COMPACT_ON_STOP`, time-gated, save-first + fail-open); 26/26 tests, package tsc exit 0.
|
||||
- **§10 #3 Wave 2/3 hooks — DONE, but CLAUDE.md prose is STALE.** codex/cursor/hermes/openclaw (+codex-desktop re-export) are real implementations with full adapter/install/uninstall/verify trees; 203/203 tests. Only `claude-desktop` remains `export {}` — a deliberate MCP-only deferral (pinned by `tests/placeholder-audit.test.ts` EXPECTED_MARKER_COUNT=1). **Doc fix (non-code):** CLAUDE.md §10 #3 (line 516) still lists all 6 packages as stubs — update to reflect only claude-desktop remains.
|
||||
- **§10 #1 Spawn-Agent P36 + P35 model fallback — DONE in code** (`Dock.tsx`/`Desktop.tsx` wiring; `SpawnAgentDialog.tsx` 3-tier LiteLLM→runtime→provider-catalog fallback, commit `14942be`). Residual: runtime verify on a clean Tauri install (**platform-blocked**).
|
||||
- **§10 #2 light-mode finish — DONE structurally** (semantic-token migration complete; the 5 `hive-950` hits are legit token defs/usages, no literal-color rot). Residual: BootScreen + header visual polish needs a binary to eyeball (**validation-blocked**).
|
||||
- **OQ-5 OpenClaw live-install — genuinely platform-blocked.** Code path implemented + tested in tmp dirs (15 tests incl. fail-open); needs a real OpenClaw gateway to verify installed-handler dep resolution. Does not block a claude-code-first Waggle launch.
|
||||
|
||||
**CI / test-infra follow-ups (do not block launch, but should be tracked):**
|
||||
- **CI e2e job is `continue-on-error: true`** — it cannot fail the pipeline. After B1, consider flipping it to blocking so a broken frontend build surfaces.
|
||||
- **No Docker-infra CI lane.** Zero workflows declare `services: postgres` or run `test:infra`; Postgres/Redis/MinIO/S3 code paths are unverified by CI. `docker-compose.yml` provides the infra locally but CI never spins it up.
|
||||
- **Dead/misleading test config:** `apps/web/playwright.config.ts` imports the uninstalled `lovable-agent-playwright-config` (would throw on load; apps/web has no specs) — delete it. `playwright-e2e.config.ts` (the `test:e2e` lane) has **no `webServer`** — it silently times out unless a server is pre-started on :3333; document or add a webServer block.
|
||||
- **Committed test cruft:** `tests/visual/r2-uat-mega.spec.ts:3` has a dead hardcoded 64-hex token (rotate if it was ever real); `tests/login-flow.spec.ts` targets the wrong port (:8083) with a stale Clerk flow — fix or delete.
|
||||
- **Build polish (cosmetic):** vite warns on `@import` order + a 1.74 MB JS chunk (>500 kB advisory). Non-fatal.
|
||||
|
||||
**Benchmark arcs (out of launch scope):** C-3 full GAIA-2 Phase 4 (needs Docker + ARE + new adapter strategy; budget recalibrate pending). LoCoMo v5 trio-strict re-judge (~$30, ~2h) is the only remaining step on C-1.
|
||||
|
||||
---
|
||||
|
||||
## 5. The Vision-E2E Harness Plan
|
||||
|
||||
**Design doc:** `docs/audits/2026-06-01-vision-e2e-harness-design.md`
|
||||
|
||||
**Recommended architecture — Option C (Hybrid).** Playwright deterministically drives and captures every surface (×dark/light) plus the 7 flows, emitting a PNG + sidecar JSON per capture **enriched with objective signals** (console errors via `page.on('console')`, failed network requests, and a Lighthouse contrast/a11y audit on heavy views). A multi-agent Workflow fans out one vision-judge subagent per capture to grade *meaning* against a 5-dimension rubric (`renders_correctly`, `no_error_state`, `flow_completes`, `theme_legible`, plus the objective `no_console_errors`). A reducer cross-checks vision vs objective signals — **a vision-PASS carrying a real console error or a Lighthouse fail is downgraded to FAIL** — and writes one report. This buys A's deterministic, replayable navigation plus a deterministic objective floor so a plausible-looking-but-broken screenshot can't fool the gate (defense in depth). Build on the existing `tests/visual` + `tests/e2e` helpers and the root `playwright.config.ts` `webServer` block (:3333, `reuseExistingServer`, `WAGGLE_TRUST_LOCALHOST=1`) — not greenfield. (Rejected: Option A lacks the objective floor; Option B's live agentic drive is non-deterministic → a flaky CI gate.)
|
||||
|
||||
**Scope.** ~19 surfaces (7 core views: chat/memory/events/capabilities/cockpit/mission-control/settings; plus room/agents/files/approvals/vault/connectors/marketplace/timeline/backup/telemetry/governance/dashboard; plus overlays: onboarding, Ctrl+K search, spawn-agent, persona switcher, shortcuts help, upgrade modal) × **dark + light** themes, plus **7 end-state-graded flows** (onboarding, chat round-trip, memory browse, spawn agent, persona switch, marketplace, settings tabs). Total ≈ **52 vision judgments/run**. Deterministic entry via `/?skipOnboarding=true&tier=power`; light theme via `data-theme='light'` on `<html>` (the old views.spec.ts dark/light-class toggle is stale and must not be the model). FAIL on any vision dimension at confidence ≥0.7 or any hard signal; WARN at 0.4–0.7 (routes to human, never auto-blocks CI).
|
||||
|
||||
**Coverage gap this fills:** today exactly ONE spec (`tests/visual/views.spec.ts`) does true pixel-diff (drift-only, brittle), three specs capture screenshots but assert nothing about their content, and **zero** tests semantically judge "does it actually look and work right." A visually-broken-but-DOM-present screen passes the current suite. The vision harness is net-new.
|
||||
|
||||
**The one key decision (needs the user's call):** **Does the Chat round-trip flow grade against a REAL LLM reply or a gracefully-handled degraded state?** Verified ground truth (`service.ts:217-258`): under the harness's own `--skip-litellm` server with no Anthropic key, `/api/chat` resolves the provider to `health:'degraded'` and returns NO assistant message.
|
||||
- **Path 1 (degraded, CI default):** "completes" = user message renders + send works + missing-LLM state handled gracefully (clear "configure API key" prompt, not a blank window/stack trace). Deterministic, free, CI-safe — but does not verify a real answer.
|
||||
- **Path 2 (real LLM, opt-in `--live-llm`):** inject a real key so chat returns an actual reply and vision grades a coherent assistant message. Highest fidelity, but non-deterministic, costs money, and the CI gate must hold a secret.
|
||||
- **Recommended:** Path 1 as the CI gate, Path 2 as an opt-in pre-release lane. (Secondary, can default: run target = local Chromium against built `apps/web` on :3333.)
|
||||
|
||||
---
|
||||
|
||||
## 6. Recommended Sequence
|
||||
|
||||
1. **Fix B1 (the root cause).** Prepend `cd packages/hive-mind-core && npx tsc --build &&` to `build:packages`. Verified: the full corrected chain exits 0 from a clean dist. This un-blocks CI e2e, both Tauri verifies, `release.yml`, and `build:all`. *(code — ~5 min)*
|
||||
2. **Re-run CI + Tauri verify on the B1 commit.** Confirm e2e's `build:packages`→`npm run build` now reaches Playwright, and that both Tauri verifies now progress past `Build packages` into the Rust/Vite/sidecar stages (and pass, or surface the *real* next failure). *(verification)*
|
||||
3. **Flip the CI e2e job to blocking** (drop `continue-on-error`) once it's green, so a broken frontend build can never again hide behind a green checkmark. *(decision + code)*
|
||||
4. **Fix the deploy artifacts (B3–B6) for whichever target ships first:**
|
||||
- Dockerfile: drop `packages/ui` COPYs, copy `apps/` + `hive-mind-*`, build packages before `npm run build`.
|
||||
- render.yaml: build via root `build:all`, set `WAGGLE_FRONTEND_DIR=./dist`, add `CORS_ORIGIN`, decide local-sidecar vs team-Postgres entrypoint.
|
||||
- Add the drizzle `migrate.ts` step to the chosen entrypoint.
|
||||
*(code + one decision)*
|
||||
5. **Decide the vision-harness chat-flow path** (Path 1 CI gate + Path 2 opt-in lane — §5). *(decision — blocks the harness build)*
|
||||
6. **Build the Option-C vision harness** on the existing :3333 webServer + tests/e2e helpers; extract the copy-pasted nav helpers (`gotoDesktop`/`skipOnboarding`/`dismissOverlay`/`openAppViaDock`) into `tests/e2e/_helpers.ts`; delete the dead `apps/web/playwright.config.ts` and fix/remove `login-flow.spec.ts` + the dead token in `r2-uat-mega.spec.ts`. *(code — ~3 sessions)*
|
||||
7. **Run the vision harness against the local web build**, triage WARN/FAIL, then close the binary-blocked residuals (§10 #1 spawn-agent clean-install, §10 #2 light-mode polish) on a real Tauri build. *(verification — platform-blocked steps last)*
|
||||
8. **Doc cleanup:** update CLAUDE.md §10 #3 to reflect only `claude-desktop` remains a (deliberate) stub. *(doc)*
|
||||
|
||||
---
|
||||
|
||||
*Synthesized 2026-06-01 from 5 parallel auditors; every red claim independently re-verified against the live repo (`839d4ce`) and the GitHub Actions API.*
|
||||
145
docs/audits/2026-06-01-vision-e2e-harness-design.md
Normal file
145
docs/audits/2026-06-01-vision-e2e-harness-design.md
Normal file
@@ -0,0 +1,145 @@
|
||||
# Vision-Based E2E Harness — Design
|
||||
|
||||
**Date:** 2026-06-01
|
||||
**Status:** DESIGN (read-only analysis; no harness code written yet)
|
||||
**Author:** audit subagent
|
||||
**Goal:** ONE comprehensive harness that "fully verifies the platform" using **vision** — a model judging screenshots for *meaning* (not pixel diffs) — built and run via **multi-agent workflows**.
|
||||
|
||||
> This is a design document. It proposes architecture options, picks a recommendation, names the single decision the user must resolve before build, and defines exact scope. It does **not** add test code.
|
||||
|
||||
---
|
||||
|
||||
## 1. What already exists (verified against the live repo)
|
||||
|
||||
Concrete, so the harness extends reality instead of a remembered shape:
|
||||
|
||||
| Asset | Location | What it gives us |
|
||||
|---|---|---|
|
||||
| Visual-regression spec | `tests/visual/views.spec.ts` | 7 views × {dark,light} = 14 **pixel-diff** screenshots; `maxDiffPixelRatio: 0.003` |
|
||||
| Visual baselines | `tests/visual/baselines/…` (28 dirs present) | Existing PNG baselines for both themes |
|
||||
| Full product audit | `tests/e2e/full-product-audit.spec.ts` | API-health checks + **dock-open helper** (`openAppViaDock`, handles `Ops`/`Extend` zone trays via `[data-dock-tray]`), console-error capture, per-app text assertions |
|
||||
| User-journey spec | `tests/e2e/user-journeys.spec.ts` | 12 journeys: nav, sidebar collapse, Ctrl+K palette, theme toggle, chat input, settings tabs, cockpit cards |
|
||||
| Playwright config | `playwright.config.ts` | `webServer` builds `apps/web` then spawns `npx tsx packages/server/src/local/start.ts --skip-litellm` on `:3333` with `WAGGLE_TRUST_LOCALHOST=1`; `reuseExistingServer: true` |
|
||||
| Multi-agent primitives | `packages/agent/src/{workflow-harness,workflow-composer,subagent-orchestrator}.ts` | In-product workflow/subagent fan-out (`createHarnessRun`, `advancePhase`, `harnessEvents`) |
|
||||
| Live MCP browsers | `mcp__plugin_playwright_playwright__*`, `mcp__chrome-devtools__*` | Turn-by-turn drive + `take_screenshot` / `take_snapshot` / `list_console_messages` / `lighthouse_audit` |
|
||||
|
||||
**Gap:** every existing check is either a **pixel diff** (brittle; flags antialiasing, not meaning) or a **substring assertion** (`text.toMatch(/persona|message/i)` — passes on a half-broken screen as long as one word renders). **Nothing judges whether a surface is actually correct, legible, and non-broken the way a human reviewer would.** That is the hole this harness fills.
|
||||
|
||||
### 1.1 Ground-truth facts that constrain the design (verified, correcting stale assumptions)
|
||||
|
||||
- **The real UI is a desktop-OS metaphor**, not a sidebar app. `Desktop.tsx` renders a `Dock` (zones `Ops`/`Extend` open `[data-dock-tray]` portals) + draggable `AppWindow`s. The 7 "views" map to dock apps (`ChatApp`, `MemoryApp`, `EventsApp`, `CapabilitiesApp`, `CockpitApp`, `MissionControlApp`, `SettingsApp`) plus standalone windows (Room, Agents/Personas, Files, Approvals, Vault, Connectors, Marketplace, Timeline, Backup, Telemetry, Governance).
|
||||
- **Deterministic entry** = `/?skipOnboarding=true&tier=power` — `useOnboarding.ts:32` short-circuits the wizard and sets `tier=power`, unlocking the full dock. `?forceWizard=true` (DEV-only) forces the wizard for onboarding-flow capture.
|
||||
- **Theme contract** = `document.documentElement` attribute `data-theme="light"`; **dark is the absence of the attribute** (`Index.tsx:11`, `useIsLightTheme.ts:14`, `index.css:140`). The `views.spec.ts` helper that toggles a `dark`/`light` *class* is partly stale and should not be the model for the new harness — set/remove `data-theme` instead.
|
||||
- **Chat round-trip under `--skip-litellm` does NOT return a real assistant reply.** `service.ts:217-258`: with no LiteLLM and no Anthropic key, provider resolves to `anthropic-proxy` / **`health: 'degraded'`** / `"no API key — configure in Settings"`. So a chat *send* surfaces an error/degraded state, not a model answer. **This is the central design fork (see §5).**
|
||||
|
||||
---
|
||||
|
||||
## 2. Rubric — what "vision verdict" means
|
||||
|
||||
Each captured surface is graded by a vision model against five dimensions. Output is structured, not prose:
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"surface": "memory:dark",
|
||||
"verdict": "PASS" | "FAIL" | "WARN",
|
||||
"confidence": 0.0-1.0,
|
||||
"dimensions": {
|
||||
"renders_correctly": { "pass": true, "note": "frame list + search bar laid out, no overlap" },
|
||||
"no_error_state": { "pass": true, "note": "no red banner, no 'Something went wrong', no empty stack trace" },
|
||||
"flow_completes": { "pass": true, "note": "expected end-state for this step is visible" },
|
||||
"theme_legible": { "pass": true, "note": "text/background contrast adequate; no dark-on-dark or white-on-white" },
|
||||
"no_console_errors": { "pass": true, "note": "objective signal injected from Playwright/CDP, not vision" }
|
||||
},
|
||||
"evidence_screenshot": "artifacts/memory-dark.png"
|
||||
}
|
||||
```
|
||||
|
||||
Rules:
|
||||
- **`renders_correctly`**, **`no_error_state`**, **`flow_completes`**, **`theme_legible`** are graded by the **vision model** from the screenshot + a per-surface expectation string.
|
||||
- **`no_console_errors`** is **not** a vision judgment — it is an objective signal captured by the driver (`page.on('console')` / `list_console_messages`) and merged into the record, filtered for known-benign noise (favicon, 401/404 on optional endpoints, WebSocket sync) as `full-product-audit.spec.ts:312` already does.
|
||||
- A surface **FAILs** if any vision dimension fails with confidence ≥ 0.7, or any real console error is present. **WARN** for low-confidence (0.4–0.7) vision fails → routes to human spot-check, never auto-blocks CI.
|
||||
- The vision judge is handed **(a)** the screenshot, **(b)** a one-line expectation ("Memory app: a searchable list of memory frames or a clean empty state"), **(c)** the rubric. It must cite *what it sees* per dimension so verdicts are auditable.
|
||||
|
||||
---
|
||||
|
||||
## 3. Scope
|
||||
|
||||
### Surfaces (capture matrix)
|
||||
**7 core views** (dock apps): `chat`, `memory`, `events`, `capabilities`, `cockpit`, `mission-control`, `settings`.
|
||||
**Dock apps / standalone windows**: `room`, `agents` (Personas), `files`, `approvals`, `vault`, `connectors`, `marketplace`, `timeline`, `backup`, `telemetry`, `governance`, `dashboard` (Home).
|
||||
**Overlays**: `onboarding wizard` (via `?forceWizard=true`), `global search` (Ctrl+K), `spawn-agent dialog`, `persona switcher`, `keyboard-shortcuts help`, `upgrade modal`.
|
||||
|
||||
### Flows (multi-step, end-state graded)
|
||||
1. **Onboarding** — wizard step-through to completion (capture each step).
|
||||
2. **Chat round-trip** — open Chat → type → send → observe response (see §5 fork: real reply vs degraded-state-handled-gracefully).
|
||||
3. **Memory browse** — open Memory → search → frame list or empty state renders.
|
||||
4. **Spawn agent** — open Spawn dialog → pick persona → confirm → agent appears in Room/Mission Control.
|
||||
5. **Persona switch** — open PersonaSwitcher → select → header reflects new persona.
|
||||
6. **Marketplace** — open Marketplace → browse packs → (install affordance present).
|
||||
7. **Settings** — open Settings → walk tabs (General/Models/Vault/Permissions/Team/Advanced) → each renders.
|
||||
|
||||
### Themes
|
||||
**dark** (no `data-theme`) and **light** (`data-theme="light"`) for every surface = full matrix ×2.
|
||||
|
||||
### Rubric dimensions (per surface)
|
||||
`renders_correctly` · `no_error_state` · `flow_completes` · `theme_legible` · `no_console_errors` (objective).
|
||||
|
||||
**Matrix size:** ~19 surfaces × 2 themes ≈ 38 static captures + 7 flow end-states × 2 themes ≈ 14 flow captures ≈ **~52 vision judgments per full run.**
|
||||
|
||||
---
|
||||
|
||||
## 4. Architecture Options
|
||||
|
||||
### Option A — Capture-then-judge (Playwright drives, Workflow fans out vision judges)
|
||||
**Mechanism:** A Playwright spec drives the scripted journey (every surface, both themes, the 7 flows), writing a numbered PNG + a sidecar JSON (`{surface, theme, expectation, consoleErrors[]}`) per capture into `artifacts/`. A separate **multi-agent Workflow** then fans out — one vision-judge subagent per screenshot — each grading against the rubric and emitting the structured verdict. A reducer agent aggregates into a single pass/fail report with confidences. Navigation is 100% deterministic (reuses `openAppViaDock`, the `data-theme` setter, the `?skipOnboarding` entry); meaning is vision-graded; the two phases are decoupled so judging is re-runnable on a frozen capture set without re-driving the browser.
|
||||
**Pros:** Deterministic, replayable navigation; capture phase is plain Playwright (CI-gateable, runs headless on Linux today); judge phase parallelizes cleanly (N independent subagents, no shared state); a frozen capture set lets you re-grade after rubric tweaks for **$0 browser cost**; objective signals (console/network/lighthouse) attach per surface; failures ship the exact PNG as evidence.
|
||||
**Cons:** Two-phase orchestration (capture artifact contract must be stable); vision judging has per-screenshot model cost (~52 calls/run); can't react mid-journey to an unexpected modal (a scripted step that mis-navigates produces a "wrong surface" capture rather than self-correcting).
|
||||
**Effort:** **Medium.** ~1 capture spec (extends existing helpers) + 1 Workflow definition (judge fan-out + reducer) + rubric prompt. ~2–3 focused sessions.
|
||||
|
||||
### Option B — Live agentic drive (agents drive MCP browser turn-by-turn, judge in real time)
|
||||
**Mechanism:** A coordinator agent drives a live MCP browser (`mcp__plugin_playwright_playwright__*` or `mcp__chrome-devtools__*`) step by step: navigate → `take_screenshot` → judge with its own vision → decide the next action from what it sees (open dock zone, dismiss a modal, retry). No pre-scripted path; the agent explores the surface list and adapts.
|
||||
**Pros:** Most "agentic" — self-corrects around unexpected overlays/state; closest to how a human QA explores; no capture/judge contract to maintain; can chase a regression it notices ("that looked off, let me re-open it").
|
||||
**Cons:** **Least deterministic** — same run can take different paths, so it's a poor CI gate (flaky, non-reproducible verdicts); live browser cost on every step; one MCP browser session is effectively serial (hard to parallelize the way a frozen-PNG fan-out does); harder to attach to the existing `npm run test:visual` lane; debugging "why did it fail" means replaying a non-deterministic trace.
|
||||
**Effort:** **Medium-High.** Less *code* but more *prompt/loop engineering* to keep it bounded (loop-guard, step budget) and to make verdicts trustworthy. Ongoing cost per run.
|
||||
|
||||
### Option C — Hybrid (Playwright drives + captures + objective signals; vision agents grade meaning) — **RECOMMENDED**
|
||||
**Mechanism:** Option A's deterministic capture, **enriched per surface with objective signals**: alongside each PNG, capture `console` errors (`page.on('console')`), failed network requests, and a `lighthouse_audit` (a11y/contrast/perf) for the heavy views. The vision Workflow then grades *meaning* while the objective signals grade *facts* — and a surface only PASSes when **both** agree. Vision catches "looks broken / illegible / wrong screen"; Lighthouse + console catch "contrast ratio 1.9:1 / uncaught TypeError / 500 on mount" that vision might rationalize away. The reducer cross-checks: a vision-PASS with a console-error or a Lighthouse-a11y-fail is downgraded to FAIL with both pieces of evidence.
|
||||
**Pros:** Everything in A, **plus** a deterministic objective floor so the harness can't be fooled by a plausible-looking screenshot; `theme_legible` is corroborated by real contrast numbers, not just the model's eye; objective signals are cheap and CI-safe; gives two independent failure detectors (defense in depth).
|
||||
**Cons:** Most moving parts (capture + console + network + lighthouse + vision + reducer); Lighthouse adds runtime per surface (budget it to the heavy views, not all 52); slightly more report schema.
|
||||
**Effort:** **Medium-High** — A's effort + per-surface signal capture (mostly wiring existing CDP/Playwright APIs the repo already imports). ~3 sessions.
|
||||
|
||||
---
|
||||
|
||||
## 5. The ONE decision the user must resolve before build
|
||||
|
||||
> **Does the Chat round-trip flow grade against a REAL LLM reply, or against a gracefully-handled degraded state?**
|
||||
|
||||
This is forced by ground truth (§1.1): under the harness's own `--skip-litellm` server with no API key, `/api/chat` resolves the provider to **`degraded`** and **returns no assistant message**. So the chat flow's `flow_completes` dimension has two mutually exclusive definitions, and the harness must commit to one before any capture script is written:
|
||||
|
||||
- **Path 1 — Stub/degraded (deterministic, free, CI-default).** "Flow completes" = the user message renders, the send affordance works, and the app handles the missing-LLM state *gracefully* (a clear "configure API key" prompt, **not** a blank window or a stack trace). Fully deterministic, zero LLM spend, runs on CI Linux today. Does **not** verify a real answer renders.
|
||||
- **Path 2 — Real LLM (high-signal, costs money + a key, flaky).** Inject a real Anthropic key into the harness server so chat returns an actual reply; vision grades that a coherent assistant message rendered. Highest fidelity for the headline flow, but introduces non-determinism (model output varies), per-run cost, and a secret the CI gate must hold.
|
||||
|
||||
A sensible resolution (pending user call): **Path 1 as the CI gate; Path 2 as an opt-in `--live-llm` lane** for pre-release runs. But the user must pick the default before build, because it dictates the chat capture script, the expectation strings, and whether CI needs a secret.
|
||||
|
||||
**Secondary decisions** (lower stakes, can default): **run target** — local Chromium against the built `apps/web` on `:3333` (recommended default; matches existing config) vs the Tauri binary (true shipping surface, but no headless screenshot path on Windows CI) vs CI Linux (the gate); and **capture-vs-live-drive** — already resolved by recommending Option C (capture).
|
||||
|
||||
---
|
||||
|
||||
## 6. Recommendation
|
||||
|
||||
**Option C (Hybrid).** It keeps Option A's deterministic, replayable, CI-gateable capture (reusing the dock-open / theme / onboarding-skip helpers already in `tests/e2e`), adds a vision Workflow for *meaning*, and backstops the vision verdict with cheap objective signals (console errors + Lighthouse contrast/a11y) so the harness has a deterministic floor and can't be fooled by a screenshot that merely *looks* fine. Build it on top of the existing `tests/visual` + `tests/e2e` infrastructure rather than greenfield: a new capture spec emits PNG + sidecar JSON, a Workflow fans out one vision-judge subagent per capture, a reducer cross-checks vision against objective signals and writes one report. Default the chat flow to **Path 1 (degraded-handled-gracefully)** for the CI gate with a **Path 2 `--live-llm`** opt-in — pending the user's call on §5.
|
||||
|
||||
---
|
||||
|
||||
## 7. Build sketch (after the decision is made)
|
||||
|
||||
1. **Capture spec** (`tests/vision/capture.spec.ts`): iterate the surface matrix × {dark,light}; reuse `openAppViaDock`; set theme via `data-theme`; for each surface write `artifacts/<surface>-<theme>.png` + `<surface>-<theme>.json` (`expectation`, `consoleErrors[]`, `networkFailures[]`, optional `lighthouse`). Drive the 7 flows to their end-state captures.
|
||||
2. **Vision Workflow** (`workflow-composer` definition or a Task fan-out): one judge per capture → structured verdict; `dispatching-parallel-agents`-style fan-out.
|
||||
3. **Reducer**: merge vision verdicts + objective signals; downgrade vision-PASS-with-hard-signal to FAIL; emit `artifacts/vision-report.json` + a Markdown summary; non-zero exit on any FAIL for the CI gate.
|
||||
4. **Lanes**: `test:vision` (Path 1, CI) and `test:vision:live` (Path 2, pre-release, requires key).
|
||||
|
||||
---
|
||||
|
||||
## 8. Why not just keep the pixel-diff + substring suite
|
||||
|
||||
Pixel diff at `0.003` flags font-hinting and wallpaper jitter as failures while passing a screen whose *content* is wrong-but-pixel-identical-to-baseline; substring asserts (`toMatch(/persona/i)`) pass on a half-rendered, error-bannered, or dark-on-dark screen as long as one keyword survives. Neither answers the actual question — *"would a human look at this and say it's working and legible?"* Vision grading answers exactly that; the hybrid's objective floor keeps it honest.
|
||||
275
docs/audits/2026-07-08-admin-cli-utility-t15-analysis.md
Normal file
275
docs/audits/2026-07-08-admin-cli-utility-t15-analysis.md
Normal file
@@ -0,0 +1,275 @@
|
||||
# Admin, CLI, Marketplace, and MCP Utility T15 Analysis - 2026-07-08
|
||||
|
||||
Status: analysis supplement plus focused admin, launcher, marketplace, CLI, and MCP runtime hardening.
|
||||
|
||||
Purpose: deepen T15 evidence for `packages/admin-web`, `packages/cli`, `packages/launcher`, `packages/marketplace`, `packages/memory-mcp`, `packages/hive-mind-mcp-server`, and `packages/hive-mind-cli`.
|
||||
|
||||
Guideline baseline: Vercel Web Interface Guidelines, fetched 2026-07-08 from `https://raw.githubusercontent.com/vercel-labs/web-interface-guidelines/main/command.md`.
|
||||
|
||||
## Sources Inspected
|
||||
|
||||
- `packages/admin-web/package.json`
|
||||
- `packages/admin-web/src/App.tsx`
|
||||
- `packages/admin-web/src/api.ts`
|
||||
- `packages/admin-web/src/pages/*.tsx`
|
||||
- `packages/admin-web/tests/admin-pages.test.ts`
|
||||
- `packages/cli/package.json`
|
||||
- `packages/cli/src/index.ts`
|
||||
- `packages/cli/src/repl.ts`
|
||||
- `packages/cli/tests/cli-runtime.test.ts`
|
||||
- `packages/launcher/package.json`
|
||||
- `packages/launcher/src/cli.ts`
|
||||
- `packages/launcher/tests/cli.test.ts`
|
||||
- `packages/launcher/tsup.config.ts`
|
||||
- `packages/marketplace/package.json`
|
||||
- `packages/marketplace/tsconfig.json`
|
||||
- `packages/marketplace/src/cli.ts`
|
||||
- `packages/marketplace/src/db.ts`
|
||||
- `packages/memory-mcp/package.json`
|
||||
- `packages/memory-mcp/src/index.ts`
|
||||
- `packages/memory-mcp/src/core/setup.ts`
|
||||
- `packages/memory-mcp/tests/*.test.ts`
|
||||
- `packages/memory-mcp/README.md`
|
||||
- `packages/hive-mind-mcp-server/package.json`
|
||||
- `packages/hive-mind-mcp-server/src/index.ts`
|
||||
- `packages/hive-mind-mcp-server/src/core/setup.ts`
|
||||
- `packages/hive-mind-mcp-server/tests/scope.test.ts`
|
||||
- `packages/hive-mind-mcp-server/src/integration.test.ts`
|
||||
- `packages/hive-mind-mcp-server/README.md`
|
||||
- `packages/hive-mind-cli/package.json`
|
||||
- `packages/hive-mind-cli/README.md`
|
||||
- `packages/hive-mind-cli/src/index.ts`
|
||||
- `packages/hive-mind-cli/src/dispatch.ts`
|
||||
- `packages/hive-mind-cli/src/setup.ts`
|
||||
- `packages/hive-mind-cli/src/commands/*.ts`
|
||||
- `packages/hive-mind-core/package.json`
|
||||
- `packages/shared/package.json`
|
||||
- `packages/core/package.json`
|
||||
- `packages/core/src/index.ts`
|
||||
- `vitest.config.ts`
|
||||
|
||||
## Commands Run
|
||||
|
||||
```powershell
|
||||
npx vitest run packages/admin-web/tests/admin-pages.test.ts --reporter=dot
|
||||
npm run test --workspace @waggle/admin-web -- --reporter=dot
|
||||
npx vitest run packages/launcher/tests/cli.test.ts --reporter=dot
|
||||
npm run test --workspace @waggle/cli -- --reporter=dot
|
||||
npx vitest run packages/cli/tests/commands.test.ts packages/cli/tests/admin.test.ts packages/cli/tests/renderer.test.ts packages/cli/tests/memory-persistence-hard.test.ts packages/cli/tests/auth.test.ts packages/cli/tests/mode-detector.test.ts packages/cli/tests/comprehensive-e2e.test.ts packages/cli/tests/cli-runtime.test.ts --reporter=dot
|
||||
npm run test --workspace waggle-memory-mcp -- --reporter=dot
|
||||
npx vitest run packages/memory-mcp/tests/scope.test.ts packages/memory-mcp/tests/erase.test.ts --reporter=dot
|
||||
npm run test --workspace waggle-memory-mcp -- --reporter=dot
|
||||
npx vitest run packages/hive-mind-mcp-server/tests/scope.test.ts packages/hive-mind-mcp-server/src/integration.test.ts --reporter=dot
|
||||
npm run test --workspace @waggle/hive-mind-mcp-server -- --reporter=dot
|
||||
npx vitest run 'packages/hive-mind-cli/src/**/*.test.ts' --reporter=dot
|
||||
npx vitest run 'src/**/*.test.ts' --reporter=dot
|
||||
npx vitest run packages/hive-mind-cli/tests/cli-help.test.ts --reporter=dot
|
||||
npm run test --workspace @waggle/hive-mind-cli -- --reporter=dot
|
||||
npx vitest run packages/marketplace/tests/categories.test.ts packages/marketplace/tests/mcp-registry.test.ts packages/marketplace/tests/cisco-scanner.test.ts --reporter=dot
|
||||
npx vitest run packages/marketplace/tests/cli-runtime.test.ts packages/marketplace/tests/categories.test.ts packages/marketplace/tests/mcp-registry.test.ts packages/marketplace/tests/cisco-scanner.test.ts --reporter=dot
|
||||
npx tsc --noEmit --project packages/admin-web/tsconfig.json
|
||||
npx tsc --noEmit --project packages/cli/tsconfig.json
|
||||
npx tsc --noEmit --project packages/marketplace/tsconfig.json
|
||||
npx tsc --noEmit --project packages/server/tsconfig.json
|
||||
npx tsc --noEmit --project packages/memory-mcp/tsconfig.json
|
||||
npx tsc --noEmit --project packages/hive-mind-mcp-server/tsconfig.json
|
||||
npx tsc --noEmit --project packages/hive-mind-cli/tsconfig.json
|
||||
npm run build --workspace @waggle/admin-web
|
||||
npm run test:rendered --workspace @waggle/admin-web
|
||||
In-app Browser smoke of built `packages/admin-web/dist` against a typed mock API on `http://localhost:3100`
|
||||
In-app Browser focused smoke of built `packages/admin-web/dist` on `http://127.0.0.1:4181` at 390 x 844 and desktop widths
|
||||
npm run build --workspace @waggle-ai/waggle
|
||||
npm run test --workspace @waggle-ai/waggle -- --reporter=dot
|
||||
npm install <packed @waggle-ai/waggle tarball> --no-audit --no-fund --prefer-offline
|
||||
npx waggle --help
|
||||
npx waggle --port <occupied-port> --skip-litellm --no-open
|
||||
npm run build --workspace @waggle/cli
|
||||
npm pack --workspace @waggle/cli --pack-destination <temp> --json
|
||||
npm install <local @waggle/* package-closure tarballs> --no-audit --no-fund --ignore-scripts --prefer-offline
|
||||
npx waggle --help
|
||||
npm install <local @waggle/* package-closure tarballs> --no-audit --no-fund --prefer-offline
|
||||
npx waggle --local
|
||||
$env:LITELLM_API_KEY='sk-test'; npx waggle --local # against local mock LiteLLM-compatible /v1/chat/completions stream
|
||||
npm run build --workspace @waggle/marketplace
|
||||
npm pack --workspace @waggle/marketplace --pack-destination <temp> --json
|
||||
npm install <packed @waggle/marketplace tarball> --no-audit --no-fund --prefer-offline
|
||||
npx waggle-market --help
|
||||
npx waggle-market definitely-not-a-command
|
||||
npm run build --workspace waggle-memory-mcp
|
||||
npm run build --workspace @waggle/hive-mind-mcp-server
|
||||
npm install <local @waggle/hive-mind-mcp-server package-closure tarballs> --no-audit --no-fund --prefer-offline
|
||||
node <installed @waggle/hive-mind-mcp-server>/dist/index.js
|
||||
npm run build --workspace @waggle/hive-mind-cli
|
||||
npm install <local @waggle/hive-mind-* package-closure tarballs> --no-audit --no-fund --prefer-offline
|
||||
npx hive-mind-cli status --help
|
||||
npx tsx packages/launcher/src/cli.ts --help
|
||||
node packages/launcher/dist/cli.js --help
|
||||
npx tsx packages/launcher/src/cli.ts --port abc
|
||||
node packages/launcher/dist/cli.js --port abc
|
||||
node packages/launcher/dist/cli.js --port <occupied-port> --skip-litellm --no-open
|
||||
npm pack --workspace @waggle-ai/waggle --pack-destination <temp> --json
|
||||
npx tsx packages/cli/src/index.ts --help
|
||||
node packages/cli/dist/index.js --help
|
||||
npx tsx packages/hive-mind-cli/src/index.ts --help
|
||||
node packages/hive-mind-cli/dist/index.js --help
|
||||
npx tsx packages/hive-mind-cli/src/index.ts init --help
|
||||
npx tsx packages/hive-mind-cli/src/index.ts status --help
|
||||
npx tsx packages/marketplace/src/cli.ts --help
|
||||
npx tsx packages/marketplace/src/cli.ts definitely-not-a-command
|
||||
node packages/marketplace/dist/cli.js --help
|
||||
node packages/marketplace/dist/cli.js definitely-not-a-command
|
||||
npm pack --workspace @waggle/marketplace --dry-run --json
|
||||
npx tsx packages/marketplace/src/cli.ts definitely-not-a-command
|
||||
node --input-type=module -e "<MCP Client protocol smoke for built memory MCP and hive-mind MCP entries>"
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- Commands that needed user-home state were redirected to `output/t15-*` data directories where practical.
|
||||
- Launcher occupied-port runtime evidence used a temporary `WAGGLE_DATA_DIR` and a test-owned `127.0.0.1` port blocker.
|
||||
- Builds generated package `dist` artifacts; this supplement does not treat generated build output as product code edits.
|
||||
|
||||
## Command Results
|
||||
|
||||
| Check | Result | UX meaning |
|
||||
|---|---:|---|
|
||||
| Admin-web package/root Vitest | Pass, 1 file / 42 tests | Admin pages render in happy-path and auth-failure test cases; the admin shell now has package-local test ownership, hash deep links, active-nav semantics, labelled critical fields, responsive CSS coverage, wrong-token guidance, and no React `act(...)` warning noise in the focused run. The remaining warning is Node's `punycode` dependency deprecation. |
|
||||
| Launcher targeted Vitest | Pass, 1 file / 22 tests | The real parser/core helpers, package metadata, built-help no-service-side-effect behavior, invalid-port validation before service setup, occupied-port recovery copy, `--no-open` success copy, packed tarball first-command help path, clean installed packed-launcher `npx` help plus occupied-port startup recovery, and clean installed long-running startup through `/health` are covered. |
|
||||
| `@waggle/cli` package test script | Pass, 8 files / 57 tests | Package-local script now uses the root Vitest runner/config and covers command parsing, admin helper formatting, renderer, auth, mode detection, memory persistence, comprehensive E2E, built-help runtime behavior, packed tarball bin-help behavior, clean local package-closure install followed by `npx waggle --help`, clean local package-closure install followed by `npx waggle --local` REPL startup, slash-command interaction, and exit, and a clean local package-closure installed streamed chat turn through a mock LiteLLM-compatible endpoint. |
|
||||
| `@waggle/cli` root-directed tests | Pass, 8 files / 57 tests | The same explicit root-directed slice passes, including built-help, packed-bin-help, local package-closure install/`npx` help, installed local REPL startup/slash-command/exit regressions, and installed streamed chat/provider plumbing against a mock LiteLLM-compatible `/v1/chat/completions` endpoint. |
|
||||
| Memory MCP package test script | Pass, 3 files / 20 tests | Package-local script now uses the root Vitest runner/config and covers scope gating, erase safety, built read-only MCP handshake, built write-scope save/recall roundtrip, and clean local package-closure install followed by MCP tool listing from the installed server. |
|
||||
| Memory MCP root-directed tests | Pass, 3 files / 20 tests | Scope gating, erase safety, built read/write MCP behavior, and installed package-closure read-only server startup pass from the root command shape. |
|
||||
| Hive-mind MCP package test script | Pass, 2 files / 15 tests | Package-local script now covers registration/scope, a built write-scope save/recall roundtrip, and clean local package-closure install followed by MCP tool listing from the installed server. |
|
||||
| Hive-mind MCP root-directed tests | Pass, 2 files / 15 tests | Registration, scope gating, built write-scope behavior, and installed package-closure read-only server startup pass from the root command shape. |
|
||||
| Hive-mind CLI package-local tests | Pass, 5 files / 46 tests | `npm run test --workspace @waggle/hive-mind-cli -- --reporter=dot` now discovers the colocated `src` tests plus runtime help tests, including local package-closure install followed by `npx hive-mind-cli status --help`. Expected mock-embedding warning banners remain noisy but non-failing. |
|
||||
| Marketplace targeted tests | Pass, 4 files / 77 tests | Categories, MCP registry, Cisco scanner behavior, source invalid-command behavior, built help behavior, package manifest/packed-file alignment, and clean installed packed-CLI `npx waggle-market` help/invalid-command behavior pass in the targeted slice. |
|
||||
| No-emit TypeScript | Pass | `admin-web`, `cli`, `marketplace`, `memory-mcp`, `hive-mind-mcp-server`, and `hive-mind-cli` typecheck with no output. |
|
||||
| Package builds | Pass | Admin web, launcher, CLI, marketplace, memory MCP, hive-mind MCP, and hive-mind CLI build scripts completed. |
|
||||
| Admin-web rendered package smoke | Pass, 14 tests | `npm run test:rendered --workspace @waggle/admin-web` builds the package and runs Playwright against the built preview. It covers all seven admin pages at 1200 x 800 and 390 x 844 with typed authenticated mock API data, real local bearer-auth middleware wrong-token/valid-token behavior through protected Fastify routes, hash URL state, `aria-current`, document scroll width, overflow outside labelled table scroll regions, labelled controls, console warning/error/pageerror collection, mobile keyboard navigation through the shell, page-level keyboard traversal from connection fields into dashboard, members, capabilities, jobs, audit, and settings controls/table regions, browser back/forward hash traversal, full-page visual snapshots for all seven pages on desktop and mobile, capability governance edit/add/decision forms, malformed analytics data recovery without blanking the shell, all-page initial API-failure recovery with accessible alerts and usable shell navigation, and rendered mutation/destructive-failure recovery for capability policy save, capability override create/remove, capability request decision, member invite, member role change, member removal, and team settings save. Command output has a Node `NO_COLOR` env warning; app console collection is clean. |
|
||||
| Source help smokes | Pass | `launcher`, `@waggle/cli`, `hive-mind-cli`, and `marketplace` source help paths are readable through `tsx`. |
|
||||
| Built launcher help/error/package paths | Pass | Source and built help exit 0, print usage, do not create `.waggle`, and no longer print the `[waggle:service] Data dir: ...` banner before help. Source and built invalid-port paths exit 1 before service setup, print focused guidance, and do not create `.waggle`; the built occupied-port path exits 1 and prints `npx waggle --port <next-port>` recovery copy. The packed tarball contains `dist/cli.js`, exposes `bin.waggle`, and the extracted first-command help path runs without service setup or user-home mutation. A clean temp project can install the packed launcher and run `npx waggle --help`, installed occupied-port startup recovery with no `.waggle` home mutation, and installed long-running startup that serves `/health`, prints `--no-open` manual-open copy, and creates the configured data dir. |
|
||||
| Built `@waggle/cli` help/package path | Pass | `node packages/cli/dist/index.js --help` and `node packages/cli/bin/waggle.js --help` exit 0, print usage, and do not create `.waggle` in a clean temp home. The packed tarball contains `bin/waggle.js`; extracted packed-bin help exits 0, prints usage, and does not create `.waggle`. A clean temp project can install the local `@waggle/shared`, `@waggle/hive-mind-core`, `@waggle/core`, `@waggle/marketplace`, `@waggle/agent`, `@waggle/weaver`, and `@waggle/cli` tarball closure, run `npx waggle --help` without `.waggle` mutation, then run `npx waggle --local`, see the local REPL banner/prompt, run `/help`, `/mode`, `/whoami`, `/models`, `/cost`, and `/clear`, create `~/.waggle/default.mind`, and exit via `/exit`. A second clean temp project can install the same local package closure, configure `~/.waggle/config.json` plus `.waggle/workspace.json`, run `npx waggle --local` against a local mock LiteLLM-compatible streaming endpoint, send a user chat message, receive streamed assistant text plus usage metadata, and verify the outbound `Authorization`, `model`, `stream`, `stream_options`, and message payload. The installed REPL proof caught and fixed `@waggle/agent` package metadata pointing at source, missing `exceljs`/`@waggle/shared`/`@waggle/marketplace` runtime declarations, and a test harness native-install issue for `better-sqlite3`. |
|
||||
| Built marketplace help | Pass | `node packages/marketplace/dist/cli.js --help` exits 0, prints usage, and does not create `~/.waggle/marketplace.db` in a clean temp home. A clean temp project can install the packed marketplace tarball and run `npx waggle-market --help` without DB side effects. |
|
||||
| Hive-mind CLI subcommand help | Pass | Source `init --help`, built `status --help`, and local package-closure installed `npx hive-mind-cli status --help` exit 0, print focused command help, and do not create `personal.mind` in a clean temp data dir. The installed proof caught and fixed a missing `@waggle/shared` runtime dependency declaration in `@waggle/hive-mind-core`. |
|
||||
| Marketplace invalid command | Pass | Source, built, and clean installed packed-CLI invalid-command smokes print `Unknown command`, show help, exit 1, and do not create `~/.waggle/marketplace.db` in a clean temp home. |
|
||||
| Built legacy memory MCP protocol smoke | Pass, read/write + installed read-only | Official MCP client connects to `packages/memory-mcp/dist/index.js`, verifies read-only tool gating, and in write scope saves then recalls a unique memory from a temp `WAGGLE_DATA_DIR` with mock embeddings. A clean temp project can also install the local `@waggle/shared`, `@waggle/hive-mind-core`, `@waggle/core`, `@waggle/wiki-compiler`, and `waggle-memory-mcp` tarball closure, launch the installed server entry, and list read-only tools. The installed proof caught and fixed a missing `glob` runtime dependency declaration in `@waggle/core`. |
|
||||
| Built hive-mind MCP protocol smoke | Pass, read/write + installed read-only | Official MCP client connects to `packages/hive-mind-mcp-server/dist/index.js`, verifies the registered surface, and in write scope saves then recalls a unique memory from a temp `HIVE_MIND_DATA_DIR` with mock embeddings. A clean temp project can also install the local `@waggle/shared`, `@waggle/hive-mind-core`, `@waggle/hive-mind-wiki-compiler`, and `@waggle/hive-mind-mcp-server` tarball closure, launch the installed server entry, and list read-only tools. |
|
||||
|
||||
## MCP Protocol Smoke Detail
|
||||
|
||||
The smoke used `@modelcontextprotocol/sdk/client/index.js` and `@modelcontextprotocol/sdk/client/stdio.js`, not hand-written framing.
|
||||
|
||||
Legacy Waggle memory MCP result:
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "waggle-memory-mcp",
|
||||
"tools": 9,
|
||||
"sampleTools": [
|
||||
"recall_memory",
|
||||
"search_entities",
|
||||
"get_identity",
|
||||
"get_awareness",
|
||||
"list_workspaces"
|
||||
],
|
||||
"withheldTools": [
|
||||
"save_memory"
|
||||
],
|
||||
"writeRoundtrip": "save_memory -> recall_memory returned the unique saved text"
|
||||
}
|
||||
```
|
||||
|
||||
Hive Mind MCP result:
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "hive-mind-mcp-server",
|
||||
"server": { "name": "hive-mind-memory", "version": "0.1.0" },
|
||||
"tools": 9,
|
||||
"resources": 4,
|
||||
"sampleTools": [
|
||||
"recall_memory",
|
||||
"search_entities",
|
||||
"get_identity",
|
||||
"get_awareness",
|
||||
"list_workspaces",
|
||||
"harvest_sources"
|
||||
],
|
||||
"sampleResources": [
|
||||
"memory://personal/stats",
|
||||
"memory://identity",
|
||||
"memory://awareness",
|
||||
"memory://workspace/{id}"
|
||||
],
|
||||
"writeRoundtrip": "save_memory -> recall_memory returned the unique saved text"
|
||||
}
|
||||
```
|
||||
|
||||
Both MCP stderr streams clearly warn when mock embeddings are active. That is acceptable for the audit lane because the commands explicitly set the provider to mock.
|
||||
|
||||
## What Is Proven Now
|
||||
|
||||
- Admin web compiles, builds, and renders tested happy paths.
|
||||
- Admin web now has a package-local test command; the focused run passes 42 tests with no React `act(...)` warnings.
|
||||
- Admin web now has a package-local rendered Playwright gate; the run passes 14 tests across all seven pages at desktop and 390px mobile widths, including visual regression snapshots, page-level keyboard traversal, browser back/forward hash traversal, malformed analytics response recovery, all-page initial API-failure recovery, mutation/destructive-failure recovery for policy save, override create/remove, request decision, member invite, member role change, member removal, and team settings save, plus real local bearer-auth middleware wrong-token/valid-token behavior.
|
||||
- Admin web now distinguishes rejected auth from server-down failures and renders real server-injected team/member/task data after a valid token in the package-local rendered gate. Live deployed/Clerk/JWT team-server auth remains outside this local utility gate and belongs to launch/deploy evidence.
|
||||
- Admin web rendered with typed mock API data under the in-app Browser with no current-port console errors; original full-page screenshots and probes are saved under `output/playwright/admin-web-t15-57795/`.
|
||||
- Admin web rendered package smoke now proves shell-level 390px mobile layout, hash deep-link navigation, `aria-current` active state, all-page table scroll containment, rendered control labels, capability governance form labels, mobile shell keyboard reachability, and page-level keyboard traversal from connection fields into the covered admin pages on the built preview.
|
||||
- Launcher source and built help are callable.
|
||||
- Launcher package-local test command now passes 22 tests and guards the real parser/core helpers, built-help no-service-side-effect path, invalid-port validation before service setup, occupied-port recovery copy, `--no-open` success copy, packed tarball first-command help path, clean installed packed-launcher `npx` help plus occupied-port startup recovery, and clean installed long-running startup through `/health`.
|
||||
- `@waggle/cli` source help, built help, bin-wrapper help, packed tarball bin help, local package-closure install/`npx` help, local package-closure installed REPL startup/slash-command/exit, local package-closure installed streamed chat via mock LiteLLM-compatible endpoint, package-local tests, and direct root tests pass.
|
||||
- Marketplace source help, source invalid-command, built help, built invalid-command, clean installed packed-CLI help, and clean installed packed-CLI invalid-command paths are callable with no marketplace DB side effects.
|
||||
- Hive-mind CLI root help, built root help, sampled source subcommand help, sampled built subcommand help, and local package-closure installed `npx` subcommand help are callable with no data-dir mutation.
|
||||
- Hive-mind CLI package-local tests now run through a documented package command and guard the installed local package closure. The installed proof found and fixed the missing `@waggle/shared` dependency in `@waggle/hive-mind-core`.
|
||||
- Legacy memory MCP package-local tests now run through a documented package command; the built server completes real read-only and write-scope MCP handshakes; and a clean local package-closure install can launch the installed server and list read-only tools. The installed proof found and fixed the missing `glob` runtime dependency declaration in `@waggle/core`.
|
||||
- Hive-mind MCP package-local tests now run through a documented package command, and the built server completes real read-only/registration and write-scope MCP handshakes. The package lane also proves clean local package-closure install and read-only MCP startup from the installed server.
|
||||
- TypeScript no-emit checks pass for every inspected package that has a `tsconfig.json`.
|
||||
- Package build scripts complete for every inspected package.
|
||||
|
||||
## Still Not Proven
|
||||
|
||||
- Admin web exhaustive keyboard/focus traversal inside every hidden or future state remains incomplete; current rendered evidence proves shell keyboard reachability, page-level traversal, and form-label/overflow contracts.
|
||||
- Launcher browser-open fallback as an integration path, and service crash/failure copy beyond occupied-port startup failure. Clean installed packed-launcher help, occupied-port startup recovery, and long-running `/health` startup are now proven.
|
||||
- `@waggle/cli` registry-only install remains dependent on publishing the internal `@waggle/*` package closure. Installed package-closure non-help startup, common no-provider slash commands, clean exit, and streamed chat/provider plumbing against a mock LiteLLM-compatible endpoint are now proven; live external-provider proof remains launch-environment dependent.
|
||||
- Marketplace install/search flows against a real populated marketplace database. Clean installed packed-CLI help and invalid-command recovery are now proven.
|
||||
- Every hive-mind CLI subcommand help variant beyond the sampled `init` and `status` paths, and registry-only install after publishing the local `@waggle/hive-mind-*` package closure.
|
||||
- Marketplace CLI missing-db and missing-config error semantics beyond the now-guarded invalid-command path.
|
||||
- Registry-only behavior for legacy `waggle-memory-mcp`, `@waggle/cli`, and hive-mind packages after the internal package closures are published.
|
||||
- Legacy `waggle-memory-mcp` installed write-scope behavior remains unproven; current installed evidence is read-only list-tools.
|
||||
|
||||
## Line-Level Findings
|
||||
|
||||
| ID | Finding | Evidence | Correction |
|
||||
|---|---|---|---|
|
||||
| T15-1 | Built, packed, and locally installed `@waggle/cli` help is now lazy enough to avoid the REPL dependency graph, and the installed local REPL starts cleanly from a package closure. | `packages/cli/src/index.ts` handles help before dynamically importing `./repl.js`; `cli-runtime.test.ts` proves built help exits 0 without creating `.waggle`; clean-home smokes also pass for `dist/index.js --help`, `bin/waggle.js --help`, extracted packed-tarball `bin/waggle.js --help`, a clean temp project that installs the local `@waggle/*` package-closure tarballs before running `npx waggle --help`, and a real-script install followed by `npx waggle --local` startup, local banner/prompt rendering, `/help`, `/mode`, `/whoami`, `/models`, `/cost`, `/clear`, `/exit`, and `~/.waggle/default.mind` creation. A second real-script install configures a mock LiteLLM-compatible endpoint, sends a chat message through the installed REPL, and asserts streamed assistant text, usage metadata, auth, model, stream flags, and message payload. `packages/agent/package.json` and `packages/weaver/package.json` now expose built `dist` entries; `@waggle/agent` declares the runtime dependencies the installed REPL loads. | Focused fixed locally for built, packed, package-closure-installed help, package-closure-installed local REPL startup/slash-command/exit, and package-closure-installed streamed chat/provider plumbing; full closure still needs registry-only proof after internal packages are published. |
|
||||
| T15-2 | Built legacy `waggle-memory-mcp` now completes read-only and write-scope MCP handshakes, and both legacy memory MCP and hive-mind MCP now have installed package-closure proof. | `packages/core/package.json` and `packages/wiki-compiler/package.json` now expose their built `dist` entries; `packages/core/package.json` declares the runtime `glob` dependency used by `file-store`; `packages/memory-mcp/tests/runtime.test.ts` builds core/wiki/memory-mcp, launches `packages/memory-mcp/dist/index.js` through the official MCP SDK, sees read-only tools, confirms `save_memory` is withheld in read scope, then saves and recalls a unique memory in write scope. It also installs the local `@waggle/shared`, `@waggle/hive-mind-core`, `@waggle/core`, `@waggle/wiki-compiler`, and `waggle-memory-mcp` package closure, launches the installed server, and lists read-only tools. `packages/hive-mind-mcp-server/tests/runtime.test.ts` additionally installs the local hive-mind package closure and lists tools from the installed server. | Focused fixed locally for built/installed legacy MCP and built/installed hive-mind MCP; registry-only proof still depends on publishing the internal package closures. |
|
||||
| T15-3 | Built marketplace CLI was not runnable under Node ESM after `tsc`, and the publish manifest pointed at unpublished source files. | Current `packages/marketplace/tsconfig.json` uses NodeNext resolution; marketplace source imports use emitted `.js` specifiers; `node packages/marketplace/dist/cli.js --help` exits 0 with no DB side effect; `packages/marketplace/package.json` now points `main`, `types`, and `exports` at emitted `dist` files; `cli-runtime.test.ts` guards built help, `npm pack --dry-run --json` file/manifest alignment, and clean installed packed-CLI `npx waggle-market` help/invalid-command behavior without DB creation. | Focused fixed locally for help, invalid-command, manifest, and installed-bin first commands; full closure still needs install/search flows against a real populated marketplace database. |
|
||||
| T15-4 | Package-local test scripts and root discovery do not consistently run the tests that exist. | `@waggle/cli`, launcher, `hive-mind-cli`, and `waggle-memory-mcp` package scripts are now fixed through root-owned or package-local Vitest lanes. Other packages outside T15 still have package-local command-shape gaps tracked under T17. | Keep the T15 package commands in the verification lane and address broader package-local script drift under T17. |
|
||||
| T15-5 | Admin web now has package-local unit and rendered test ownership, and the focused test lanes are no longer noisy. | `packages/admin-web/package.json` defines `npm run test --workspace @waggle/admin-web` and `npm run test:rendered --workspace @waggle/admin-web`; `packages/admin-web/tests/admin-pages.test.ts` passes 42 tests without React `act(...)` warnings, and `packages/admin-web/tests/admin-rendered.spec.ts` passes 14 built-preview Playwright tests including real local bearer-auth middleware wrong-token/valid-token behavior. Members native confirm was already replaced with in-app confirmation. | Focused fixed locally; remaining T15 closure is registry-only proof after internal package publication. |
|
||||
| T15-6 | Hive-mind CLI README promises per-command help, and the sampled subcommand help paths are now side-effect free. | `packages/hive-mind-cli/src/index.ts` handles root and subcommand `--help` before dispatch; `cli-help.test.ts` guards source `init --help`, built `status --help`, and local package-closure installed `npx hive-mind-cli status --help` so none creates `personal.mind`. The installed RED test exposed `@waggle/hive-mind-core` importing `@waggle/shared` without declaring it; `packages/hive-mind-core/package.json` now declares the runtime dependency. | Focused fixed locally for sampled help and local package-closure install; broader closure still needs registry-only proof after publishing the internal packages and optional sampling across every subcommand help page. |
|
||||
| T15-7 | Marketplace invalid command exited successfully and help/default construction opened the DB before validation. | Current `packages/marketplace/src/cli.ts` handles help and unknown commands before `MarketplaceDB` construction; source and built invalid-command smokes exit 1, print help, and leave a clean temp home without `marketplace.db`; `cli-runtime.test.ts` guards this behavior. | Focused fixed locally. |
|
||||
| T15-8 | Launcher help, common startup-error UX, and packed/installed first-command behavior are now owned by the package test lane. | `packages/launcher/src/cli-core.ts` owns the real parser and startup copy, and `packages/launcher/src/cli.ts` imports the server lazily after help and validation. `packages/launcher/tests/cli.test.ts` passes 22 tests covering built help without service banners or `.waggle` creation, invalid ports exiting 1 before service setup, occupied ports producing `npx waggle --port <next-port>` recovery copy, `--no-open` success copy, `npm pack` tarball extraction followed by packed `dist/cli.js --help`, clean installed packed-launcher `npx` help plus occupied-port startup recovery without `.waggle` home mutation, and clean installed long-running startup that serves `/health`, prints manual-open copy, and creates the configured data dir. Source and built invalid-port smokes also exit 1 with focused guidance. | Focused fixed locally for help, invalid port, occupied port, `--no-open` copy, packed first-command help, installed-bin occupied-port recovery, and installed long-running startup; remaining closure needs browser-open fallback integration and service crash/failure copy beyond occupied-port startup failure. |
|
||||
| T15-9 | Legacy `waggle-memory-mcp` README is stale relative to startup behavior. | `packages/memory-mcp/README.md` advertises zero-config ONNX auto-download; `packages/memory-mcp/src/core/setup.ts` now falls back to mock unless a provider is configured. | Update README/setup copy so users understand mock vs semantic search behavior, or align implementation with the documented zero-config path. |
|
||||
| T15-10 | Hive-mind MCP README tool names do not match the current registered surface. | `packages/hive-mind-mcp-server/README.md` lists tools such as `add_relation`, `get_entity`, `switch_workspace`, `harvest_conversations`, `compact_memory`, and `cleanup_deprecated`; integration tests and MCP smoke show registered names such as `create_relation`, `save_entity`, `list_workspaces`, `create_workspace`, `harvest_import`, `cleanup_frames`, and `cleanup_entities`. | Regenerate the README tool table from registration tests or update it manually with a doc test. |
|
||||
| T15-11 | Admin web shell and dense tables are no longer functionally unusable on 390px mobile viewports. | `packages/admin-web/src/admin.css` collapses the fixed sidebar into a full-width top section at `@media (max-width: 720px)` and adds labelled `.admin-table-scroll` regions. Rendered package smoke at 390 x 844 covers all seven pages, checks document scroll width, and allows overflow only inside labelled table scroll regions. | Focused fixed for shell/mobile chrome and dense table containment. |
|
||||
| T15-12 | Admin web pages are addressable by hash, active navigation is semantic, and browser history traversal stays aligned. | `packages/admin-web/src/App.tsx` initializes from `window.location.hash`, listens to `hashchange`, writes hashes on nav clicks, and sets `aria-current="page"` on the active nav item. Unit coverage proves `#members` initialization and click-to-`#capabilities`; rendered package smoke proves hash URL and active `aria-current` across all seven pages at desktop/mobile widths, then goes Dashboard -> Members -> Capabilities -> back -> back -> forward -> forward and verifies URL, heading, and active nav at each step. | Focused fixed locally. |
|
||||
| T15-13 | Rendered admin controls now have accessible labels and browser metadata in the covered states. | Unit coverage proves Team Slug/Auth Token labels plus `name`/`autocomplete`, invite email/role labels, member role labels, Team Name label/metadata, and wrong-token guidance. Rendered package smoke scans all visible `input`, `select`, and `textarea` controls across seven pages plus capability policy edit, override add, request decision forms, and the real-auth connection state. | Focused fixed for rendered happy-path, governance form, and real-auth connection states; any future hidden-form states still need coverage when introduced. |
|
||||
| T15-14 | Admin analytics no longer blanks the shell on malformed successful responses, initial API failure is announced accessibly across pages, key admin mutations recover cleanly when rejected, and wrong-token auth failures show specific guidance. | The RED malformed-data rendered test reproduced missing recovery UI for `{ tokenUsage: ... }` analytics data. `Analytics.tsx` now validates the runtime response shape before rendering cards, clears stale data on load/error, and exposes incomplete-data recovery as `role="alert"`. A second RED rendered test reproduced missing accessible alerts during all-page API failure; Dashboard, Members, Jobs, Audit, Team Settings, and all Capabilities tabs now expose their existing error banners as `role="alert"`. New RED/GREEN rendered mutation tests cover capability policy save, capability override create/remove, capability request decision, member invite, member role change, member removal, and team settings save failures; the covered forms disable or announce the active action while pending, preserve the user's context after rejection, expose the rejection through `role="alert"`, and keep the shell usable. The real-auth rendered test routes built admin API calls through the real local `securityMiddleware`, proves a wrong token announces `Authentication failed`, then proves a valid token renders real server-injected team/member/task data. The GREEN rendered run passes 14/14 and includes page-level keyboard traversal plus full-page visual snapshots for each covered desktop/mobile admin page. | Focused fixed for malformed analytics data, initial page-load API failure, covered mutation/destructive failures, current page-level keyboard traversal, package-local visual regression, and local bearer-auth behavior. |
|
||||
|
||||
## T15 Acceptance
|
||||
|
||||
T15 remains open until either:
|
||||
|
||||
1. Admin/CLI/MCP utility surfaces are explicitly deferred from the five-persona score, or
|
||||
2. Evidence proves all of the following:
|
||||
|
||||
- Admin-web rendered gate remains green with local bearer-auth evidence.
|
||||
- `@waggle/cli`, launcher, marketplace CLI, memory MCP, hive-mind MCP, and hive-mind CLI built entries can run their published first commands from a clean environment. Current `@waggle/cli`, legacy memory MCP, hive-mind MCP, and `hive-mind-cli` evidence covers local package closures, including `@waggle/cli` installed local REPL startup/slash-command/exit and streamed chat/provider plumbing; launcher and marketplace evidence covers clean installed packed tarballs; registry-only proof for internal package closures depends on publishing those packages.
|
||||
- Utility help and invalid-input paths do not mutate user data or exit 0 on errors.
|
||||
- Package-local and root test commands either pass or have documented, passing alternatives.
|
||||
- MCP servers have at least one protocol-level smoke for read-only and write-capable scopes.
|
||||
- README/setup docs match actual command names, provider behavior, and data-dir behavior.
|
||||
|
||||
## Phase Impact
|
||||
|
||||
This does not change Phase 1. T15 remains a Phase 2/Launch final-product gate after in-app P0 blockers are cleared, unless the user explicitly asks to include admin/CLI/MCP utility work in Phase 1.
|
||||
113
docs/audits/2026-07-08-ai-tool-hook-t16-analysis.md
Normal file
113
docs/audits/2026-07-08-ai-tool-hook-t16-analysis.md
Normal file
@@ -0,0 +1,113 @@
|
||||
# T16 AI-Tool Hook Lifecycle UX Analysis
|
||||
|
||||
Date: 2026-07-08
|
||||
Scope: Launcher AI-tool detection, launch, hook install, hook verify, hook uninstall, live output, and the `packages/hive-mind-hooks-*` package set.
|
||||
Mode: analysis plus focused package-runtime evidence.
|
||||
|
||||
## Bottom Line
|
||||
|
||||
T16 is functional at the unit, package, manifest, backend-route, component-test, package-runtime, real safe-launch, and rendered-Launcher transition layers, but it is not ready for a 9/10 UX claim.
|
||||
|
||||
The strongest evidence is the hook package test suite, shared manifest contract, backend route tests, official package typechecks, compiled bin help smokes, package-local hook/shim test scripts, a fresh package-pack lifecycle lane that runs `npx @waggle/hive-mind-hooks-<id> install/verify/uninstall` for all six hook-capable packages, a real detected-CLI observed launch smoke, a real isolated `/api/tools/hooks` install/verify/uninstall route smoke for all six hook-capable tools, a rendered Launcher smoke with a mock local API, codified all-six rendered hook install/verify/uninstall transitions, focused regressions for observed tool output after exit plus hook result stdout/stderr/structured-failure visibility, focused rendered install/offline/long-output/adapter Playwright coverage, focused third-party adapter launch coverage, and Codex WindowsApps recovery coverage. The hook-result regressions include Backup/Recovery result labels, Verify check failed/manual-approval rows, uninstall restore/cleanup rows, long-output summary rows, empty-output Verify recovery copy, and Claude Desktop launch-only copy. The remaining blockers are user-facing: packaged desktop integration and noisy-but-passing hook output.
|
||||
|
||||
## User Jobs
|
||||
|
||||
- Detect installed AI tools.
|
||||
- Launch a detected tool in the current workspace.
|
||||
- Optionally pass a prompt when the tool supports inline prompt args.
|
||||
- See whether the tool is already running and inspect live output for observed launches.
|
||||
- Install hooks without corrupting an existing tool configuration.
|
||||
- Verify hooks and understand failures or manual trust steps.
|
||||
- Uninstall hooks byte-identically, or remove only Waggle-managed files when Waggle created the config.
|
||||
- Understand that Claude Desktop is launchable but not hook-capable.
|
||||
- Recover when the sidecar, hive-mind CLI, or target AI tool is unavailable.
|
||||
- For advanced users, add a third-party adapter and expect detected tools to behave coherently.
|
||||
|
||||
## Source Model
|
||||
|
||||
- Built-in tool manifests live in `packages/shared/src/tool-detection.ts`.
|
||||
- The canonical built-ins are `claude-code`, `claude-desktop`, `cursor`, `codex`, `codex-desktop`, `hermes`, and `openclaw`.
|
||||
- All seven built-ins are launchable.
|
||||
- Six built-ins are hook-capable: all except `claude-desktop`.
|
||||
- `packages/hive-mind-hooks-claude-desktop` is an intentional stub with no `bin`.
|
||||
- `packages/hive-mind-hooks-codex-desktop` is a thin wrapper around the Codex hook package and writes to the shared `~/.codex` hook config.
|
||||
- Detection uses `getToolRegistry()`, which merges built-ins plus validated third-party manifests from `~/.waggle/adapters/*.json`.
|
||||
- `/api/tools/launch` now validates IDs against the runtime registry, so launchable third-party adapters can launch through the sidecar route and can receive server-applied `promptArgTemplate` prompts.
|
||||
- `/api/tools/hooks` remains intentionally limited to the known hook-capable built-ins; third-party hook management needs a separate safe hook command/package policy before it should be exposed.
|
||||
- On Windows, PATH detection now prefers spawnable `where.exe` hits such as `.exe`, `.cmd`, `.bat`, or `.com` over extensionless POSIX npm shims. Standard npm `.cmd` shims are resolved to their `node <module>` target so prompts/args stay literal; unknown `.cmd`/`.bat` files fall back to a quoted `cmd.exe call`. Codex found only through the restricted WindowsApps app alias is reported as installed but not launchable, with recovery copy instead of a failing Launch button.
|
||||
|
||||
## Command Evidence
|
||||
|
||||
| Check | Result | Notes |
|
||||
|---|---:|---|
|
||||
| `npx vitest run packages/shared/tests/tool-manifests.test.ts packages/agent/tests/tool-manifest-loader.test.ts packages/agent/tests/phase4-hooks-cohort.test.ts packages/agent/tests/tool-launcher.test.ts packages/agent/tests/hook-packages-runtime.test.ts packages/server/tests/tools-routes-launch.test.ts packages/server/tests/tools-routes.test.ts --reporter=dot` | Pass, 7 files / 96 tests | Proves manifest, loader, cohort, backend route, process, launch, hook route contracts, and package-packed installed hook lifecycle. Output includes expected mock embedding warning noise from server setup. |
|
||||
| `npm run test -w apps/web -- src/components/os/apps/LauncherApp.test.tsx src/lib/launcher-prompt-args.test.ts src/lib/adapter.launcher.test.ts --reporter=dot` | Pass, 3 files / 28 tests | Proves Launcher A/B toggle, one hook-capable non-Claude example, live-output pane wiring, prompt arg helpers, and adapter launcher methods. Emits Node `punycode` deprecation warnings. |
|
||||
| `npm run build` + in-app Browser rendered smoke on `/launcher?watch=1&skipOnboarding=true&skipBoot=true&skipBriefing=true` | Pass, partial state matrix | Fresh production web build passed. Browser DOM snapshot API failed with `TypeError: o.incrementalAriaSnapshot is not a function`, so evidence used the supported in-app Browser screenshot and targeted DOM-evaluate APIs. Artifacts: `output/playwright/launcher-t16-54147/launcher-t16-rendered-summary.json` plus five screenshots for mixed state, prompt summary, install success, verify failure, and running output. |
|
||||
| `npx vitest run packages/hive-mind-hooks-core/tests --reporter=dot` | Pass, 5 files / 82 tests | Proves shared hook core handlers, install primitives, JSON register merge, path helpers, and fail-open signal behavior. |
|
||||
| `npx vitest run packages/hive-mind-hooks-codex/tests packages/hive-mind-hooks-codex-desktop/tests packages/hive-mind-hooks-cursor/tests packages/hive-mind-hooks-hermes/tests packages/hive-mind-hooks-openclaw/tests packages/hive-mind-hooks-claude-code/tests packages/hive-mind-shim-core/tests --reporter=dot` | Pass, 55 files / 491 tests, 1 skipped | Proves package-level install/verify/uninstall, lifecycle handlers, Codex Desktop parity, shim core, fail-open behavior, and temp-config reversibility. Output is noisy with expected logs and warnings. |
|
||||
| Official package typechecks for `hive-mind-shim-core`, `hive-mind-hooks-core`, `claude-code`, `codex`, `codex-desktop`, `cursor`, `hermes`, and `openclaw` | Pass, 8/8 | Each package script runs `tsc --build && tsc --noEmit -p tsconfig.test.json`. |
|
||||
| `npm run build --workspace @waggle/hive-mind-hooks-claude-desktop` | Pass | Confirms the intentional no-bin Claude Desktop stub still builds. |
|
||||
| Compiled bin help smokes for `claude-code-hooks`, `codex-hooks`, `codex-desktop-hooks`, `cursor-hooks`, `hermes-hooks`, and `openclaw-hooks` | Pass, 6/6 | Local compiled bin entrypoints boot and show usage. |
|
||||
| `npx vitest run packages/agent/tests/hook-packages-runtime.test.ts --reporter=verbose` | Pass, 1 file / 1 test | Builds and packs `@waggle/hive-mind-shim-core`, `@waggle/hive-mind-hooks-core`, and all six hook-capable packages into tarballs, installs the local package closure into a clean temp project, then runs production-like `npx --yes @waggle/hive-mind-hooks-<id> install/verify/uninstall` for `claude-code`, `codex`, `codex-desktop`, `cursor`, `hermes`, and `openclaw`. Each install pins a fake `hive-mind-cli --help` target, verify passes, uninstall removes the pointer, and the config is restored or removed as expected. |
|
||||
| `npm run test --workspace @waggle/hive-mind-hooks-core -- --reporter=dot` | Pass, 5 files / 82 tests | Package-local script now delegates to the root Vitest config with a package-specific path. |
|
||||
| Package-local `npm run test` for `hive-mind-hooks-claude-code`, `codex`, `codex-desktop`, `cursor`, `hermes`, `openclaw`, and `hive-mind-shim-core` | Pass, 55 files / 492 tests | Package-local scripts now run the intended root-config lanes. The shim script builds the in-monorepo CLI first, then its integration test verifies the CLI ESM resolver fix: `hive-mind-cli mcp call` now resolves the ESM-only MCP server via `import.meta.resolve`. |
|
||||
| `npx vitest run packages/agent/tests/tool-launcher.test.ts packages/agent/tests/tool-detection.test.ts packages/agent/tests/tool-registry.test.ts packages/server/tests/tools-routes-launch.test.ts packages/server/tests/launch-args.test.ts --reporter=dot` | Pass, 5 files / 108 tests | Proves registry-aware detection metadata, launchable third-party adapter launch, prompt template application through `/api/tools/launch`, built-in launch/hook contracts, and route/process persistence behavior. Output includes expected mock embedding warning noise from server setup. |
|
||||
| `npx vitest run packages/agent/tests/tool-launcher.test.ts packages/agent/tests/tool-detection.test.ts --reporter=dot` | Pass, 2 files / 73 tests | Adds Windows real-world launcher guardrails: `where.exe` now prefers spawnable `.cmd`/`.exe` hits over extensionless npm shims, standard npm `.cmd` shims resolve to their Node module target instead of raw `cmd.exe`, unknown `.cmd`/`.bat` files use a quoted fallback, and async child-spawn failures no longer become unhandled sidecar crashes. |
|
||||
| `npx vitest run packages/agent/tests/tool-launcher.test.ts packages/agent/tests/tool-detection.test.ts --reporter=dot` | Pass, 2 files / 76 tests | Adds hook-management command guardrails: Windows hook commands resolve the Node-installed `npx.cmd` instead of `execFile('npx')` or a broken local shim, and default exec capture now uses the shared `.cmd` resolver. |
|
||||
| `npx vitest run packages/server/tests/tools-routes-launch.test.ts packages/agent/tests/tool-process-tracker.test.ts packages/agent/tests/tool-launcher.test.ts --reporter=dot` | Pass, 3 files / 98 tests | Revalidates launch route, process tracker, observed launch, hook route, persistence/reconcile, registry-aware launch, and Windows command invocation behavior after the real-tool fix. Output still includes expected mock embedding warning noise. |
|
||||
| `npm run build:packages` | Pass | Rebuilt shared/core/agent/server package output so the sidecar imports the updated `@waggle/agent` dist for real-tool Playwright evidence. |
|
||||
| Built package detection probe via `node -e "import('./packages/agent/dist/tool-detection.js')..."` | Pass | On this Windows host, Codex resolves to `C:\Program Files\WindowsApps\OpenAI.Codex_26.623.19656.0_x64__2p2nqsd0c76g0\app\resources\codex.exe` and is now reported as `installed: true`, `launchable: false`, `version: null`, with the WindowsApps recovery diagnostic. OpenClaw still resolves through the spawnable npm `.cmd` path in the broader package probe. |
|
||||
| `npx vitest run packages/agent/tests/tool-detection.test.ts --reporter=dot`; `npx vitest run src/components/os/apps/LauncherApp.test.tsx --reporter=dot` from `apps/web` | Pass, 30 agent tests + 15 Launcher tests | Adds focused Codex WindowsApps regressions: the detector reports the restricted app alias as installed but not launchable, and the Launcher hides the Launch button while showing recovery copy instead of generic adapter-not-configured text. Web test output includes the expected Node `punycode` deprecation warning. |
|
||||
| `node -e "import('./packages/agent/dist/tool-command.js')..."` | Pass | Built helper resolves the real OpenClaw npm `.cmd` shim to `node C:\Users\MarkoMarkovic\AppData\Roaming\npm\node_modules\openclaw\openclaw.mjs` and preserves metacharacter args such as `foo&echoBAD` and `100%` without `cmd.exe`. |
|
||||
| `WAGGLE_E2E_REAL_TOOLS=1 WAGGLE_E2E_PORT=34242 WAGGLE_E2E_BASE_URL=http://127.0.0.1:34242 npx playwright test tests/e2e/launcher-real-tool-lifecycle.spec.ts --project=chromium --reporter=list` | Pass, 1 file / 1 test | Fresh production build plus clean sidecar rendered Launcher, detected a real safe CLI (`OpenClaw` on this host), launched it through `/api/tools/launch` with safe `--version` args in observed mode, streamed real output, observed exit code 0, and verified `/api/tools/processes` cleared the pid. |
|
||||
| `WAGGLE_E2E_REAL_HOOKS=1 WAGGLE_E2E_HOOK_HOME=<temp> USERPROFILE=<temp> HOME=<temp> WAGGLE_E2E_PORT=34247 WAGGLE_E2E_BASE_URL=http://127.0.0.1:34247 npx playwright test tests/e2e/launcher-real-hook-lifecycle.spec.ts --project=chromium --reporter=list` | Pass, 1 file / 1 test | Fresh production build plus clean sidecar drove `/api/tools/hooks` through real `install`, `verify`, and `uninstall` for all six hook-capable tools: `claude-code`, `codex`, `codex-desktop`, `cursor`, `hermes`, and `openclaw`. Each case ran against a throwaway `HOME`/`USERPROFILE`, asserted config and pointer creation, Verify returned `All checks passed.`, uninstall removed the pointer and restored or removed config as appropriate, and OpenClaw's managed hook dir was removed. The first broadened run timed out at the default 30s Playwright test limit; the spec now uses a 180s timeout for the 18 synchronous route calls. |
|
||||
| `npm run test -w apps/web -- src/lib/adapter.authgate.test.ts src/lib/adapter.sse.test.ts src/lib/adapter.launcher.test.ts src/components/os/apps/launcher/ToolOutputPane.test.tsx src/components/os/apps/LauncherApp.test.tsx --reporter=dot` | Pass, 5 files / 72 tests | Adds and verifies the observed-output regression: after an `exit` event, `streamToolOutput()` closes the EventSource and does not reconnect/replay old buffered output. Also protects hook result visibility: install success is summarized with `Backup` and `Recovery` labels instead of a raw `stdout:` row, verify failure preserves stderr even when `error` is generic, Verify `[FAIL]` output is summarized as `Check failed` with the manual approval detail and no raw `[FAIL]`, uninstall output labels restore/cleanup rows as `Changed file`, `Restored from`, `Created file removed`, `Backup removed`, and `Pointer removed` without implying an install pointer, long hook output is capped behind a `More output` summary while keeping recovery guidance, a structured hook failure with no stderr/error is not replaced by raw `HTTP 400`, an empty-output Verify failure shows retry/uninstall/reinstall recovery copy, installed Claude Desktop is explicitly labeled as launch-only with no hook actions, and a detected launchable third-party adapter gets a Launch action and sends its raw prompt. Output includes expected Node `punycode` deprecation warnings. |
|
||||
| `npx playwright test tests/e2e/launcher-rendered-states.spec.ts --project=chromium --reporter=list` | Pass, 1 file / 5 tests | Production-build rendered Launcher proof now covers sidecar-offline recovery with an inline `Retry tool detection` action, long hook stderr summarization with `More output`, hidden-line count, and recovery guidance, standard install output with `Changed file`, `Install pointer`, `Backup`, and `Recovery` labels without raw hook command chatter, all six hook-capable tools rendering install/verify/uninstall state transitions with `Hooks active` refreshes, and a non-built-in launchable adapter state with Launch-only/no-hook copy, prompt routing, and launch payload assertion. |
|
||||
| `npm run test -w apps/web -- src/test/motion-class-hygiene.test.ts src/test/wave-u-chat-action-row.test.tsx --reporter=dot`; `npm run test -w apps/web -- src/test/build-warning-hygiene.test.ts src/test/motion-class-hygiene.test.ts --reporter=dot`; `npm run build` | Pass, 3 focused web test files + production build | Adds source hygiene guards that ban Tailwind-ambiguous `duration-[var(--mo-*)]` / `ease-[var(--mo-*)]` class tokens, require named motion utilities, and prevent the adapter from dynamically importing `shape-selection.ts`. The production build no longer emits the prior Tailwind ambiguity warnings or the `shape-selection.ts` dynamic/static import warning. Remaining build/playwright noise includes `NO_COLOR`/`FORCE_COLOR`, mock embedding banners, and expected hook negative-path logs. |
|
||||
| `npm run test -w apps/web -- src/test/build-warning-hygiene.test.ts --reporter=dot`; `npm run typecheck:web`; `npm run build`; `WAGGLE_E2E_PORT=4320 WAGGLE_E2E_BASE_URL=http://localhost:4320 npx playwright test tests/e2e/user-journeys.spec.ts --project=chromium --grep "J3:|J5:|J6:|J-mobile: Command Center|J-mobile: first-run onboarding|J-route-coverage|J10:|J11:" --reporter=list` | Pass, 4 build-hygiene tests, web typecheck, production build, focused rendered smoke 8 passed / 1 skipped | Route surfaces, closed shell overlays, ChatHost, and PostHog analytics are lazy-loaded and guarded. Current production build no longer emits the Vite large-chunk warning; startup JS is 421.96 kB minified / 114.08 kB gzip, and PostHog is split into a separate 208.95 kB chunk. Focused Chromium smoke covers Workspace Switcher, keyboard shortcuts, Command Center mobile, first-run onboarding mobile, route shells, Home, and keyboard-help overlay. |
|
||||
| `npx tsc --noEmit --project packages/shared/tsconfig.json`; `npx tsc --noEmit --project packages/agent/tsconfig.json`; `npx tsc --noEmit --project packages/server/tsconfig.json`; `npm run typecheck:web` | Pass, 4/4 | Proves the shared detection metadata, agent launch/process contracts, server route, and web UI stay type-consistent after the adapter launch fix. |
|
||||
| Fresh in-app Browser route smoke on `http://127.0.0.1:8096/launcher` with sidecar `3336` | Pass for HTTP 400 symptom and empty-output recovery; T16 still partial | The real route rendered Tool Launcher, detected installed tools, and exposed hook actions. Clicking read-only `Verify` on the first hook-capable tool now renders `verify failed (exit 1)`, `No hook output was returned`, and retry/uninstall/reinstall guidance; it does not render `HTTP 400`, and console errors/warnings for the interaction were empty. The real hook command produced no stdout/stderr detail in that run, so real installed target-app install/verify/uninstall states still need proof. |
|
||||
| In-app Browser mocked Verify check failure on `http://127.0.0.1:8104/launcher` | Pass for visible check/manual-trust copy; console not clean evidence | A browser-scoped API mock rendered installed Codex, clicked the single Verify action, and fulfilled the hook route with `[PASS]` and `[FAIL]` Verify stdout. The visible panel showed `CHECK FAILED`, `hook command trusted: manual approval required in Codex settings`, and `RECOVERY`, while raw `[FAIL]` was absent. The Browser DOM snapshot API again hit the known `incrementalAriaSnapshot` issue, so proof used targeted DOM evaluation and screenshot evidence. Console logs were contaminated by earlier failed mock attempts and background polling timeouts, so component tests remain the clean console owner. |
|
||||
| In-app Browser rendered Claude Desktop mock state on `/launcher` | Pass for visible state; console not used as clean evidence | A browser-scoped API mock rendered installed Claude Desktop with one Launch button, no Install hooks or Verify buttons, a `Launch only` badge, and `Hooks are not supported for Claude Desktop yet.` copy. The Browser DOM snapshot API hit the known `incrementalAriaSnapshot` issue, so proof used targeted locators and screenshot evidence. Failed earlier mock attempts left stale console log entries in Browser's collector, so the component regression is the clean console owner for this state. |
|
||||
| In-app Browser mocked Codex uninstall cleanup on `http://127.0.0.1:8105/launcher` | Pass for visible restore/cleanup copy; console not clean evidence | A browser-scoped API mock rendered installed Codex, clicked the single Uninstall hooks action, and fulfilled the hook route with standard uninstall stdout. The visible panel showed `Codex: uninstall OK`, `CHANGED FILE`, `RESTORED FROM`, `CREATED FILE REMOVED`, `BACKUP REMOVED`, and `POINTER REMOVED`; `Install pointer` and raw `- backup removed` text were absent. The test tab completed onboarding via the visible `Skip setup` control first; sidecar-off shell polling still produced background console errors, so component tests remain the clean console owner. |
|
||||
|
||||
## UX Findings
|
||||
|
||||
| ID | Severity | Finding | Evidence | Correction Needed |
|
||||
|---|---:|---|---|---|
|
||||
| T16-1 | Rendered fixed; packaged residual | Rendered Launcher hook states are now codified across all six hook-capable tools, but packaged desktop hook-status transitions are not yet proven. | The in-app Browser smokes render installed, not installed, hooks-active, running, Phase 4/unsupported, prompt summary, install success, verify failure, mocked uninstall cleanup, and live-output states. A codified Playwright spec now proves sidecar-offline retry, long stderr summarization, standard install changed-file/pointer/backup/recovery labels, all six hook-capable tools rendering install/verify/uninstall state transitions with `Hooks active` refreshes, and rendered non-built-in adapter launch-only/prompt behavior. A gated real-tool Playwright smoke now renders Launcher with a real detected CLI and proves observed safe launch/output/exit/process-clear through the sidecar. A gated route-level smoke now proves real hook install/verify/uninstall for all six hook-capable tools against an isolated profile. | Add packaged desktop hook-status evidence, or explicitly defer packaged hook management from final scoring. |
|
||||
| T16-2 | Route + rendered fixed; packaged residual | Real detected-CLI launch, the full hook-capable route lifecycle, and the full rendered installed-app hook matrix are proven; packaged desktop integration is not. | The packed package command lifecycle is proven hermetically for all six hook-capable packages. A gated Playwright smoke drove real OpenClaw detection and safe observed `--version` launch through the production sidecar route, streamed output, saw exit 0, and verified process tracking cleared the pid. The route lifecycle smoke now drives `/api/tools/hooks` through real `install`, `verify`, and `uninstall` for `claude-code`, `codex`, `codex-desktop`, `cursor`, `hermes`, and `openclaw` against an isolated `USERPROFILE/HOME`, proving config/pointer creation and cleanup without touching the user's real profile. The rendered state spec mocks the local API adapter but covers every supported hook-capable card's install, verify, uninstall, and refreshed hooks-active UI transitions. No packaged-desktop hook-status transition has been proved. | Add packaged desktop state evidence, or document approved deferrals for installed-app config-editing UI flows. |
|
||||
| T16-3 | Local fixed | Package-pack `npx @waggle/hive-mind-hooks-<id>` resolution is now proven for all Launcher hook targets. | `hook-packages-runtime.test.ts` installs the packed local package closure into a temp project and invokes every hook-capable package with the same package-name shape Launcher uses: `npx --yes @waggle/hive-mind-hooks-<id> install/verify/uninstall`. | Keep this in the release lane; registry-only proof after actual publication remains launch/deploy evidence rather than a local code blocker. |
|
||||
| T16-4 | Rendered fixed; release residual | Hook result copy now has a compact structured panel for focused install/verify/uninstall results: backup paths are labeled, raw `stdout:` is hidden, Verify check failures become `Check failed` rows with manual approval details, uninstall restore/cleanup rows do not imply install state, long output is capped behind a count summary, structured live failures are not masked as `HTTP 400`, empty-output Verify failures show recovery copy, offline detection has an inline Retry action, route-level lifecycle is real-proved for all six hook-capable tools, and rendered all-tool install/verify/uninstall transitions are codified. | Red component tests reproduced install success hiding a stdout backup path, verify failure dropping stderr when `error` was generic, empty-output Verify showing only an exit code, Verify `[FAIL]` output showing as generic raw output, uninstall cleanup rows appearing as generic Backup/Install pointer state, and long stderr flooding the result panel. A red adapter test reproduced the live shape `{ ok:false, code:1, stdout:'', stderr:'' }` being overwritten as `HTTP 400`; the adapter now preserves that hook envelope. The focused Launcher/web suite now proves `Backup` and `Recovery` labels for install output, stderr preservation, no raw `stdout:` row for the covered install case, `Check failed` manual-approval output without raw `[FAIL]`, uninstall restore/cleanup labels, `More output` summarization for long hook output, and empty-output recovery copy. Browser evidence on mocked Verify and Uninstall states renders the manual approval detail, `RECOVERY`, and restore/cleanup labels. A fresh real Browser smoke renders `verify failed (exit 1)` plus retry/uninstall/reinstall guidance instead of `HTTP 400`. A codified rendered Playwright spec proves standard install changed-file/pointer/backup/recovery labels, offline retry, long-output summarization, and all-six hook-card install/verify/uninstall transitions. A gated route Playwright smoke proves real install/verify/uninstall for all six hook-capable tools succeeds through `/api/tools/hooks` after Windows `npx` command resolution was fixed. | Keep the lifecycle panel regression green; finish packaged desktop status proof and warning hygiene. |
|
||||
| T16-5 | Focused fixed | Claude Desktop's unsupported-hook state is explicit in the UI. | Manifest marks it non-hook-capable and its package is a no-bin stub. A red component test reproduced the old state where installed Claude Desktop had only Launch and no explanation. The Launcher now shows a `Launch only` badge, `Hooks are not supported for Claude Desktop yet.` copy, one Launch button, and no Install hooks or Verify actions; a Browser-rendered mocked state confirmed the visible layout. | Keep this focused regression in the T16 lane. |
|
||||
| T16-6 | Focused fixed | Launchable third-party adapters can be detected and launched through the route/UI contract; hook management remains intentionally built-in-only. | Red tests reproduced the gap: `launchTool()` rejected a registered `foo-cli`, `/api/tools/launch` rejected the adapter id before applying its prompt template, detection omitted launch/prompt metadata, and Launcher rendered the adapter as a non-actionable Phase 4 item. The current contract uses registry metadata for detection, validates launch IDs against `getToolRegistry()`, applies `promptArgTemplate` server-side, tracks adapter process IDs as strings, and shows a Launch action plus prompt routing for launchable detected adapters. Focused backend/agent tests pass 108/108, tracker/route regression tests pass 96/96, the focused web Launcher suite passes 72/72, shared/agent/server/web typechecks pass, and rendered Playwright coverage proves a non-built-in adapter shows launch-only/no-hook copy, routes the prompt, and sends the expected launch payload. | Keep third-party hook management disabled until a safe hook command/package policy exists. |
|
||||
| T16-7 | Local fixed | Hook package-local test scripts now run their intended lanes. | Package-local `npm run test --workspace ...` now passes for hook core, all six hook-capable packages, and shim core. The fix also replaced the CLI's CommonJS-only `createRequire().resolve()` path with ESM-compatible `import.meta.resolve` for the MCP server entry. | Keep these package-local scripts in the release lane; remaining package-local command-shape gaps are tracked under T17. |
|
||||
| T16-8 | Partially fixed | Standard hook/test output is still too noisy, but the Tailwind motion-token ambiguity warnings, `shape-selection.ts` dynamic/static import warning, and Vite large-chunk warning are fixed. | Passing root-run hook tests still emit install logs, fail-open warnings, sidecar-unreachable drops, and server embedding degradation banners. Playwright output still includes `NO_COLOR`/`FORCE_COLOR` and mock-embedding noise. The old Tailwind ambiguity warnings from `duration-[var(--mo-base)]`, `duration-[var(--mo-fast)]`, and `ease-[var(--mo-ease)]` no longer appear after replacing them with named motion utilities guarded by `motion-class-hygiene.test.ts`; the defeated `shape-selection.ts` dynamic import no longer appears after promoting the adapter dependency to a static import guarded by `build-warning-hygiene.test.ts`; the oversized startup chunk no longer appears after lazy-loading routes, closed shell overlays, ChatHost, and PostHog analytics behind `build-warning-hygiene.test.ts`. | Quieten or isolate the remaining expected warnings in the standard release lane so real hook failures stand out. |
|
||||
| T16-9 | Partially fixed | Launcher prompt metadata is fixed, including third-party prompt-template metadata, but prompt-support transparency still needs a focused pass. | Current Launcher source gives the optional prompt textarea `id`, `name`, `aria-label`, `autocomplete`, and a visible label. The accepts/ignores summary now includes adapters whose manifests have `promptArgTemplate`, but it still appears only after text exists. | Make prompt support obvious per tool before launch. This can be bundled with T10 form/focus work. |
|
||||
| T16-10 | Focused fixed | Observed live output no longer reconnects and replays duplicate terminal output after process exit. | The red regression in `adapter.sse.test.ts` reproduced the issue: after `line` and `exit`, the EventSource stayed open and could reconnect. `streamToolOutput()` now closes its EventSource on a valid `exit` event; the current focused Launcher/web suite passes 5 files / 72 tests. | Keep this regression in the T16 lane; rendered no-duplicate screenshot evidence can be refreshed when the broader Launcher state matrix is rerun. |
|
||||
| T16-11 | Focused fixed | Windows npm shim launch/version behavior is now safer, failed child spawns no longer crash the sidecar, and restricted Codex WindowsApps aliases no longer show a failing Launch path. | Real evidence found `where.exe openclaw` returning an extensionless POSIX shim before `openclaw.cmd`, which made Node spawn fail. Detection now prefers spawnable Windows hits, standard npm `.cmd` shims resolve to their Node module target, unknown `.cmd`/`.bat` files use a quoted fallback, and `defaultSpawnDetached`/`defaultSpawnObserved` guard async child `error` events. The first real Playwright smoke reproduced the old behavior as a sidecar crash; the final smoke passed. On this Windows host, Codex detects only through a WindowsApps app alias that refuses command-line exec; detection now marks that install `launchable: false`, and Launcher shows recovery copy instead of a Launch button. | Keep the Codex WindowsApps regression. A future direct-launch path should require a supported PATH CLI or a proven desktop-specific launch bridge. |
|
||||
| T16-12 | Focused fixed | Windows hook-management command execution now resolves `npx` correctly. | A gated route smoke first failed because `/api/tools/hooks` returned HTTP 400 with empty stdout/stderr: `execFile('npx')` on Windows cannot resolve the npm shim, and a bare `npx.cmd` can pick the wrong shim under npm-started PATHs. `runHookCommand()` now prefers the `npx.cmd` beside `process.execPath`, and default exec capture uses the shared `.cmd` resolver. The broadened route smoke passes real install/verify/uninstall for all six hook-capable tools in an isolated profile. | Keep the gated route smoke in the release lane. |
|
||||
|
||||
## Persona Impact
|
||||
|
||||
| Persona | Current T16 cap | Why |
|
||||
|---|---:|---|
|
||||
| Engineer / power user | 8/10 | They can now trust packed-package command lifecycle, focused/rendered third-party launch behavior, Windows npm-shim launch handling, one real observed CLI launch, all six real hook route lifecycles, and the all-six rendered hook matrix. Packaged desktop status evidence and warning hygiene still cap trust. |
|
||||
| Solo founder | 8/10 | Hook setup edits personal AI-tool configs; packed lifecycle evidence plus Backup/Recovery, install pointer, manual-approval, uninstall cleanup, long-output, offline retry copy, one real safe launch, all six isolated route lifecycles, and all-six rendered transitions improve confidence. Remaining concern is packaged desktop status transitions. |
|
||||
| Team admin | 7/10 | Unsupported Claude Desktop messaging is now explicit, one real launch path is proven, all six hook-capable route lifecycles are proven, and all-six rendered state transitions are covered, but team rollout still needs packaged desktop evidence and quieter release output. |
|
||||
| Researcher | 8/10 | Less central, but memory capture trust depends on hooks failing open and reporting status clearly. |
|
||||
| Mobile executive | 8/10 | Less central, but launch/hook management still needs clear compact states if surfaced on smaller screens. |
|
||||
|
||||
## Acceptance For Closing T16
|
||||
|
||||
- Rendered Launcher evidence covers detected, not detected, installing, hooks active, verify success, verify failure, uninstall, all six hook-capable install/verify/uninstall transitions, sidecar offline, running observed output without duplicate replay, and unsupported Claude Desktop.
|
||||
- Command lifecycle evidence covers install, verify, and uninstall for all six hook-capable packages using the production-like invocation path.
|
||||
- `npx`/package-pack resolution is verified for every hook package that Launcher can invoke; registry-only proof is captured after actual publication.
|
||||
- Hook result UI exposes backup/pointer paths and manual trust steps without dumping raw logs as the primary UX.
|
||||
- Third-party adapter launch behavior has focused route/UI coverage and rendered non-built-in adapter proof.
|
||||
- Real detected CLI launch behavior has at least one safe observed smoke, and hook install/verify/uninstall has isolated route lifecycle evidence for all six hook-capable tools.
|
||||
- Hook/shim package-local test commands pass; any broader package-local command-shape gaps are tracked under T17.
|
||||
|
||||
## Packet Decision
|
||||
|
||||
Keep T16 as `Phase 2 Pending`. The rendered Launcher proof now includes all six hook-capable install/verify/uninstall transitions, packed-package `npx` lifecycle is locally proven, hook/shim package-local scripts now pass, observed output no longer reconnects after exit, component tests prove hook stdout/stderr details are not dropped, covered install output now renders Backup/Recovery labels instead of raw stdout, rendered standard install output shows changed-file and install-pointer labels, Verify `[FAIL]` output now renders as `Check failed` with manual-approval detail, uninstall output now renders restore/cleanup labels without implying install state, long hook output is summarized behind `More output`, offline detection now has rendered Retry recovery, Claude Desktop is explicitly launch-only, launchable third-party adapters are covered through detection/route/UI/rendered tests, the live Verify path no longer renders generic `HTTP 400` or bare empty-output exit codes, Windows npm shims are launchable/version-probed, Codex WindowsApps installs are blocked with recovery copy instead of a failing Launch button, one real detected CLI launch/output/exit/process-clear lifecycle is proven, all six real `/api/tools/hooks` install/verify/uninstall route lifecycles are proven in an isolated profile, the Tailwind motion-token ambiguity warnings are gone, and the `shape-selection.ts` dynamic/static import warning is gone. Packaged desktop hook-status transitions and remaining warning hygiene still block the final "complete UX, all parts functional" claim unless the user explicitly defers AI-tool hook lifecycle from the five-persona score.
|
||||
85
docs/audits/2026-07-08-analysis-completion-audit.md
Normal file
85
docs/audits/2026-07-08-analysis-completion-audit.md
Normal file
File diff suppressed because one or more lines are too long
222
docs/audits/2026-07-08-browser-companion-t19-analysis.md
Normal file
222
docs/audits/2026-07-08-browser-companion-t19-analysis.md
Normal file
@@ -0,0 +1,222 @@
|
||||
# Browser Companion T19 Analysis - 2026-07-08
|
||||
|
||||
Status: implementation supplement. T19 is narrowed, not fully closed.
|
||||
|
||||
Purpose: deepen T19 evidence for `apps/browser-ext`, the Chrome MV3 Browser Companion that saves pages and selections into Waggle memory.
|
||||
|
||||
## Sources Inspected
|
||||
|
||||
- `apps/browser-ext/manifest.json`
|
||||
- `apps/browser-ext/popup.html`
|
||||
- `apps/browser-ext/popup.js`
|
||||
- `apps/browser-ext/background.js`
|
||||
- `apps/browser-ext/content.js`
|
||||
- `apps/browser-ext/README.md`
|
||||
- `packages/server/src/local/routes/browser-ext.ts`
|
||||
- `packages/server/src/local/routes/memory.ts`
|
||||
- `packages/server/src/local/cors-config.ts`
|
||||
- `apps/web/src/components/os/settings/CoverageCompassCard.tsx`
|
||||
|
||||
Guideline baseline: Vercel Web Interface Guidelines, fetched 2026-07-08 from `https://raw.githubusercontent.com/vercel-labs/web-interface-guidelines/main/command.md`.
|
||||
|
||||
## Commands Run
|
||||
|
||||
```powershell
|
||||
node --check apps/browser-ext/popup.js
|
||||
node --check apps/browser-ext/background.js
|
||||
node --check apps/browser-ext/content.js
|
||||
node -e "<parse manifest and print required permissions/hosts>"
|
||||
npx tsc --noEmit --project packages/server/tsconfig.json
|
||||
Test-NetConnection 127.0.0.1 -Port 3333
|
||||
<one-off Playwright persistent Chromium load with --load-extension=apps/browser-ext>
|
||||
<fresh sidecar on 127.0.0.1:3333; direct /api/browser-ext and /api/memory/frames save-flow checks>
|
||||
<Playwright persistent Chromium unpacked extension background save with no allowlisted extension ID>
|
||||
<Playwright persistent Chromium unpacked extension background save with WAGGLE_BROWSER_EXT_IDS set to the extension ID>
|
||||
<Memory UI render check after Browser Companion save>
|
||||
node output/playwright/browser-companion-toolbar-3333/run-toolbar-popup-smoke.mjs
|
||||
npx tsx -e "<check WAGGLE_DEV_ALLOW_ANY_EXTENSION against concrete chrome-extension origin>"
|
||||
npx vitest run packages/server/tests/local/browser-ext-auth.test.ts packages/server/tests/local/network-auth.test.ts tests/browser-companion-background.test.ts
|
||||
npx eslint apps/browser-ext/background.js apps/browser-ext/popup.js apps/browser-ext/content.js packages/server/src/local/cors-config.ts packages/server/src/local/security-middleware.ts packages/server/src/local/routes/browser-ext.ts packages/server/tests/local/browser-ext-auth.test.ts packages/server/tests/local/network-auth.test.ts tests/browser-companion-background.test.ts --no-warn-ignored
|
||||
git diff --check
|
||||
node output/playwright/browser-companion-toolbar-3333/run-secure-default-live-smoke.mjs
|
||||
WAGGLE_T19_WEB_URL=http://127.0.0.1:34613 node output/playwright/browser-companion-toolbar-3333/run-popup-button-click-live-smoke.mjs
|
||||
npx tsx output/playwright/browser-companion-toolbar-3333/run-packaged-id-pairing-smoke.mjs
|
||||
```
|
||||
|
||||
## Command Results
|
||||
|
||||
| Check | Result | UX meaning |
|
||||
|---|---:|---|
|
||||
| Extension JS syntax | Pass | `popup.js`, `background.js`, and `content.js` parse. |
|
||||
| Manifest parse | Pass | MV3 manifest has `activeTab`, `storage`, `contextMenus`, localhost host permissions, `popup.html`, `background.js`, and one content script. |
|
||||
| Server typecheck | Pass | `packages/server` typechecks with the browser extension route and CORS code. |
|
||||
| `127.0.0.1:3333` listener | Not running | Disconnected popup state is the expected smoke state in this environment. |
|
||||
| Playwright unpacked-extension load | Partial pass | Chromium loaded the extension and rendered `popup.html`; screenshot captured at `output/playwright/browser-companion-disconnected-state.png`. |
|
||||
| Direct sidecar save contract | Pass | `POST /api/memory/frames?extract=false` with Browser Companion-shaped content saves frames, duplicate detection works, invalid source returns 400, and Memory UI renders the saved frames. |
|
||||
| Unallowlisted unpacked extension background save | Fail | With default env, `chrome.runtime.sendMessage({ type: 'save-memory' })` returns `{ saved: false, error: 'HTTP 500' }`; server logs `CORS: origin not allowed`. |
|
||||
| Prior allowlisted unpacked extension background save | Historical pass with caveat | Earlier artifact with `WAGGLE_BROWSER_EXT_IDS=ebcejdmgclnmaaghmhhcfelbpcmfebfm` returned `{ saved: true, frameId: 1 }`, duplicate returned `{ duplicate: true }`, and Memory UI rendered the imported frame; the current default-auth toolbar probe below shows CORS allowlisting alone is not sufficient under the bearer-token security model. |
|
||||
| Historical search provenance mismatch | Fixed | Earlier `GET /api/memory/frames` showed `source: import` while `/api/memory/search` reported the same frame as `source: user_stated`. The route now rehydrates frame provenance after `MultiMind` replaces `source` with the mind label, and the popup button-click live smoke confirms both selection/page search results return `source: import`. |
|
||||
| Toolbar popup open over normal page | Evidence blocker | The probe loaded the extension, selected text in a normal HTTP page, and `chrome.action.openPopup()` returned success, but Playwright never observed a `chrome-extension://<id>/popup.html` page. Real toolbar-click evidence still needs a different automation path or a manual/recorded protocol. |
|
||||
| Pre-fix paired extension save under default auth | Fail | With `WAGGLE_BROWSER_EXT_IDS=<extension id>` and default auth, the content script extracted the selected text and page body, but extension-origin save returned `401 MISSING_TOKEN`; artifact: `output/playwright/browser-companion-toolbar-3333/toolbar-popup-summary-allowlisted-current-auth.json`. |
|
||||
| Legacy localhost-trust extraction/save | Pass with caveat | With `WAGGLE_TRUST_LOCALHOST=1`, the same content-script extraction saved an imported frame with `source: import`; artifact: `output/playwright/browser-companion-toolbar-3333/toolbar-popup-summary-trust-localhost.json`. This proves extraction/save mechanics, not the secure default UX. |
|
||||
| Pre-fix dev allow-any extension CORS check | Fail | `WAGGLE_DEV_ALLOW_ANY_EXTENSION=1` added `chrome-extension://` to allowed origins, but exact-match CORS returned `false` for a concrete `chrome-extension://ebcejdmgclnmaaghmhhcfelbpcmfebfm` origin. |
|
||||
| Focused extension auth bootstrap regression tests | Pass | `packages/server/tests/local/browser-ext-auth.test.ts`, `packages/server/tests/local/network-auth.test.ts`, and `tests/browser-companion-background.test.ts` now pass 39/39, including the explicit `activeWorkspaceId` health contract. |
|
||||
| Extension syntax after pairing patch | Pass | `node --check` passes for `background.js`, `popup.js`, and `content.js`. |
|
||||
| Focused lint and diff hygiene | Pass | Focused ESLint is clean; `git diff --check` exits 0, with only existing CRLF warnings from Git. |
|
||||
| Secure-default loaded-extension smoke | Pass with known gaps | Fresh sidecar, default bearer auth, `WAGGLE_BROWSER_EXT_IDS=ebcejdmgclnmaaghmhhcfelbpcmfebfm`, and unpacked extension pass `output/playwright/browser-companion-toolbar-3333/run-secure-default-live-smoke.mjs`. The extension service worker omits `Origin` and sends `sec-fetch-site: none`; `background.js` sends `X-Waggle-Extension-Id`, token bootstrap succeeds, the token is stored during save, content-script selection is saved through `chrome.runtime.sendMessage({ type: 'save-memory' })`, and `/api/memory/frames` returns the imported frame. Toolbar-popup page exposure remains a known gap. |
|
||||
| Direct popup keyboard/click/restricted-state smoke | Pass with caveat | `output/playwright/browser-companion-toolbar-3333/run-popup-button-click-live-smoke.mjs` opens the popup document with a test shim for the target tab that Chrome normally supplies to a toolbar popup, tabs through Save selection, Save page, and Open Waggle, captures a visible focus ring, presses Enter on Save selection, clicks Save page, receives saved toasts, confirms both frames through `/api/memory/frames`, confirms `/api/memory/search` returns both captures as `source: import`, and, when `WAGGLE_T19_WEB_URL` is set, confirms the rendered `/memory` app shows both saved captures with the `imported` provenance chip and no Clerk/CSP/page errors. The same smoke now opens a restricted `chrome://` tab and proves the popup stays connected, labels the destination as `Personal memory`, disables both save buttons with non-primary styling, and shows the persistent "normal webpage" recovery copy. This proves popup keyboard/focus basics, disabled/restricted-page UX, popup save wiring, secure save effects, Memory search provenance, and rendered Memory UI visibility, not native toolbar-bubble exposure. |
|
||||
| Stable packaged-ID pairing smoke | Pass with caveat | `output/playwright/browser-companion-toolbar-3333/run-packaged-id-pairing-smoke.mjs` creates a temporary extension copy with a generated manifest public key, derives the Chrome extension ID, starts an isolated sidecar with `WAGGLE_BROWSER_EXT_IDS=<derived-id>`, proves the loaded service worker URL uses the same stable ID, fetches `/api/browser-ext/session-token` from `chrome-extension://<id>`, saves selected page text through the background pairing path, stores the token in `chrome.storage.local`, confirms `/api/memory/frames`, and confirms `/api/memory/search` preserves `source: import`. This proves production-shaped stable-ID pairing semantics, not a signed Web Store or installer-distributed package. |
|
||||
| Agent catch-up recall provenance regression | Pass | `packages/agent/tests/orchestrator-recall-hardening.test.ts` now proves an imported workspace memory returned through `Orchestrator.recallMemory('catch me up')` carries `recalledFrames[].source === 'import'` and does not degrade to `unknown`. This covers the existing chat `auto_recall` catch-up provenance shape; future recall result shapes still need their own evidence if added to judging. |
|
||||
|
||||
## 2026-07-09 Implementation Update
|
||||
|
||||
Implemented:
|
||||
|
||||
- Added `GET /api/browser-ext/session-token`, auth-exempt only for bootstrap and gated by a valid allowlisted Browser Companion extension origin.
|
||||
- Fixed Browser Companion CORS matching so `WAGGLE_BROWSER_EXT_IDS=<id>` and `WAGGLE_DEV_ALLOW_ANY_EXTENSION=1` work with concrete `chrome-extension://<id>` origins.
|
||||
- Fixed the MV3 service-worker no-`Origin` case: `background.js` now sends `X-Waggle-Extension-Id`, and the token route accepts it only with `sec-fetch-site: none` and an allowlisted extension ID.
|
||||
- Updated `background.js` to fetch and store the session token before health/save requests, and retry once after a 401.
|
||||
- Mapped missing/expired/not-allowlisted pairing failures to actionable extension copy instead of raw `HTTP 401` / `MISSING_TOKEN`.
|
||||
- Added popup `role="status"` / `aria-live="polite"`, sticky recovery messages, and a visible restricted-page explanation when the content script cannot run.
|
||||
- Made disabled primary actions visibly inactive, and changed the popup destination label to `Memory destination` with honest id/fallback copy instead of presenting an id as a workspace name.
|
||||
- Added context-menu handler regression coverage for registration, selected-text save payload, and success badge feedback.
|
||||
- Fixed `/api/memory/search` provenance for imported frames by rehydrating the DB frame source after `MultiMind` replaces `source` with the mind label.
|
||||
- Added a stable packaged-ID pairing smoke that generates a temporary manifest key, derives the Chrome extension ID, starts the sidecar with that ID allowlisted, and proves token bootstrap/save/search provenance through the production-shaped extension-ID pairing path.
|
||||
- Fixed agent catch-up recall provenance by selecting and carrying `frame.source` through `fetchRecentFrames()` and the workspace catch-up branch in `Orchestrator.recallMemory()`.
|
||||
|
||||
Focused verification:
|
||||
|
||||
- `npx vitest run packages/server/tests/local/browser-ext-auth.test.ts packages/server/tests/local/network-auth.test.ts tests/browser-companion-background.test.ts` -> pass, 39/39, including the explicit `activeWorkspaceId` health contract.
|
||||
- `npx vitest run packages/server/tests/local-mode.test.ts` -> pass, 21/21, including the imported-frame search provenance regression.
|
||||
- `node --check apps/browser-ext/background.js; node --check apps/browser-ext/popup.js; node --check apps/browser-ext/content.js` -> pass.
|
||||
- `npx tsc --noEmit --project packages/server/tsconfig.json` -> pass.
|
||||
- Focused ESLint for touched extension/server/test files -> pass.
|
||||
- `git diff --check` -> pass.
|
||||
- `node output/playwright/browser-companion-toolbar-3333/run-secure-default-live-smoke.mjs` -> pass for extension load, content extraction, token bootstrap during save, token storage, background save, and `/api/memory/frames` imported-frame confirmation.
|
||||
- `node output/playwright/browser-companion-toolbar-3333/run-popup-button-click-live-smoke.mjs` -> pass for extension load, normal-page selection, popup Tab order (`save-selection`, `save-page`, `open-waggle`), visible Save selection focus ring, Enter-to-save selection, Save page click, saved toasts, token route, `/api/memory/frames` confirmation, and `/api/memory/search` `source: import` confirmation for both captures.
|
||||
- `WAGGLE_T19_WEB_URL=http://127.0.0.1:34613 node output/playwright/browser-companion-toolbar-3333/run-popup-button-click-live-smoke.mjs` -> pass for secure popup keyboard Save selection and Save page click, `/api/memory/frames`, `/api/memory/search` `source: import`, rendered `/memory` confirmation that both markers and the `imported` provenance chip are visible with no Clerk/CSP/page errors, and a restricted-page popup state with both save buttons disabled, non-primary disabled styling, `Personal memory` destination copy, and persistent normal-webpage recovery text.
|
||||
- `npx tsx output/playwright/browser-companion-toolbar-3333/run-packaged-id-pairing-smoke.mjs` -> pass for generated stable extension ID, isolated sidecar allowlisting, loaded service worker ID match, `chrome-extension://<id>` token bootstrap, content-script extraction, background save, token storage, `/api/memory/frames` confirmation, and `/api/memory/search` `source: import`.
|
||||
- `npx vitest run packages/agent/tests/orchestrator-recall-hardening.test.ts` -> pass, 15/15, including imported workspace provenance in catch-up `recalledFrames`.
|
||||
|
||||
Remaining T19 scope:
|
||||
|
||||
- Native toolbar-bubble exposure proof while a normal page remains active; Playwright still does not expose the popup as a page after `chrome.action.openPopup()`. Direct popup-document keyboard/click behavior is now proven with an active-tab shim.
|
||||
- Actual native context-menu click proof, or explicit deferral. The registration and click handler are now regression-covered.
|
||||
- Any future recall result shape outside `/api/memory/search` and the existing chat `auto_recall`/catch-up `recalledFrames` path, if it is included in judge scoring.
|
||||
- Signed Web Store/installer-distributed extension evidence, if release packaging itself enters the score. Stable extension-ID pairing against the sidecar is now proven.
|
||||
|
||||
Playwright disconnected-state data:
|
||||
|
||||
```json
|
||||
{
|
||||
"status": "Not connected",
|
||||
"workspace": "-",
|
||||
"toast": "Start Waggle desktop on this machine, then re-open this popup.",
|
||||
"toastClass": "err",
|
||||
"saveSelectionDisabled": true,
|
||||
"savePageDisabled": true
|
||||
}
|
||||
```
|
||||
|
||||
Important limitation: opening `chrome-extension://<id>/popup.html` as a tab is not identical to clicking the toolbar popup over a normal web page. The new popup-button smoke patches only the active-tab lookup so the popup reads the target tab that the native toolbar bubble would receive from Chrome. This is strong evidence for popup button wiring and save effects, but still not proof that Playwright can observe the native toolbar bubble itself.
|
||||
|
||||
Additional live save-flow artifacts:
|
||||
|
||||
- `output/playwright/browser-companion-save-3333/summary.json`: direct sidecar health/save/duplicate/search checks and Memory UI screenshot after direct save.
|
||||
- `output/playwright/browser-companion-save-3333/extension-background-summary.json`: unpacked extension background save without extension ID allowlist; save fails with `HTTP 500`.
|
||||
- `output/playwright/browser-companion-save-3333/extension-background-allowlisted-summary.json`: unpacked extension background save with the detected extension ID allowlisted; save and duplicate pass.
|
||||
- `output/playwright/browser-companion-save-3333/allowlisted-memory-ui-summary.json`: `GET /api/memory/frames`, `/api/memory/search`, and rendered Memory UI after the allowlisted extension save.
|
||||
- `output/playwright/browser-companion-toolbar-3333/run-toolbar-popup-smoke.mjs`: one-off current toolbar/extraction probe script.
|
||||
- `output/playwright/browser-companion-toolbar-3333/toolbar-popup-summary-allowlisted-current-auth.json`: pre-fix paired extension attempt; content extraction succeeds, save fails with `401 MISSING_TOKEN`.
|
||||
- `output/playwright/browser-companion-toolbar-3333/toolbar-popup-summary-trust-localhost.json`: legacy-trust attempt; content extraction succeeds and saves an imported frame.
|
||||
- `output/playwright/browser-companion-toolbar-3333/run-secure-default-live-smoke.mjs`: post-fix secure-default loaded-extension smoke.
|
||||
- `output/playwright/browser-companion-toolbar-3333/secure-default-live-summary.json`: post-fix evidence; extension load, content extraction, MV3 no-Origin header probe, token bootstrap during save, background save, and `/api/memory/frames` imported-frame confirmation pass. Toolbar popup exposure and `/api/memory/search` are recorded as known gaps.
|
||||
- `output/playwright/browser-companion-toolbar-3333/run-popup-button-click-live-smoke.mjs`: direct popup-document keyboard/click smoke with active-tab shim.
|
||||
- `output/playwright/browser-companion-toolbar-3333/popup-button-click-live-summary.json`: post-fix evidence; Tab order reaches Save selection, Save page, and Open Waggle, Save selection has a visible focus ring and saves via Enter, Save page clicks through, both flows show saved toasts, create frames through the secure sidecar path, return from `/api/memory/search` as `source: import`, and, in the latest run with `WAGGLE_T19_WEB_URL`, render both captures in `/memory` with `imported` provenance and no Clerk/CSP/page errors. The same run proves a restricted-page popup state with both save buttons disabled, non-primary disabled styling, `Personal memory` destination copy, and persistent normal-webpage recovery text.
|
||||
- `output/playwright/browser-companion-toolbar-3333/run-packaged-id-pairing-smoke.mjs`: generated-key stable extension-ID pairing smoke.
|
||||
- `output/playwright/browser-companion-toolbar-3333/packaged-id-pairing-summary.json`: post-fix evidence; generated stable ID `bcbhhonimhnecnfoacokibkbedoigbpc`, service worker URL ID match, sidecar allowlisting, token bootstrap, background save, token storage, `/api/memory/frames`, and `/api/memory/search` `source: import` all pass.
|
||||
- Screenshots:
|
||||
- `output/playwright/browser-companion-save-3333/extension-popup-connected-tab.png`
|
||||
- `output/playwright/browser-companion-save-3333/extension-popup-allowlisted-connected-tab.png`
|
||||
- `output/playwright/browser-companion-save-3333/memory-after-extension-save.png`
|
||||
- `output/playwright/browser-companion-save-3333/memory-after-allowlisted-extension-save.png`
|
||||
- `output/playwright/browser-companion-toolbar-3333/toolbar-active-page-selection.png`
|
||||
- `output/playwright/browser-companion-toolbar-3333/popup-button-click-keyboard-focus.png`
|
||||
- `output/playwright/browser-companion-toolbar-3333/popup-button-click-target-selection.png`
|
||||
- `output/playwright/browser-companion-toolbar-3333/popup-button-click-save-selection.png`
|
||||
- `output/playwright/browser-companion-toolbar-3333/popup-button-click-save-page.png`
|
||||
- `output/playwright/browser-companion-toolbar-3333/popup-button-click-memory-ui.png`
|
||||
- `output/playwright/browser-companion-toolbar-3333/popup-restricted-disabled-state.png`
|
||||
|
||||
## What Is Proven Now
|
||||
|
||||
- The extension files exist at `apps/browser-ext/*`; there is no `apps/browser-ext/src` directory.
|
||||
- The manifest is structurally valid and requests the expected MV3 capabilities.
|
||||
- The extension can be loaded unpacked by Chromium in this environment.
|
||||
- The disconnected status can render with an error toast and disabled save buttons.
|
||||
- The server-side route/CORS code typechecks.
|
||||
- The direct write API exists at `POST /api/memory/frames` and accepts `content`, optional `workspace`/`workspaceId`, `importance`, and `source`; it sanitizes content, validates `source`, deduplicates, and stores a frame.
|
||||
- The direct write API can persist Browser Companion-shaped selection/page content, and the Memory UI can render the result.
|
||||
- The content script can extract selected text and page body from a normal HTTP page.
|
||||
- Focused tests prove the secure default token-bootstrap path: an allowlisted extension origin can fetch the session token, `background.js` stores it, and subsequent save calls send `Authorization: Bearer <token>`.
|
||||
- Chromium MV3 service-worker fetches to localhost omit `Origin` and send `sec-fetch-site: none`; this is now covered by tests and the live smoke.
|
||||
- With an allowlisted extension ID and default auth, the loaded extension now extracts selected page content, bootstraps/stores a token during save, saves via `background.js`, and `/api/memory/frames` returns the imported frame.
|
||||
- The popup Tab order reaches Save selection, Save page, and Open Waggle; Save selection has a visible focus ring and saves by keyboard Enter; Save page clicks through the popup UI, shows a saved toast, and creates a frame when the active tab is supplied by the live smoke shim.
|
||||
- Restricted-page/content-script-unavailable state now keeps the popup connected, labels the destination honestly as `Personal memory`, disables both save buttons, uses non-primary disabled styling, and shows persistent recovery copy.
|
||||
- Browser Companion selection/page captures now preserve `source: import` in `/api/memory/search`, matching `/api/memory/frames`.
|
||||
- The rendered `/memory` app now shows the secure popup-saved selection/page captures and the `imported` provenance chip in the same live smoke run.
|
||||
- The context-menu registration and selected-text handler are covered in `tests/browser-companion-background.test.ts`.
|
||||
- Stable extension-ID pairing is proven with a temporary generated manifest key: the derived ID matches Chromium's loaded service worker ID, the sidecar accepts that ID via `WAGGLE_BROWSER_EXT_IDS`, the extension stores the token, saves selected content, and search provenance remains `source: import`.
|
||||
- Agent catch-up `auto_recall` now preserves imported workspace provenance in `recalledFrames`, allowing the server chat stream to emit honest provenance instead of suppressing it as `unknown`.
|
||||
- With an allowlisted extension ID but default auth, the pre-fix toolbar probe failed as `401 MISSING_TOKEN`; the code path is now fixed in focused server/background tests, the secure-default loaded-extension smoke, and the direct popup keyboard/click smoke.
|
||||
|
||||
## Still Not Proven
|
||||
|
||||
- Native toolbar-bubble behavior while a normal web page is the active tab. Playwright still needs a reliable toolbar-popup protocol or manual/recorded release evidence.
|
||||
- Context-menu save via an actual browser context-menu click. Registration and handler behavior are covered, but the native browser menu item itself has not been clicked in an automated browser.
|
||||
- Signed Web Store/installer-distributed extension behavior, if release packaging itself enters scoring. Stable extension-ID pairing is covered by `packaged-id-pairing-summary.json`.
|
||||
- Live dev escape hatch behavior. Focused CORS tests cover concrete `chrome-extension://<id>` origins; the live smoke used the production-shaped extension-ID allowlist.
|
||||
- CORS/auth-denied recovery UX screenshot with a real extension origin and sidecar, if this state enters judge scoring.
|
||||
- Screen-reader announcement of status/toast changes.
|
||||
- Packaged desktop/sidecar port behavior.
|
||||
|
||||
## Line-Level Findings
|
||||
|
||||
| ID | Finding | Evidence | Correction |
|
||||
|---|---|---|---|
|
||||
| T19-1 | Coverage Compass claimed browser extensions were `covered` without enough end-to-end behavior evidence. | `apps/web/src/components/os/settings/CoverageCompassCard.tsx:31` | Fixed 2026-07-10: Browser AI extensions now render as `partial` with explicit popup/capture coverage and native toolbar/context-menu work still pending; `CoverageCompassCard.test.tsx` guards the honest state. |
|
||||
| T19-2 | Toast/status updates are visual only; no live region or alert role is present. | `apps/browser-ext/popup.html:69`, `apps/browser-ext/popup.html:84`, `apps/browser-ext/popup.js:19` | Fixed 2026-07-09: popup toast now has `role="status"` and `aria-live="polite"`. |
|
||||
| T19-3 | Restricted-page/content-script-unavailable state disables save controls without a visible explanation. | `apps/browser-ext/popup.js:56` to `apps/browser-ext/popup.js:59` | Fixed 2026-07-09: content-script failures show persistent "normal webpage" recovery copy. |
|
||||
| T19-4 | Disconnected recovery toast auto-clears after 3.5 seconds. | `apps/browser-ext/popup.js:23`, disconnected smoke | Fixed 2026-07-09: health/setup errors are sticky while the popup remains open. |
|
||||
| T19-5 | Disabled primary action could still read as visually primary in the popup. | `apps/browser-ext/popup.html:46`, `apps/browser-ext/popup.html:73`, screenshot `output/playwright/browser-companion-disconnected-state.png`; latest evidence `output/playwright/browser-companion-toolbar-3333/popup-restricted-disabled-state.png` and `popup-button-click-live-summary.json` | Fixed 2026-07-09: disabled primary buttons use muted non-primary styling, opacity stays readable at `1`, and the restricted-page live smoke proves the style is not honey-primary. |
|
||||
| T19-6 | Health endpoint returns an active workspace id, while popup labels it as the memory destination. | `packages/server/src/local/routes/browser-ext.ts`, `apps/browser-ext/popup.js`, `packages/server/tests/local/browser-ext-auth.test.ts`, `popup-button-click-live-summary.json` | Fixed 2026-07-09: health now exposes `activeWorkspaceId` explicitly while preserving legacy `activeWorkspace`, and the popup labels the value as `Workspace id: ...` or `Personal memory` instead of implying a friendly workspace name. |
|
||||
| T19-7 | Full save result is now verified for the secure-default background path and direct popup keyboard/click path, including rendered Memory UI confirmation, but not for the native toolbar bubble itself. | `apps/browser-ext/background.js`, `packages/server/src/local/routes/memory.ts`, `output/playwright/browser-companion-toolbar-3333/secure-default-live-summary.json`, `output/playwright/browser-companion-toolbar-3333/popup-button-click-live-summary.json`, `output/playwright/browser-companion-toolbar-3333/popup-button-click-keyboard-focus.png`, `output/playwright/browser-companion-toolbar-3333/popup-button-click-memory-ui.png` | Add a reliable native toolbar-bubble protocol or manual release evidence for the browser-owned toolbar popup. |
|
||||
| T19-8 | Unallowlisted extension origins previously failed as generic 500s, not a recoverable setup state. | `packages/server/src/local/index.ts:2197`, `packages/server/src/local/index.ts:2201`, `apps/browser-ext/background.js:45`, `output/playwright/browser-companion-save-3333/extension-background-summary.json` | Focused fixed 2026-07-09: token bootstrap returns an intentional setup denial and background maps it to sticky setup copy. Remaining: live unallowlisted-extension screenshot if this state enters judging. |
|
||||
| T19-9 | Memory search provenance could disagree with direct frame provenance. | `output/playwright/browser-companion-save-3333/allowlisted-memory-ui-summary.json` showed `/api/memory/frames` as `import` while `/api/memory/search` reported `user_stated`; red regression reproduced the mismatch. | Fixed 2026-07-09: `/api/memory/search` rehydrates DB frame provenance, `local-mode.test.ts` covers imported-frame search provenance, and `popup-button-click-live-summary.json` confirms selection/page captures return as `source: import`. |
|
||||
| T19-10 | Extension-ID CORS allowlisting was not enough under default bearer auth. | `packages/server/src/local/security-middleware.ts:312` to `packages/server/src/local/security-middleware.ts:387`; `apps/browser-ext/background.js:14` to `apps/browser-ext/background.js:15`; pre-fix artifact shows `401 MISSING_TOKEN`. | Fixed 2026-07-09 with `/api/browser-ext/session-token`, MV3 no-Origin header handling, background token storage, 39/39 focused tests, secure-default loaded-extension smoke, and direct popup keyboard/click smoke. Remaining: native toolbar-bubble exposure evidence. |
|
||||
| T19-11 | README dev setup says `WAGGLE_DEV_ALLOW_ANY_EXTENSION=1` accepts any extension, but pre-fix CORS exact-match logic did not accept concrete extension origins. | `apps/browser-ext/README.md:19` to `apps/browser-ext/README.md:26`; `packages/server/src/local/cors-config.ts:38` to `packages/server/src/local/cors-config.ts:57`; pre-fix `npx tsx` check returned `concreteOriginAllowed: false`. | Fixed 2026-07-09 in `browserExtensionOriginAllowed()` and covered by focused CORS tests. |
|
||||
| T19-12 | Toolbar-popup automation still cannot expose the native toolbar bubble over a normal page. | `output/playwright/browser-companion-toolbar-3333/toolbar-popup-summary-allowlisted-current-auth.json` shows `chrome.action.openPopup()` returned success but no popup page was exposed to Playwright; `popup-button-click-live-summary.json` proves the keyboard/click behavior through a direct popup-document fallback. | Add a reliable native toolbar-bubble test protocol, manual release checklist with screenshots/video, or an alternate browser automation route that keeps the normal page active without a shim. |
|
||||
| T19-13 | Popup keyboard/focus basics lacked live evidence and an explicit focus ring. | `apps/browser-ext/popup.html`, `output/playwright/browser-companion-toolbar-3333/popup-button-click-keyboard-focus.png`, `output/playwright/browser-companion-toolbar-3333/popup-button-click-live-summary.json` | Fixed 2026-07-09: popup buttons now have a visible `:focus-visible` ring, and the live smoke proves Tab order, focused Save selection geometry/style, and Enter-to-save behavior. |
|
||||
| T19-14 | Production-shaped packaged pairing lacked stable extension-ID evidence. | `apps/browser-ext/manifest.json` has no release key, and earlier smokes only used the unpacked extension ID from the source folder. | Fixed 2026-07-09: `run-packaged-id-pairing-smoke.mjs` generates a temporary manifest key, derives the stable Chrome extension ID, starts the sidecar with that ID allowlisted, verifies the loaded service worker ID matches, and proves token bootstrap/save/search provenance through the stable-ID path. |
|
||||
| T19-15 | Agent catch-up recall could suppress provenance because workspace catch-up rows did not carry `source`, so imported captures surfaced as `unknown` in `recalledFrames`. | `packages/agent/src/orchestrator.ts`, `packages/agent/src/context-loader.ts`, `packages/server/src/local/routes/chat.ts`, red-to-green `packages/agent/tests/orchestrator-recall-hardening.test.ts` | Fixed 2026-07-09: catch-up recent/important frames now select and propagate `source`, and the regression proves imported workspace memories return `source: import` rather than `unknown`. |
|
||||
|
||||
## T19 Acceptance
|
||||
|
||||
T19 remains open until either:
|
||||
|
||||
1. Browser Companion is explicitly deferred from the five-persona score, or
|
||||
2. Evidence proves all of the following:
|
||||
|
||||
- Unpacked or packaged extension loads.
|
||||
- Connected and disconnected states render with persistent, accessible recovery.
|
||||
- Save selection and save page succeed against a running sidecar under the default secure auth model, without relying on `WAGGLE_TRUST_LOCALHOST=1`.
|
||||
- Extension pairing stores or supplies a valid bearer token, or a deliberate reviewed auth exemption exists for the extension route.
|
||||
- Native context-menu save succeeds or is explicitly scoped out; registration and handler behavior are already covered.
|
||||
- CORS/auth-denied, missing-token, and extension-ID-missing states are understandable and do not appear as generic `HTTP 500` or raw `MISSING_TOKEN`.
|
||||
- Memory UI shows the captured frame with source/provenance that a Researcher can understand.
|
||||
- Memory search preserves the same provenance shown by the frame list, and the existing chat `auto_recall`/catch-up `recalledFrames` path preserves imported workspace provenance; any future recall result shape must do the same if included in judging.
|
||||
- Popup keyboard/focus basics pass; status/toast live-region markup exists, with screen-reader announcement proof still required if judged separately.
|
||||
|
||||
## Phase Impact
|
||||
|
||||
This does not change Phase 1. T19 remains a Phase 2/Launch final-product gate after in-app P0 blockers are cleared, unless the user explicitly asks to include Browser Companion work in Phase 1.
|
||||
1769
docs/audits/2026-07-08-complete-ux-usage-audit.md
Normal file
1769
docs/audits/2026-07-08-complete-ux-usage-audit.md
Normal file
File diff suppressed because it is too large
Load Diff
193
docs/audits/2026-07-08-desktop-wrapper-t14-analysis.md
Normal file
193
docs/audits/2026-07-08-desktop-wrapper-t14-analysis.md
Normal file
@@ -0,0 +1,193 @@
|
||||
# Desktop Wrapper T14 Analysis - 2026-07-08
|
||||
|
||||
Status: analysis supplement plus focused release-workflow, tray, packaged-startup, and sidecar hardening.
|
||||
|
||||
Purpose: deepen T14 evidence for the Tauri desktop wrapper, installer/update path, tray behavior, sidecar startup, and native-to-web UX bridge.
|
||||
|
||||
## Sources Inspected
|
||||
|
||||
- `app/src-tauri/tauri.conf.json`
|
||||
- `app/src-tauri/capabilities/default.json`
|
||||
- `app/src-tauri/src/lib.rs`
|
||||
- `app/src-tauri/src/tray.rs`
|
||||
- `app/src-tauri/src/service.rs`
|
||||
- `app/src-tauri/Cargo.toml`
|
||||
- `app/package.json`
|
||||
- `app/tests/auto-update.test.ts`
|
||||
- `app/scripts/*.test.ts`
|
||||
- `apps/web/src/App.tsx`
|
||||
- `apps/web/src/lib/tauri-bindings.ts`
|
||||
- `apps/web/src/providers/ServiceProvider.tsx`
|
||||
- `.github/workflows/release.yml`
|
||||
- `scripts/check-sidecar-resources.mjs`
|
||||
- `scripts/build-sidecar.mjs`
|
||||
- `scripts/bundle-node.mjs`
|
||||
- `scripts/bundle-native-deps.mjs`
|
||||
- `scripts/stage-sidecar-deps.mjs`
|
||||
|
||||
Guideline baseline: Vercel Web Interface Guidelines, fetched 2026-07-08 from `https://raw.githubusercontent.com/vercel-labs/web-interface-guidelines/main/command.md`.
|
||||
|
||||
## Commands Run
|
||||
|
||||
```powershell
|
||||
npx tsc --noEmit --project app/tsconfig.json
|
||||
npx vitest run app/tests/auto-update.test.ts app/scripts/signing-config.test.ts app/scripts/installer-config.test.ts app/scripts/bundle-runtimes.test.ts --reporter=dot
|
||||
node scripts/check-sidecar-resources.mjs
|
||||
cargo check --manifest-path app/src-tauri/Cargo.toml
|
||||
npm run test -- tauri-bindings.test.ts adapter.tauri-branch.test.ts --reporter=dot
|
||||
npx vitest run app/tests/e2e/startup.test.ts app/tests/e2e/chat.test.ts app/tests/e2e/workspaces.test.ts --reporter=dot
|
||||
node -e "<parse tauri config/capability JSON files>"
|
||||
app/src-tauri/resources/node.exe --version
|
||||
npx vitest run packages/server/tests/tauri-config.test.ts --reporter=dot
|
||||
npm run test -w apps/web -- src/lib/tauri-bindings.test.ts --reporter=dot
|
||||
npx tsc --noEmit --project apps/web/tsconfig.json
|
||||
npm run build:packages
|
||||
npm run tauri:build:local
|
||||
npx vitest run packages/server/tests/tauri-config.test.ts app/tests/auto-update.test.ts --reporter=dot
|
||||
cargo test --manifest-path app/src-tauri/Cargo.toml service_script -- --nocapture
|
||||
npx vitest run packages/server/tests/local/network-auth.test.ts --reporter=dot
|
||||
node scripts/bundle-node.mjs
|
||||
npx tauri build --debug --no-bundle
|
||||
Start-Process app/src-tauri/target/debug/waggle.exe with isolated WAGGLE_DATA_DIR; poll http://127.0.0.1:3333/health
|
||||
```
|
||||
|
||||
## Command Results
|
||||
|
||||
| Check | Result | UX meaning |
|
||||
|---|---:|---|
|
||||
| `app` TypeScript | Pass | Desktop build/signing/installer scripts typecheck. Refreshed in continuation run. |
|
||||
| Tauri helper/static tests | Pass, 4 files / 85 tests | Auto-update config, signing config, installer config, and runtime-bundling helpers are covered. |
|
||||
| Sidecar resource preflight | Pass | Local resources include Node runtime, native deps, and staged `node_modules`; `service.js` exists. Refreshed in continuation run. |
|
||||
| Rust `cargo check` | Pass | Tauri Rust shell compiles in dev profile, including the tray native quit/settings changes. Refreshed after tray hardening. |
|
||||
| Web-side Tauri binding tests from `apps/web` | Pass, focused binding 17/17 and prior binding/adapter bundle 19/19 | IPC command wrappers, adapter Tauri branches, the desktop navigation bridge, and native service/update notice mapping are covered when run through the web package. Output includes `punycode` deprecation warnings. Refreshed after service/update event surfacing. |
|
||||
| Root Vitest command for web-side Tauri tests | Fails by command shape | Root Vitest excludes `apps/**`; this is a verification-documentation gap, not a product runtime failure. |
|
||||
| App service E2E | Pass, 3 files / 11 tests | Service startup, health, settings persistence, chat SSE, workspace/session, and memory-scope API flows work in the Vitest harness. Output is noisy with mock-embedding and unknown-model cost warnings. |
|
||||
| JSON parse | Pass | Tauri config, build/dev overrides, and capability JSON parse. |
|
||||
| Bundled Node runtime | Pass | `app/src-tauri/resources/node.exe -p "process.version + ' abi=' + process.versions.modules"` returns `v22.22.2 abi=127` on this local build, matching the Node ABI that staged native deps. `bundle-node.mjs` now defaults to the current staging Node version, with `WAGGLE_BUNDLED_NODE_VERSION` available for an intentional pin. |
|
||||
| Release workflow/tray/static desktop bridge guard | Pass, 1 file / 25 tests | `packages/server/tests/tauri-config.test.ts` now requires release builds to run `npm run build:packages` before each desktop sidecar bundle point, guards the tray menu against unsupported Pause/About/Quit-event actions, requires the web app to mount both desktop navigation and shell-event listeners, and guards updater-disabled startup plus Node ABI preflight behavior. |
|
||||
| Web app TypeScript | Pass | `npm run typecheck:web` confirms the App-level desktop event bridge and Tauri binding types compile. |
|
||||
| Updater-disabled static contract | Pass, 2 files / 35 tests | `tauri.conf.json` has updater config absent, updater capability is not exposed, and Rust no longer initializes `tauri_plugin_updater` while signed updater artifacts are missing. |
|
||||
| Packaged debug binary build without installer | Pass | `npx tauri build --debug --no-bundle` builds `app/src-tauri/target/debug/waggle.exe` after sidecar/web resource preflight. |
|
||||
| Full local debug bundle/MSI/NSIS | Pass locally | `npm run tauri:build:local --prefix app` completed and produced `Waggle_0.2.0_x64_en-US.msi` (150,135,110 bytes, 2026-07-10 18:59) and `Waggle_0.2.0_x64-setup.exe` (97,806,701 bytes, 2026-07-10 19:08). Both local debug artifacts are currently unsigned. |
|
||||
| Packaged debug startup smoke | Pass | Launching `target/debug/waggle.exe` with an isolated `WAGGLE_DATA_DIR` reaches `/health` with `status: ok`, `mode: local`, `database.healthy: true`, `serviceHealth.watchdogRunning: true`; captured stderr contains no `CORS: origin not allowed`, `ERR_MODULE_NOT_FOUND`, or `NODE_MODULE_VERSION` errors. |
|
||||
|
||||
## Native Event Consumer Matrix
|
||||
|
||||
The tray-specific false affordances found in the first T14 pass are now narrowed:
|
||||
Pause Agents and About Waggle are no longer offered from the tray, Settings has a
|
||||
web bridge to the shipped `/settings` route, and Quit uses Tauri's native
|
||||
`app.exit(0)` path. The updater is intentionally disabled until signed updater
|
||||
artifacts exist, so update availability is future-ready in the web mapper but
|
||||
not emitted by the native shell in the current build. Remaining active native
|
||||
events are service-watchdog signals, not tray menu actions.
|
||||
|
||||
```text
|
||||
waggle://navigate
|
||||
waggle://service-restart-needed
|
||||
waggle://service-status
|
||||
```
|
||||
|
||||
| Native event / behavior | Source | Current consumer evidence | UX reading |
|
||||
|---|---|---|---|
|
||||
| Tray icon click / Open Waggle | `app/src-tauri/src/tray.rs` | Native show/focus; no React consumer needed | Source-wired, still needs packaged smoke. |
|
||||
| Close window to tray | `app/src-tauri/src/lib.rs:135-140` | Native hide on close; no React consumer needed | Source-wired, still needs packaged smoke. |
|
||||
| `Ctrl+Shift+W` global shortcut | `app/src-tauri/src/lib.rs` | Native toggle visibility; no React consumer needed | Source-wired, still needs target-OS smoke. |
|
||||
| Tray Settings | `app/src-tauri/src/tray.rs`, `apps/web/src/App.tsx`, `apps/web/src/lib/tauri-bindings.ts` | Rust shows/focuses the main window, emits `waggle://navigate` to `/settings`, and the web app mounts `TauriDesktopEventBridge` to route only shipped desktop destinations. | Source-wired, still needs packaged smoke. |
|
||||
| Tray Quit | `app/src-tauri/src/tray.rs` | Native `app.exit(0)` path; no React consumer needed. | Source-wired to Tauri `RunEvent::Exit`, still needs packaged smoke. |
|
||||
| Pause Agents tray action | `app/src-tauri/src/tray.rs` | Menu item and emitter removed. | Hidden until there is real pause/resume behavior. |
|
||||
| About Waggle tray action | `app/src-tauri/src/tray.rs`, `apps/web/src/App.tsx` | Menu item and `/about` emitter removed; no missing-route target remains. | Hidden until there is a real About destination. |
|
||||
| `waggle://update-available` | `apps/web/src/lib/tauri-bindings.ts`, `apps/web/src/App.tsx` | `listenDesktopShellEvents()` keeps a future `Update available` toast mapper, but Rust updater registration/emission is disabled while updater signing is not provisioned. | Future-ready web mapping only; signed-update release-channel proof remains deferred. |
|
||||
| `waggle://service-status` | `app/src-tauri/src/service.rs`, `apps/web/src/lib/tauri-bindings.ts`, `apps/web/src/App.tsx` | `restarting` maps to a reconnecting toast; `failed` maps to a destructive stopped-service recovery toast. | Source/unit-wired to visible UI, still needs packaged watchdog smoke. |
|
||||
| `waggle://service-restart-needed` | `app/src-tauri/src/service.rs`, `apps/web/src/lib/tauri-bindings.ts`, `apps/web/src/App.tsx` | `listenDesktopShellEvents()` maps restart-needed events to a visible local-service restart toast. | Source/unit-wired to visible UI, still needs packaged watchdog smoke. |
|
||||
|
||||
Continuation refresh:
|
||||
|
||||
```text
|
||||
rg -n -F "waggle://pause-agents" app/src-tauri apps/web/src -> no product emitter/listener
|
||||
rg -n -F "waggle://navigate" app/src-tauri apps/web/src -> tray emitter + web bridge
|
||||
rg -n -F "waggle://quit" app/src-tauri apps/web/src -> no product emitter/listener; tray uses app.exit(0)
|
||||
rg -n -F "waggle://update-available" app/src-tauri apps/web/src -> web shell-event listener only; native updater emission disabled
|
||||
rg -n -F "waggle://service-status" app/src-tauri apps/web/src -> Rust emitters + web shell-event listener
|
||||
rg -n -F "waggle://service-restart-needed" app/src-tauri apps/web/src -> Rust emitter + web shell-event listener
|
||||
```
|
||||
|
||||
Inference: Settings now has the frontend listener required for its delegated route
|
||||
behavior, Quit no longer delegates to React, service-watchdog signals now surface
|
||||
as visible toasts through the same desktop bridge, and update availability has a
|
||||
future-ready web mapper while native updater emission remains disabled. The
|
||||
packaged startup/smoke layer is now proven for app boot and sidecar health; tray,
|
||||
close-to-tray, shortcut, and forced watchdog-restart interactions still need
|
||||
targeted packaged interaction evidence.
|
||||
|
||||
## What Is Proven Now
|
||||
|
||||
- Static desktop TypeScript, Rust compilation, config JSON, resource staging, and service-level API flows are in good shape.
|
||||
- The active installed sidecar path is `packages/server/src/local/service.ts` -> `scripts/build-sidecar.mjs` -> `app/src-tauri/resources/service.js` -> `app/src-tauri/src/service.rs`.
|
||||
- `scripts/check-sidecar-resources.mjs` correctly prevents a raw Tauri build from silently omitting staged runtime resources.
|
||||
- `ServiceProvider` has a generic boot/reconnect path with three retries and broadcasts connect-settled state; several routed surfaces show "service unreachable" states.
|
||||
- Release workflow builds workspace packages, then stages Node, native deps, sidecar deps, and frontend before Tauri action builds Windows/macOS draft release artifacts.
|
||||
- Native tray click/Open, close-to-tray, and the global shortcut are implemented in Rust rather than delegated to missing web listeners.
|
||||
- Tray Settings is source-wired through a Tauri desktop navigation bridge to `/settings`; unsupported `/about` navigation is no longer emitted.
|
||||
- Tray Quit is source-wired through Tauri `app.exit(0)`, so the existing `RunEvent::Exit` sidecar cleanup path is reachable from the menu.
|
||||
- Pause Agents and About Waggle are no longer exposed as tray actions until there is real product behavior behind them.
|
||||
- Native service-watchdog events now have React consumers, and the future update event mapper is ready: `waggle://update-available`, `waggle://service-status`, and `waggle://service-restart-needed` map to concise toasts, guarded by `tauri-bindings.test.ts` and the static desktop bridge test.
|
||||
- Packaged debug startup now works locally: the Tauri shell creates the tray icon, starts the bundled sidecar from `target/debug/resources/service.js`, reaches `/health`, and reports healthy database plus running watchdog.
|
||||
- Startup blockers found and fixed by packaged smoke: invalid `plugins.dialog` config, updater plugin initialization without updater config, debug-sidecar source path resolution, bundled Node/native ABI mismatch, and Tauri webview `http://tauri.localhost` CORS rejection.
|
||||
|
||||
## Still Not Proven
|
||||
|
||||
- Packaged debug app launch and sidecar health on Windows are proven locally; clean installed MSI/NSIS launch on Windows/macOS remains unproven.
|
||||
- Published release artifact availability and signed-installer trust are still not proved; the local debug MSI/NSIS artifacts are present but `Get-AuthenticodeSignature` reports `NotSigned` for both.
|
||||
- Actual packaged tray menu behavior for Open, Settings, and Quit.
|
||||
- Close-to-tray behavior in the installed binary.
|
||||
- `Ctrl+Shift+W` global shortcut behavior on target OS.
|
||||
- Packaged-app proof that forced watchdog events produce the expected visible toasts.
|
||||
- Port-conflict recovery in the installed app.
|
||||
- Installer warning/trust experience for unsigned, self-signed, or properly signed channels.
|
||||
- Auto-update user experience. The updater config, capability, and Rust runtime registration are intentionally disabled for v1 until signed updater artifacts exist; the web mapper remains future-ready.
|
||||
|
||||
## Line-Level Findings
|
||||
|
||||
| ID | Finding | Evidence | Correction |
|
||||
|---|---|---|---|
|
||||
| T14-1 | Service app-level events previously had no React consumers, and update mapping was not future-ready. | Current `tauri-bindings.ts` maps `waggle://update-available`, `waggle://service-status`, and `waggle://service-restart-needed` to toast notices, and `App.tsx` mounts `listenDesktopShellEvents()` through `TauriDesktopEventBridge`; guarded by focused binding tests and `tauri-config.test.ts`. Native update emission remains disabled until signed updater artifacts exist. | Focused fixed locally for service events and future update mapping; packaged smoke still needs to prove real service-watchdog events produce visible toasts. |
|
||||
| T14-2 | About tray action previously targeted a route that does not exist. | Current `tray.rs` no longer includes `About Waggle` or `/about`; guarded by `tauri-config.test.ts`. | Focused fixed locally by removing the unsupported tray action. |
|
||||
| T14-3 | Quit tray action previously delegated to an unconsumed web event. | Current `tray.rs` uses `app.exit(0)`; guarded by `tauri-config.test.ts` and `cargo check`. | Focused fixed locally; packaged smoke still needs to prove sidecar cleanup through the real menu. |
|
||||
| T14-4 | Update UX is intentionally disabled and must not break startup. | `app/tests/auto-update.test.ts` and `packages/server/tests/tauri-config.test.ts` require updater config absent, updater capability not exposed, no `UpdaterExt` import, no updater plugin registration, and no startup `.updater()` check. Packaged smoke originally panicked on `plugins.updater: null`; the current binary no longer does. | Keep update UI hidden/deferred, or re-enable signed updater artifacts and visible update handling end to end. |
|
||||
| T14-5 | Generic service reconnect exists, and native watchdog events now surface through the desktop bridge. | `ServiceProvider.tsx` still owns generic reconnect, while `listenDesktopShellEvents()` turns native watchdog status/restart events into visible recovery toasts. | Focused fixed locally; installed-app watchdog failure/restart proof remains. |
|
||||
| T14-6 | Web-side Tauri tests are not discoverable from the root Vitest command. | Root command exits "No test files found" because root config excludes `apps/**`; running from `apps/web` passes 2 files / 19 tests. | Document the correct web-package command in the final verification lane or align root test discovery. |
|
||||
| T14-7 | Installed-app UX still lacks real rendered evidence. | Current command evidence is source/static/API-level only. | Capture packaged app startup, tray, close-to-tray, shortcut, service recovery, and installer trust evidence before a full 9/10 claim. |
|
||||
| T14-8 | Release packaging previously skipped the package build step used by the PR Tauri verification lane. | Historical `.github/workflows/release.yml` went from `npm install` directly to sidecar bundling; current workflow runs `npm run build:packages` before both Windows and macOS sidecar bundle steps, guarded by `tauri-config.test.ts`. | Focused fixed locally; full release closure still needs signed/public artifacts and installed-app smoke. |
|
||||
| T14-9 | Settings tray action previously emitted an unconsumed route event. | Current `App.tsx` mounts `TauriDesktopEventBridge`; `tauri-bindings.ts` listens for `waggle://navigate` and accepts only `/settings`; focused web binding tests pass 15/15. | Focused fixed locally; packaged smoke still needs to prove the real tray menu reaches Settings. |
|
||||
| T14-10 | Packaged startup previously panicked before the UI could load. | Direct debug-exe smoke exposed `plugins.dialog` object deserialization and updater `null` deserialization panics. Current config removes `plugins.dialog`; current Rust does not register updater while config is absent; static tests guard both. | Fixed for debug packaged startup; installer artifact smoke remains. |
|
||||
| T14-11 | Packaged sidecar previously launched the dev `service.ts` path from the wrong root. | Smoke exposed `ERR_MODULE_NOT_FOUND` for `D:\packages\server\src\local\service.ts`. `service.rs` now prefers bundled `resources/service.js` when present, falls back to ancestor-searched dev source only when needed, and has Rust unit tests for both paths. | Fixed; packaged startup smoke reaches `/health`. |
|
||||
| T14-12 | Bundled Node and staged native deps could silently have incompatible ABIs. | Smoke exposed `better_sqlite3.node` built for ABI 127 running under bundled Node ABI 115. `bundle-node.mjs` now defaults to the current staging Node version; `check-sidecar-resources.mjs` fails on ABI mismatch before packaging. | Fixed locally; CI remains aligned because CI stages under Node 20 unless intentionally changed. |
|
||||
| T14-13 | Tauri webview health calls could be rejected by local CORS. | Packaged logs showed `CORS: origin not allowed` before allowing `http://tauri.localhost`; `network-auth.test.ts` now covers Tauri webview localhost origins. | Fixed; current packaged smoke has no CORS rejection in stderr. |
|
||||
|
||||
## Correction Decision
|
||||
|
||||
Do not treat all tray items equally:
|
||||
|
||||
1. Keep Rust-native handling for Open, close-to-tray, and `Ctrl+Shift+W`, then verify them in a packaged smoke.
|
||||
2. Keep Settings as the only delegated tray route and route it through the tested desktop navigation bridge.
|
||||
3. Keep Quit native through `app.exit(0)` so sidecar cleanup is reachable.
|
||||
4. Keep Pause Agents and About Waggle hidden until they have real product behavior.
|
||||
5. Keep service watchdog events surfaced through accessible toasts, keep update mapping future-ready while updater is disabled, then prove watchdog behavior in a packaged smoke.
|
||||
|
||||
## T14 Acceptance
|
||||
|
||||
T14 remains open until either:
|
||||
|
||||
1. Desktop wrapper/release UX is explicitly deferred from the five-persona score, or
|
||||
2. Evidence proves all of the following:
|
||||
|
||||
- Packaged app launches and reaches Home or a clear service-recovery screen. Current debug smoke proves sidecar `/health`; rendered Home still needs packaged visual proof.
|
||||
- Tray Open, Settings, and Quit actions are proved in a packaged smoke; Pause and About remain intentionally removed until implemented.
|
||||
- Close-to-tray and `Ctrl+Shift+W` are verified on a target OS lane.
|
||||
- Sidecar startup, port conflict, crash/restart, and restart-needed states are visible and recoverable.
|
||||
- Installer/signing expectations are documented for the actual release channel.
|
||||
- Update UX is either fully signed and user-visible or intentionally disabled without misleading UI.
|
||||
|
||||
## Phase Impact
|
||||
|
||||
This does not change Phase 1. T14 remains a Phase 2/Launch final-product gate after in-app P0 blockers are cleared, unless a Phase 1 verification command directly requires a small supporting fix.
|
||||
105
docs/audits/2026-07-08-developer-substrate-t17-analysis.md
Normal file
105
docs/audits/2026-07-08-developer-substrate-t17-analysis.md
Normal file
@@ -0,0 +1,105 @@
|
||||
# T17 Developer API, Background Worker, and Substrate UX Analysis
|
||||
|
||||
Date: 2026-07-08
|
||||
Scope: SDK, server API tests, worker jobs, WaggleDance protocol, agent/core/shared/optimizer/weaver, hive-mind substrate, shim core, and wiki compiler verification lanes.
|
||||
Mode: analysis plus focused tooling fix.
|
||||
|
||||
## Bottom Line
|
||||
|
||||
T17 is broadly tested, but not yet a clean 9/10 developer or release-review experience.
|
||||
|
||||
The good news: all 13 scoped workspaces pass direct `tsc --noEmit`, the major root-run test lanes pass, the agent suite passes 195 files / 3097 tests, the server-owned release lane passes 185 files / 2128 tests with one worker, the isolated performance lane passes 13/13, the focused Playwright `webServer` startup blocker is fixed, all scoped package-local test lanes now have working commands, the worker's scheduled-job wrapper now delegates to real allowlisted handlers instead of returning a placeholder, Waggle worker delegation now queues child chat jobs while capability and legacy knowledge responses report concrete availability and gaps, and `/cli allow|deny` now persists and hot-applies the governed CLI allowlist. The remaining problem is command trust outside those named lanes: the fast parallel server invocation can starve startup hooks, warning output remains noisy, and developer recovery journeys are not yet end to end.
|
||||
|
||||
## User Jobs
|
||||
|
||||
- Use `@waggle/sdk` to validate, install, and run skills/plugins.
|
||||
- Trust local server APIs for chat, workspaces, memory, marketplace, billing, backup, compliance, hooks, and startup recovery.
|
||||
- Trust background worker jobs for dispatch and job processing.
|
||||
- Trust WaggleDance protocol behavior and signal handling.
|
||||
- Trust memory substrate, shim core, wiki compiler, optimizer, shared contracts, and agent runtime behavior that power the UI.
|
||||
- Run documented root and package-local verification commands without false failures, skipped tests, or unreadable logs.
|
||||
|
||||
## Scope Inventory
|
||||
|
||||
Scoped packages:
|
||||
|
||||
- `packages/agent`
|
||||
- `packages/core`
|
||||
- `packages/hive-mind-core`
|
||||
- `packages/hive-mind-shim-core`
|
||||
- `packages/hive-mind-wiki-compiler`
|
||||
- `packages/optimizer`
|
||||
- `packages/sdk`
|
||||
- `packages/server`
|
||||
- `packages/shared`
|
||||
- `packages/waggle-dance`
|
||||
- `packages/weaver`
|
||||
- `packages/wiki-compiler`
|
||||
- `packages/worker`
|
||||
|
||||
Source inventory found 512 test/spec files under these scoped packages and roughly 8389 `describe`/`test`/`it` declarations by simple source scan. This is a broad test surface, not an absence-of-tests problem.
|
||||
|
||||
## Command Evidence
|
||||
|
||||
| Check | Result | Notes |
|
||||
|---|---:|---|
|
||||
| Direct `npx tsc --noEmit --project packages/<target>/tsconfig.json` for all 13 scoped packages | Pass, 13/13 | `agent`, `core`, `hive-mind-core`, `hive-mind-shim-core`, `hive-mind-wiki-compiler`, `optimizer`, `sdk`, `server`, `shared`, `waggle-dance`, `weaver`, `wiki-compiler`, and `worker` all typecheck. |
|
||||
| `npm run test -w @waggle/agent -- --reporter=dot` | Pass, 195 files / 3097 tests | Broad agent runtime, orchestration, memory recall, personas, security, tools, and workflow coverage. Output includes reranker loading/status logs. |
|
||||
| `npm run test -w @waggle/core -- --reporter=dot` | Pass, 19 files / 296 tests | Core config, vault, compliance, quota, team sync, and storage tests pass. Output includes embedding provider probes, vault permission warnings, and intentional failure logs. |
|
||||
| `npm run test -w @waggle/optimizer -- --reporter=dot` | Pass, 1 file / 21 tests | Optimizer tests pass through the package script. |
|
||||
| `npm run test -w @waggle/weaver -- --reporter=dot` | Pass, 3 files / 31 tests | Weaver consolidation tests pass through the package script. |
|
||||
| `npx vitest run packages/hive-mind-core/tests packages/hive-mind-shim-core/tests packages/wiki-compiler/tests --config vitest.config.ts --reporter=dot` | Pass, 71 files / 875 tests | Substrate, shim core, and wiki compiler tests pass through the root runner. Output is very noisy with embedding fallback banners and intentional error-path logs. |
|
||||
| `npx vitest run packages/sdk/tests --config vitest.config.ts --reporter=dot` | Pass, 5 files / 89 tests | SDK tests pass from root, including the filesystem-safe plugin-id regression. |
|
||||
| `npm run test -w @waggle/shared -- --reporter=dot` | Pass, 5 files / 40 tests | Shared contract tests now have a package-owned root-config command. |
|
||||
| `npm run test -w @waggle/waggle-dance -- --reporter=dot` | Pass, 3 files / 42 tests | WaggleDance protocol tests now have a package-owned root-config command. |
|
||||
| `npm run test -w @waggle/worker -- --reporter=dot` | Pass, 4 files / 46 tests | Worker execution and handler tests now have a package-owned root-config command. |
|
||||
| `npm run test -w @waggle/hive-mind-wiki-compiler -- --reporter=dot` | Pass, 3 files / 26 tests | Colocated `src/*.test.ts` files are now included explicitly in root discovery; resolver tests isolate provider selection from Vitest CJS/ESM interop. |
|
||||
| `npm run test:perf -- --reporter=dot` | Pass, 1 file / 13 tests | Dedicated wall-clock benchmark lane; the default Vitest gate excludes `packages/server/tests/performance/**`. |
|
||||
| `npm run test -w @waggle/server -- --reporter=dot` | Pass, 185 files passed / 1 skipped; 2128 tests passed / 1 skipped | Server-owned deterministic release lane uses one worker and silent console output; duration 349.32s. A faster parallel invocation remains useful feedback but is not the release gate because server boot hooks can contend for resources. |
|
||||
| Focused Playwright marketplace slice, port 34203 | Pass, 4/4 | The old ports `34201` and `34202` failed before assertions because transitive `tsx@4.22.3` used `esbuild` host `0.28.0` while resolving the root Windows binary `0.21.5`. Pinning the repo's direct `tsx` dependency to `4.21.0` dedupes it to root `esbuild@0.27.7`; `npx tsx -e` succeeds and Playwright `webServer` starts the sidecar before assertions. |
|
||||
| `npm run test -w @waggle/hive-mind-core -- --reporter=dot` | Pass, 59 files / 745 tests | Package script now delegates to the root Vitest config, which owns the workspace aliases and shared setup. |
|
||||
| `npm run test -w @waggle/hive-mind-shim-core -- --reporter=dot` | Pass, 10 files / 105 tests | Package-local script now runs the root Vitest config against the shim-core tests. The integration lane also verifies the CLI ESM resolver path for the MCP server entry. |
|
||||
| `npm run test -w @waggle/wiki-compiler -- --reporter=dot` | Pass, 2 files / 25 tests | Package script now delegates to the root Vitest config and shared setup. |
|
||||
| `npm run test -w @waggle/sdk -- --reporter=dot` | Pass, 5 files / 89 tests | Package script now delegates to the root Vitest config, avoiding the incomplete workspace-local dependency tree. |
|
||||
|
||||
## Source Findings
|
||||
|
||||
| ID | Severity | Finding | Evidence | Correction Needed |
|
||||
|---|---:|---|---|---|
|
||||
| T17-1 | Resolved | Package-local test scripts failed for packages whose root-run tests passed. | `hive-mind-core` now passes 59 files / 745 tests, `wiki-compiler` passes 2 / 25, and `sdk` passes 5 / 89 through their workspace commands; `hive-mind-shim-core` already owned the same root-config pattern. | Keep the package scripts on the root-config delegation pattern and guard them in the T17 verification lane. |
|
||||
| T17-2 | Resolved for the named server lane | The standard server lane needed stable release semantics. | `npm run test -w @waggle/server -- --reporter=dot` passes 185 files / 2128 tests with one worker; `npm run test:perf -- --reporter=dot` passes 13/13 separately. The prior parallel run had four startup-hook timeouts, so it remains fast feedback rather than the release gate. | Keep the package-owned single-worker lane and isolated perf command documented. |
|
||||
| T17-3 | Resolved for the standard server lane | Marketplace sync behavior leaked into normal server verification. | `packages/server/tests/local/marketplace-sync.test.ts` now stubs fetch and console output through a hermetic helper; the focused lane passes 13/13 without external sync logs. | Keep external catalog adapter coverage in a separately named live/integration lane. |
|
||||
| T17-4 | Partially resolved | Passing backend/substrate runs were too noisy for reviewer use. | Root Vitest now runs with `silent: true`; the mock-provider degradation banner is suppressed only in test setup via `WAGGLE_SUPPRESS_EMBEDDING_WARNING=1`, while production runs remain loud. Direct subprocess diagnostics and selected live/integration logs still need cleanup. | Keep the quiet default and finish warning categorization for live/integration lanes. |
|
||||
| T17-5 | Resolved | Some packages had no local test script even though their behavior was covered from root. | `@waggle/worker` passes 4 files / 46 tests, `@waggle/waggle-dance` passes 3 / 42, `@waggle/shared` passes 5 / 40, and `@waggle/hive-mind-wiki-compiler` passes 3 / 26 through package-owned root-config scripts. | Keep these package commands in the verification lane. |
|
||||
| T17-6 | P2 | Developer-facing happy paths are tested, but coherent recovery journeys are not fully sampled. | Unit/API tests cover many pieces, the worker's scheduled `cron` wrapper now has explicit validation plus real-handler delegation coverage, Waggle worker tests cover real child-job enqueue, honest missing-capability reporting, and concrete legacy knowledge gaps, and command-route tests cover persisted `/cli allow|deny` updates with live tool permission changes. This packet still does not prove SDK docs/examples, bad config setup, worker failure UI copy, or server API consumer ergonomics as end-to-end developer journeys. | Add developer-journey smoke docs/tests or explicitly defer these from the five-persona score. |
|
||||
| T17-7 | Resolved | Playwright webServer startup could fail before product assertions because `tsx` and esbuild binaries were misaligned. | Old focused marketplace Playwright runs on ports 34201 and 34202 failed before tests with `Host version "0.28.0" does not match binary version "0.21.5"`. Current package tree pins direct `tsx@4.21.0`, dedupes to `esbuild@0.27.7`, passes `npx tsx -e`, and the focused marketplace Playwright slice passes 4/4 on port `34203`. | Keep the direct `tsx` pin or equivalent matching host/binary invariant. |
|
||||
|
||||
## Persona Impact
|
||||
|
||||
| Persona | Current T17 cap | Why |
|
||||
|---|---:|---|
|
||||
| Engineer / power user | 8/10 | Package-local lanes and a stable server release command now work, but warning hygiene and developer recovery journeys remain open. |
|
||||
| Team admin / security reviewer | 8/10 | Server, worker, compliance, vault, and backup APIs pass tests, but noisy failure-looking logs and unclear command lanes reduce release confidence. |
|
||||
| Solo founder | 8/10 | Less direct, but the visible product depends on these APIs and background jobs. |
|
||||
| Researcher | 8/10 | Memory substrate tests are broad, but embedding-noise and command-shape failures undercut provenance confidence. |
|
||||
| Mobile executive | 8/10 | Indirect impact through stability and release confidence. |
|
||||
|
||||
## Acceptance For Closing T17
|
||||
|
||||
- Package-local scripts either pass or clearly delegate to the correct root/project-reference lane.
|
||||
- Root verification discovers all intended package tests, or every intentional separate lane is documented.
|
||||
- Full server tests and server performance tests have stable release semantics: default deterministic lane plus isolated perf/live-integration lanes where needed.
|
||||
- Marketplace sync tests in the standard lane are hermetic and quiet, or are moved to a live-integration lane.
|
||||
- Playwright `webServer` startup uses a matching `tsx`/esbuild host/binary pair and can start the sidecar before assertions. Current focused evidence passes 4/4 on port `34203`.
|
||||
- Expected warning noise is suppressed, filtered, or explicitly summarized so real failures stand out.
|
||||
- SDK/server/worker/WaggleDance/substrate developer journeys have happy-path and recovery/error-path evidence, or are explicitly deferred from the five-persona score.
|
||||
|
||||
## Packet Decision
|
||||
|
||||
Keep T17 as `Phase 2 Pending` / tooling-release-confidence gate. Package-local test-script failures, perf-lane separation, and standard-server marketplace leakage are fixed. Warning hygiene and developer journey evidence still block the final "complete UX, all parts functional, five judges at 9/10" claim unless the user explicitly defers developer API, background worker, and substrate verification from the score.
|
||||
|
||||
### 2026-07-10 follow-up: group execution recovery
|
||||
|
||||
The agent-group surface had a concrete end-user dead end that was not covered by the prior worker evidence: local `/api/agent-groups/:id/run` returned a synthetic job ID, while local `/api/jobs/:id` did not exist. The local sidecar now persists bounded in-memory job state, runs persona-backed groups through `SubagentOrchestrator`, reports worker progress/output, supports cancellation through the agent-loop abort signal, validates group members/strategies, and exposes local job status/cancel routes. The cloud route now queues the worker-supported `group` job shape instead of inserting an unsupported `group_execution` row. Focused route/orchestrator coverage passes 19/19; package builds, server/agent/shared typechecks, and the web production build pass.
|
||||
|
||||
This closes the named group-run recovery gap. T17 remains pending for the separate warning-hygiene and SDK/server consumer recovery journeys listed above.
|
||||
@@ -0,0 +1,214 @@
|
||||
# First-Run Onboarding T1/T2/T12 Analysis - 2026-07-08
|
||||
|
||||
Status: analysis-only supplement. No product code was changed.
|
||||
|
||||
Purpose: close the audit blind spot left by the standard `?skipOnboarding=true` harness. This run exercised a clean local data dir without skip flags, then followed the first-run path through onboarding, template creation, first-task auto-send, and post-onboarding chat.
|
||||
|
||||
## Evidence
|
||||
|
||||
Source inspected:
|
||||
|
||||
- `apps/web/src/hooks/useOnboarding.ts`
|
||||
- `apps/web/src/components/os/AppShell.tsx`
|
||||
- `apps/web/src/components/os/overlays/OnboardingWizard.tsx`
|
||||
- `apps/web/src/components/os/overlays/onboarding/{WelcomeStep,WhoAreYouStep,ModelGateStep,ImportStep,TemplateStep,FirstTaskStep}.tsx`
|
||||
- `apps/web/src/components/os/overlays/onboarding/constants.ts`
|
||||
- `packages/server/src/local/routes/onboarding.ts`
|
||||
- Existing tests under `apps/web/src/test/*onboarding*` and `tests/e2e/*onboarding*`
|
||||
|
||||
Commands:
|
||||
|
||||
```powershell
|
||||
npm run build
|
||||
```
|
||||
|
||||
Fresh runtime:
|
||||
|
||||
```powershell
|
||||
$env:WAGGLE_PORT='3431'
|
||||
$env:WAGGLE_TRUST_LOCALHOST='1'
|
||||
$env:WAGGLE_DISABLE_MARKETPLACE_SYNC='1'
|
||||
$env:EMBEDDING_PROVIDER='mock'
|
||||
$env:VITE_CLERK_PUBLISHABLE_KEY=''
|
||||
$env:CLERK_SECRET_KEY=''
|
||||
$env:WAGGLE_DATA_DIR = "$env:TEMP\waggle-first-run-smoke-3431-20260708050443"
|
||||
npx tsx packages/server/src/local/start.ts --skip-litellm
|
||||
```
|
||||
|
||||
Artifacts:
|
||||
|
||||
- Trace: `output/playwright/first-run-onboarding-3431/first-run-onboarding-summary.json`
|
||||
- Reopen trace: `output/playwright/first-run-onboarding-3431/after-reopen-wait.json`
|
||||
- Screenshots:
|
||||
- `output/playwright/first-run-onboarding-3431/desktop-00-welcome.png`
|
||||
- `output/playwright/first-run-onboarding-3431/desktop-01-profile-empty.png`
|
||||
- `output/playwright/first-run-onboarding-3431/desktop-02-profile-filled.png`
|
||||
- `output/playwright/first-run-onboarding-3431/desktop-03-model-gate.png`
|
||||
- `output/playwright/first-run-onboarding-3431/desktop-04-import.png`
|
||||
- `output/playwright/first-run-onboarding-3431/desktop-05-template.png`
|
||||
- `output/playwright/first-run-onboarding-3431/desktop-06-first-task.png`
|
||||
- `output/playwright/first-run-onboarding-3431/desktop-07-after-lets-go.png`
|
||||
- `output/playwright/first-run-onboarding-3431/desktop-08-after-reopen-wait.png`
|
||||
- `output/playwright/first-run-onboarding-3431/mobile-00-welcome.png`
|
||||
- `output/playwright/first-run-onboarding-3431/mobile-01-profile-empty.png`
|
||||
|
||||
Runtime outcome:
|
||||
|
||||
- `npm run build` passed with the known Tailwind ambiguous-class warnings, dynamic import warning, and large main chunk warning.
|
||||
- Fresh server started on `http://127.0.0.1:3431` with a temporary data dir and mock embeddings.
|
||||
- Desktop first-run path rendered the onboarding takeover instead of shell chrome.
|
||||
- Desktop completed: Welcome -> Profile -> Model Gate -> Import skip -> Template -> First Task -> workspace chat.
|
||||
- A second fresh mobile browser context opened before completion and rendered Welcome plus Profile at 390 x 844.
|
||||
- Post-completion reopen reached `/workspaces/research-hub/chat` with an empty composer after generation completed.
|
||||
|
||||
## Findings
|
||||
|
||||
### FRO-1: First-run accountless console health (verified fixed for sampled paths)
|
||||
|
||||
The first-run smoke captured 15 console/page errors. They are the same T1 class already found on skipped route smokes:
|
||||
|
||||
- inline script blocked by `script-src 'self'`
|
||||
- Clerk script blocked by local CSP
|
||||
- Clerk failed to load / timeout
|
||||
|
||||
Current verification update:
|
||||
|
||||
- `clean first-run onboarding loads without Clerk, CSP, or page errors` passed on port `34196`.
|
||||
- `no console errors on initial load` passed on port `34196`.
|
||||
- `no critical console errors on load` passed on port `34196`.
|
||||
|
||||
Impact:
|
||||
|
||||
- Solo Founder and Team Admin could not receive a final 9/10 while accountless local-first onboarding emitted auth/security failures. The current focused console-health checks close this sampled cap; explicit Clerk-enabled auth remains a separate state bundle.
|
||||
- Phase 1 T1 must keep the first-run lane in regression, not only the skip-onboarding route lane.
|
||||
|
||||
Correction:
|
||||
|
||||
- Keep accountless local mode as the default unless Clerk is explicitly enabled.
|
||||
- Add first-run console capture to the T1 verification lane.
|
||||
|
||||
### FRO-2: Desktop onboarding is functionally complete
|
||||
|
||||
The desktop path reached the terminal chat route and auto-sent the seeded first task. The final route was:
|
||||
|
||||
```text
|
||||
http://127.0.0.1:3431/workspaces/research-hub/chat
|
||||
```
|
||||
|
||||
The flow was logical overall:
|
||||
|
||||
- Welcome explains the local-first promise.
|
||||
- Profile captures useful personalization signals.
|
||||
- Model gate allowed continuation because a local model was available.
|
||||
- Template recommendation correctly floated Research Hub for a consulting/research profile.
|
||||
- First task seeded the chat with the chosen template hint.
|
||||
|
||||
Impact:
|
||||
|
||||
- This is a strong Solo Founder evidence lane now that sampled T1 console health is verified fixed.
|
||||
|
||||
Correction:
|
||||
|
||||
- Preserve this end-to-end contract while fixing the polish items below.
|
||||
|
||||
### FRO-3: Model gate status copy can be stale while Continue is enabled (focused fixed)
|
||||
|
||||
The model gate screenshot showed the status strip still saying `Checking your models...` while the Continue button was enabled and clickable.
|
||||
|
||||
Current verification update:
|
||||
|
||||
- When `useHasWorkingModel()` already reports a working model, onboarding now renders a `Model ready` status instead of the setup/checking gate.
|
||||
- `ModelGateStep.test.tsx` passed 5/5, including a regression that hides the setup gate and checking copy while Continue is enabled.
|
||||
|
||||
Impact:
|
||||
|
||||
- This does not block completion, but it weakens trust. A user sees two contradictory states: still checking vs ready to continue.
|
||||
|
||||
Correction:
|
||||
|
||||
- Once `hasWorkingModel` is true, replace the checking copy with a ready state that names the working provider/model, or hide the checking strip.
|
||||
|
||||
### FRO-4: High-volume Claude Code auto-detect makes import too easy for day-zero setup (focused fixed)
|
||||
|
||||
The import step detected 5,514 Claude Code items and placed `Import my history` as a primary button directly in onboarding.
|
||||
|
||||
Current verification update:
|
||||
|
||||
- High-volume detected histories now switch to volume-aware copy, format the count (`5,514 items from Claude Code`), make `Review after setup` the primary action, and demote immediate import to explicit `Import 5,514 now`.
|
||||
- `apps/web/src/test/onboarding-import-step.test.tsx` passed 2/2, covering the high-volume deliberate-review flow and preserving the simple `Import my history` CTA for small detected histories.
|
||||
- `npm run typecheck:web` passed after the component change.
|
||||
|
||||
Impact:
|
||||
|
||||
- Functionally impressive, but risky for first-run UX. A fresh user can trigger a large import before seeing the product, understanding review consequences, or choosing a workspace.
|
||||
- Researcher trust and Solo Founder setup speed are both affected.
|
||||
|
||||
Correction:
|
||||
|
||||
- Keep the detection signal, but make high-volume import a deliberate secondary choice.
|
||||
- Add volume-aware copy such as "Review import options" or "Import later from Memory" for large detected histories.
|
||||
- Preserve the skip path and route users to Memory Harvest after setup.
|
||||
|
||||
### FRO-5: Mobile profile primary action reachability (verified fixed)
|
||||
|
||||
At 390 x 844, the Welcome step fits well. After Continue, the Profile step becomes taller than the viewport and the primary Continue button starts below the visible area. The document itself has no horizontal overflow, but the critical bottom action is not visible without scrolling inside the onboarding content area.
|
||||
|
||||
Evidence:
|
||||
|
||||
- `mobile-01-profile-empty.png`
|
||||
- Trace out-of-bounds entry: `Continue` button bottom at `880` in an `844` px viewport.
|
||||
- Current verification update: `J-mobile: first-run onboarding keeps primary actions reachable at 390px width` passed 1/1 on port `34194`. The focused test clears storage, opens `/?forceWizard=true`, advances from Welcome to Profile, asserts the Profile Continue button bottom is within the 844 px viewport, checks visible horizontal overflow, and fails on Clerk/CSP/page errors.
|
||||
|
||||
Impact:
|
||||
|
||||
- The original finding capped Mobile Executive and Solo Founder mobile first-run paths below 9/10. The current focused route evidence closes that specific cap; other first-run trust items remain separate caps.
|
||||
- This reinforces the mobile supplement lesson: document-level scroll width is insufficient; judge evidence must include visible control bounds and vertical reachability.
|
||||
|
||||
Correction:
|
||||
|
||||
- On mobile, make the onboarding action row sticky within the wizard, reduce vertical density, or split Profile into a lighter first pass plus optional details.
|
||||
- Ensure the step title, progress, and primary action are visible or clearly reachable at 390 x 844.
|
||||
|
||||
### FRO-6: First-task auto-send briefly leaves the same text in the composer (focused fixed)
|
||||
|
||||
Immediately after `Let's go`, the chat showed the user bubble and active generation while the composer still contained the first-task text. A reopen after 12 seconds showed the composer empty, so this appears transient.
|
||||
|
||||
Current verification update:
|
||||
|
||||
- `ChatApp` now clears the untouched auto-send seed immediately when the first-task send is consumed, before the send promise resolves, and restores it only if send returns `false`.
|
||||
- `lane-c-input-power.test.tsx` passed 12/12, including the regression that the auto-sent first task clears before the pending send resolves.
|
||||
|
||||
Impact:
|
||||
|
||||
- Not a persistence/data bug, but the handoff can look like a duplicate-send risk during the most important first success moment.
|
||||
|
||||
Correction:
|
||||
|
||||
- Clear or disable the composer immediately when auto-send starts, and show the sent state distinctly.
|
||||
|
||||
## Ticket Updates
|
||||
|
||||
| Finding | Ticket | Phase | Required closure |
|
||||
|---|---|---:|---|
|
||||
| FRO-1 first-run Clerk/CSP errors | T1 | 1 | Verified fixed for sampled accountless paths: first-run, initial load, and full-product critical console checks passed 3/3 on port `34196`. |
|
||||
| FRO-5 mobile onboarding Continue below viewport | T2/T12 | 1 | Verified fixed in current build: 390 x 844 first-run Profile keeps Continue reachable with visible-bounds evidence in `tests/e2e/user-journeys.spec.ts` on port `34194`. |
|
||||
| FRO-3 stale model gate checking copy | T10/T12 | 2 | Focused fixed: ready model state hides the setup/checking gate and shows `Model ready`. |
|
||||
| FRO-4 high-volume import CTA risk | T7/T12 | 2 | Focused fixed: large detected histories now require deliberate review/secondary-action copy, with regression coverage in `onboarding-import-step.test.tsx`. |
|
||||
| FRO-6 transient first-task composer duplicate | T12 | 2 | Focused fixed: auto-send clears the untouched first-task composer seed before the send promise resolves, with failure restore. |
|
||||
|
||||
## Judge Implications
|
||||
|
||||
- Solo Founder: first-run onboarding is now a required screenshot/evidence lane, not optional. Sampled T1 console health, mobile FRO-5, high-volume import CTA risk, model-ready copy, and first-task composer handoff are currently focused fixed.
|
||||
- Researcher: FRO-4 is now focused fixed for the first-run CTA risk; deeper Harvest review/recovery states still belong in the broader T12/T19 evidence packet.
|
||||
- Engineer: FRO-1 console health and first-task route evidence remain relevant.
|
||||
- Team Admin: sampled FRO-1 auth/security noise and FRO-4 first-run import consequence copy are currently fixed; deeper data-review governance still belongs in broader state evidence.
|
||||
- Mobile Executive: the original FRO-5 blocker is currently closed by the focused 390 x 844 journey; keep it in regression because it is a direct visual/responsive path.
|
||||
|
||||
## Approval Impact
|
||||
|
||||
Phase 1 remains the right first implementation phase, but its T1/T2 verification must include this supplement:
|
||||
|
||||
- T1: keep accountless first-run onboarding console health in regression and add explicit Clerk-enabled state evidence later.
|
||||
- T2: mobile onboarding Profile primary-action reachability, in addition to mobile Settings.
|
||||
|
||||
No product-code changes are approved by this document.
|
||||
447
docs/audits/2026-07-08-five-persona-judge-runbook.md
Normal file
447
docs/audits/2026-07-08-five-persona-judge-runbook.md
Normal file
@@ -0,0 +1,447 @@
|
||||
# Five-Persona Judge Runbook - 2026-07-08
|
||||
|
||||
Status: analysis artifact. This is not an implementation plan and does not approve product-code changes.
|
||||
|
||||
Purpose: make the final "five judges score 9/10" gate executable. The existing `tests/vision/personas.spec.ts` is useful evidence for live chat, persistence, screenshots, and cross-persona isolation, but it does not cover the full product UX. This runbook defines the current-source evidence packet required after Phase 1 and any judge-blocking Phase 2/3 items land.
|
||||
|
||||
Companion artifacts:
|
||||
|
||||
- `docs/audits/2026-07-08-five-persona-judge-scorecards.md`
|
||||
- `docs/audits/2026-07-08-ux-route-scenario-manifest.md`
|
||||
- `docs/audits/2026-07-08-ux-state-failure-scenario-matrix.md`
|
||||
- `docs/audits/2026-07-08-ux-correction-register.md`
|
||||
- `docs/audits/2026-07-08-web-guidelines-line-findings.md`
|
||||
- `docs/audits/2026-07-08-source-inventory-consistency-audit.md`
|
||||
- `docs/audits/2026-07-08-state-failure-t12-analysis.md`
|
||||
- `docs/audits/2026-07-08-mobile-executive-t2-t12-analysis.md`
|
||||
- `docs/audits/2026-07-08-runtime-a11y-t10-analysis.md`
|
||||
- `docs/audits/2026-07-08-first-run-onboarding-t1-t2-t12-analysis.md`
|
||||
|
||||
## Hard Preconditions
|
||||
|
||||
Do not treat a judge run as final unless all are true:
|
||||
|
||||
1. T1, T2, T3, T4, T5, and T11 are closed or explicitly reclassified with evidence.
|
||||
2. T7 trust-critical dialog work is closed for every persona route, or the affected persona is capped below pass.
|
||||
3. T10 accessibility/form/focus findings that touch judge routes are closed or explicitly deferred with score impact, including shell overlay semantics and close behavior.
|
||||
4. T12 state bundles are declared for every persona and cross-checked against the focused T12 supplement.
|
||||
5. Mobile evidence includes visible element-bounds checks for critical controls, first-run onboarding Profile, and selected overlay close proof; document-level overflow alone is insufficient.
|
||||
6. T13/T14/T15/T16/T17/T18/T19 are either evidenced or explicitly deferred by the user from the five-persona score.
|
||||
7. The final app run is built from current source on a fresh port and clean data dir unless a return-state scenario intentionally reuses data.
|
||||
8. Console status is captured from navigation start, not after the page settles.
|
||||
|
||||
## Existing Harness Boundary
|
||||
|
||||
`tests/vision/personas.spec.ts`:
|
||||
|
||||
- Creates five isolated workspaces.
|
||||
- Sends two chat turns per persona.
|
||||
- Saves screenshots and JSON under `tests/vision/artifacts/personas`.
|
||||
- Checks substantive assistant history and no cross-persona prompt leakage.
|
||||
|
||||
It does not prove:
|
||||
|
||||
- Auth/accountless boot quality.
|
||||
- Home Start Here first-action quality.
|
||||
- Settings/billing/profile UX.
|
||||
- Mobile 390 px behavior.
|
||||
- Command Center, shortcuts, Launcher, MCP, Files, Events.
|
||||
- Vault, Approvals, backup/restore, Team governance.
|
||||
- Public launch, desktop wrapper, admin/CLI/MCP utility, hook lifecycle, Browser Companion extension, developer/substrate, or ops gates.
|
||||
|
||||
Conclusion: use `tests/vision/personas.spec.ts` as one evidence source, not as the final judge.
|
||||
|
||||
## Standard Evidence Commands
|
||||
|
||||
Use a fresh port to avoid stale-server evidence:
|
||||
|
||||
```powershell
|
||||
$env:WAGGLE_E2E_PORT='3397'
|
||||
$env:WAGGLE_E2E_BASE_URL='http://127.0.0.1:3397'
|
||||
$env:WAGGLE_E2E_SKIP_LITELLM='1'
|
||||
$env:WAGGLE_E2E_DATA_DIR="$env:TEMP\\waggle-ux-judge-3397"
|
||||
$env:WAGGLE_DISABLE_MARKETPLACE_SYNC='1'
|
||||
$env:EMBEDDING_PROVIDER='mock'
|
||||
$env:VITE_CLERK_PUBLISHABLE_KEY=''
|
||||
$env:CLERK_SECRET_KEY=''
|
||||
```
|
||||
|
||||
Core verification before screenshots:
|
||||
|
||||
```powershell
|
||||
npm run typecheck:web
|
||||
npm run ux:contrast
|
||||
npm run ux:color-guard
|
||||
npm run build
|
||||
node node_modules/playwright/cli.js test tests/e2e/full-product-audit.spec.ts tests/e2e/full-wiring-audit.spec.ts tests/e2e/phase-ab-verification.spec.ts tests/e2e/power-user-stress.spec.ts tests/e2e/user-journeys.spec.ts tests/visual/views.spec.ts --project=chromium --reporter=list
|
||||
```
|
||||
|
||||
Optional live-LLM persona evidence:
|
||||
|
||||
```powershell
|
||||
$env:WAGGLE_E2E_SKIP_LITELLM='0'
|
||||
node node_modules/playwright/cli.js test tests/vision/personas.spec.ts --project=chromium --reporter=list
|
||||
```
|
||||
|
||||
Important: query parameter `?tier=power` controls the UI disclosure tier (`simple`, `professional`, `power`, `admin`) through onboarding state. It is not the billing tier (`FREE`/Solo, `TRIAL`, `TEAMS`, `ENTERPRISE`). Judge evidence must name both separately.
|
||||
|
||||
## Evidence Directory
|
||||
|
||||
Use a timestamped evidence root:
|
||||
|
||||
```text
|
||||
docs/audits/evidence/2026-07-08-five-persona-judge/<run-id>/
|
||||
```
|
||||
|
||||
Required files:
|
||||
|
||||
```text
|
||||
00-command-log.md
|
||||
00-console-summary.json
|
||||
00-route-coverage.md
|
||||
00-deferrals.md
|
||||
persona-1-solo-founder/
|
||||
persona-2-researcher/
|
||||
persona-3-engineer/
|
||||
persona-4-team-admin/
|
||||
persona-5-mobile-executive/
|
||||
score-summary.md
|
||||
```
|
||||
|
||||
Each persona folder must contain:
|
||||
|
||||
```text
|
||||
state-bundle.md
|
||||
steps.md
|
||||
screenshots/
|
||||
console.json
|
||||
network.json
|
||||
route-evidence.md
|
||||
scorecard.md
|
||||
blockers.md
|
||||
```
|
||||
|
||||
## Score Caps
|
||||
|
||||
Apply caps before subjective scoring:
|
||||
|
||||
| Condition | Cap |
|
||||
|---|---:|
|
||||
| Critical console error in persona route | 7/10 |
|
||||
| Primary route blocked or blank | 6/10 |
|
||||
| Severe mobile clipping/overflow in mobile persona path | 7/10 |
|
||||
| First-run onboarding primary action hidden in required mobile path | 7/10 |
|
||||
| Required overlay opens but cannot close in persona path | 7/10 |
|
||||
| Required overlay has no accessible name/landmark or unnamed primary icon-only actions | 8/10 |
|
||||
| Native browser dialog in a trust-critical persona step | 8/10 |
|
||||
| Critical axe finding in a persona primary route | 8/10 |
|
||||
| Serious keyboard access finding in a persona primary route | 8/10 |
|
||||
| Active Pro copy in pricing/billing/gating path | 8/10 |
|
||||
| Missing route evidence owner for a primary route | 8/10 |
|
||||
| Missing required state bundle evidence | 8/10 |
|
||||
| Missing non-main gate decision for relevant T13-T19 path | 8/10 |
|
||||
|
||||
The final goal requires every persona to score at least 9/10, so any active cap below 9 is a blocker.
|
||||
|
||||
## Persona 1: Solo Founder
|
||||
|
||||
State bundle:
|
||||
|
||||
- Account mode: accountless local.
|
||||
- Billing tier: Solo / `FREE`.
|
||||
- UI disclosure tier: `simple` first, then `power` only for route evidence if needed.
|
||||
- Model state: no model recovery and one working-model or skipped-LLM explanation.
|
||||
- Data state: fresh install, no workspace first, then one created workspace.
|
||||
- Offline/error state: accountless Clerk/CSP lane.
|
||||
- Viewports: desktop 1440 x 900 and mobile Home spot-check.
|
||||
|
||||
Route sequence:
|
||||
|
||||
1. `/auth`
|
||||
2. first-run onboarding or approved skip path
|
||||
3. `/home`
|
||||
4. `/workspaces`
|
||||
5. `/workspaces/:workspaceId/chat`
|
||||
6. return to `/home`
|
||||
|
||||
Required screenshots:
|
||||
|
||||
- `auth-accountless.png`
|
||||
- `onboarding-welcome.png`
|
||||
- `onboarding-profile.png`
|
||||
- `onboarding-model-gate.png`
|
||||
- `onboarding-first-task.png`
|
||||
- `home-start-here.png`
|
||||
- `workspace-create-or-list.png`
|
||||
- `first-chat.png`
|
||||
- `home-return-next-action.png`
|
||||
- `mobile-home.png`
|
||||
|
||||
Must prove:
|
||||
|
||||
- Home gives a clear next move within 10 seconds.
|
||||
- No Clerk/CSP console noise in accountless mode.
|
||||
- Clean-data first-run onboarding reaches the wizard without Clerk/CSP console errors.
|
||||
- Mobile first-run Profile keeps the primary Continue action visible or clearly reachable if mobile first-run is scored.
|
||||
- No active Pro copy in the path.
|
||||
- Memory behavior is honest: no unsupported promise that context will be remembered without evidence.
|
||||
|
||||
Current blockers from the packet:
|
||||
|
||||
- T1, T3, T11.
|
||||
- T2 if mobile Home/Settings or mobile first-run onboarding are used in the score.
|
||||
|
||||
## Persona 2: Researcher
|
||||
|
||||
State bundle:
|
||||
|
||||
- Account mode: accountless or authenticated, but declared.
|
||||
- Billing tier: Solo unless Teams feature is intentionally tested.
|
||||
- UI disclosure tier: `power`.
|
||||
- Model state: working or skipped-LLM with memory UI focus.
|
||||
- Data state: populated memory plus empty/no-result state.
|
||||
- Offline/error state: missing source or failed export path.
|
||||
- Viewports: desktop 1440 x 900; mobile Memory spot-check if scored.
|
||||
|
||||
Route sequence:
|
||||
|
||||
1. `/memory`
|
||||
2. memory search/no-results
|
||||
3. memory provenance/trust detail
|
||||
4. wiki/timeline/evolution view
|
||||
5. archive/delete/export confirmation flow
|
||||
6. `/workspaces/:workspaceId/chat`
|
||||
|
||||
Required screenshots:
|
||||
|
||||
- `memory-overview.png`
|
||||
- `memory-search-result.png`
|
||||
- `memory-empty-or-no-results.png`
|
||||
- `trust-provenance.png`
|
||||
- `wiki-or-timeline.png`
|
||||
- `memory-confirmation-modal.png`
|
||||
- `memory-chat-explanation.png`
|
||||
|
||||
Must prove:
|
||||
|
||||
- Researcher can tell what is stored, where it came from, and how to correct/remove it.
|
||||
- Delete/export flows do not use native `confirm`/`prompt`.
|
||||
- Long memory text/titles do not break layout.
|
||||
- The product does not overclaim memory persistence.
|
||||
|
||||
Current blockers:
|
||||
|
||||
- T5, T7, T10, T11, T12.
|
||||
- T19 if browser capture is included in the Researcher journey.
|
||||
|
||||
## Persona 3: Engineer / Power User
|
||||
|
||||
State bundle:
|
||||
|
||||
- Account mode: accountless local.
|
||||
- Billing tier: Solo, plus explicit deferral/evidence for Teams-only surfaces.
|
||||
- UI disclosure tier: `power` or `admin`.
|
||||
- Model state: local/no-LLM and one working-provider lane if using chat.
|
||||
- Data state: at least one workspace, detected or undetected tools, MCP catalog present.
|
||||
- Offline/error state: marketplace local-only and tool/hook unavailable states.
|
||||
- Viewports: desktop 1440 x 900; keyboard-only path.
|
||||
|
||||
Route sequence:
|
||||
|
||||
1. `/home`
|
||||
2. Command Center via `Ctrl+K`
|
||||
3. `Ctrl+Shift+N` to active workspace chat
|
||||
4. `/launcher`
|
||||
5. `/mcps`
|
||||
6. `/files`
|
||||
7. `/settings/events`
|
||||
8. representative CLI/MCP utility evidence if T15 not deferred
|
||||
|
||||
Required screenshots:
|
||||
|
||||
- `command-center.png`
|
||||
- `shortcut-chat-result.png`
|
||||
- `launcher-tool-state.png`
|
||||
- `launcher-hook-state.png`
|
||||
- `mcp-hub.png`
|
||||
- `files.png`
|
||||
- `events-logs.png`
|
||||
- `keyboard-focus-path.png`
|
||||
|
||||
Must prove:
|
||||
|
||||
- `Ctrl+Shift+N` opens the intended chat route.
|
||||
- Workspace Switcher does not block unrelated navigation.
|
||||
- Standard audit avoids live external marketplace dependency.
|
||||
- Tool and MCP states are explained without broken JSON or secret leakage.
|
||||
|
||||
Current blockers:
|
||||
|
||||
- T4, T6, T10, T11, T15, T16, T17, T18 unless deferred.
|
||||
|
||||
## Persona 4: Team Admin / Security Reviewer
|
||||
|
||||
State bundle:
|
||||
|
||||
- Account mode: authenticated or accountless with billing/admin limitations declared.
|
||||
- Billing tier: Teams for team/admin surfaces, Solo for gating comparison, legacy Pro collapsed to Solo where relevant.
|
||||
- UI disclosure tier: `professional` and `admin`.
|
||||
- Model state: not central unless settings model copy is inspected.
|
||||
- Data state: vault item, approval grant, backup metadata, team governance state.
|
||||
- Offline/error state: backup failure or restore failure copy.
|
||||
- Viewports: desktop 1440 x 900; mobile Settings/Profile spot-check.
|
||||
|
||||
Route sequence:
|
||||
|
||||
1. `/settings`
|
||||
2. `/settings/vault`
|
||||
3. `/approvals`
|
||||
4. backup/restore section
|
||||
5. `/team`
|
||||
6. `/payment-success`
|
||||
7. `/payment-cancelled`
|
||||
8. admin web evidence if T15 not deferred
|
||||
|
||||
Required screenshots:
|
||||
|
||||
- `settings-billing.png`
|
||||
- `vault-secret-hidden.png`
|
||||
- `approvals-list.png`
|
||||
- `approval-revoke-confirmation.png`
|
||||
- `backup-create.png`
|
||||
- `restore-confirmation-result.png`
|
||||
- `team-governance.png`
|
||||
- `payment-success.png`
|
||||
- `payment-cancelled.png`
|
||||
|
||||
Must prove:
|
||||
|
||||
- Active billing copy is Solo/Teams/Enterprise.
|
||||
- Secret values are not exposed unintentionally.
|
||||
- Restore/revoke/delete use in-app confirmation and visible result states.
|
||||
- Checkout success/cancel recovery has a clear next action.
|
||||
|
||||
Current blockers:
|
||||
|
||||
- T3, T7, T10, T11, T13, T14, T15 unless deferred.
|
||||
|
||||
## Persona 5: Mobile Executive
|
||||
|
||||
State bundle:
|
||||
|
||||
- Account mode: accountless local.
|
||||
- Billing tier: Solo unless Team account view is intentionally sampled.
|
||||
- UI disclosure tier: `simple`, with `power` as route-discovery comparison only.
|
||||
- Model state: no-model or verified-model banner must fit.
|
||||
- Data state: at least one workspace and some memory.
|
||||
- Offline/error state: overlay close and readable empty/error state.
|
||||
- Viewport: 390 x 844 primary; optional tablet 1024 x 768.
|
||||
|
||||
Route sequence:
|
||||
|
||||
1. mobile `/home`
|
||||
2. mobile `/settings`
|
||||
3. mobile `/settings/profile`
|
||||
4. mobile `/memory`
|
||||
5. mobile workspace chat
|
||||
6. Command Center, Workspace Switcher, Notification Inbox, or Create Workspace open/close, depending on the selected mobile path
|
||||
7. theme/profile/billing controls
|
||||
|
||||
Required screenshots:
|
||||
|
||||
- `mobile-home.png`
|
||||
- `mobile-onboarding-welcome.png`
|
||||
- `mobile-onboarding-profile.png`
|
||||
- `mobile-settings-general.png`
|
||||
- `mobile-settings-billing.png`
|
||||
- `mobile-settings-models.png`
|
||||
- `mobile-profile.png`
|
||||
- `mobile-memory.png`
|
||||
- `mobile-chat.png`
|
||||
- `mobile-overlay-open.png`
|
||||
- `mobile-overlay-closed.png`
|
||||
|
||||
Must prove:
|
||||
|
||||
- No horizontal overflow.
|
||||
- No clipped primary controls.
|
||||
- First-run Profile primary Continue is visible, sticky, or clearly reachable.
|
||||
- Critical visible controls stay in-bounds even when document-level scroll width is clean.
|
||||
- Touch targets and focus states are visible.
|
||||
- Overlay does not trap scroll/focus after close.
|
||||
- Required overlays expose an accessible name or landmark and named primary icon-only actions.
|
||||
|
||||
Current blockers:
|
||||
|
||||
- T2, T3, T10, T11, T12, including first-run onboarding evidence.
|
||||
|
||||
## Deferral Rules
|
||||
|
||||
Deferrals are allowed during analysis, but a final 9/10 claim needs the user to explicitly approve them.
|
||||
|
||||
Each deferral must include:
|
||||
|
||||
```text
|
||||
Ticket:
|
||||
Surface:
|
||||
Persona affected:
|
||||
Reason deferred:
|
||||
Why it does not affect this score:
|
||||
Evidence still collected:
|
||||
Expiry / revisit trigger:
|
||||
```
|
||||
|
||||
No implicit deferrals. If a persona journey touches T13-T19 and the gate is not fixed/evidenced, the score remains capped until the user scopes it out.
|
||||
|
||||
## Scorecard Template
|
||||
|
||||
```text
|
||||
Persona:
|
||||
Run id:
|
||||
Date:
|
||||
Current commit:
|
||||
Evidence folder:
|
||||
|
||||
State bundle:
|
||||
- Account mode:
|
||||
- Billing tier:
|
||||
- UI disclosure tier:
|
||||
- Model state:
|
||||
- Data state:
|
||||
- Offline/error state:
|
||||
- Viewport:
|
||||
- Non-main gate decisions:
|
||||
|
||||
Routes covered:
|
||||
|
||||
Console status:
|
||||
|
||||
Screenshots inspected:
|
||||
|
||||
Score:
|
||||
- Functional completion /2:
|
||||
- Flow, IA, discoverability /2:
|
||||
- Trust, error handling, recovery /2:
|
||||
- Visual, accessibility, responsive quality /2:
|
||||
- Performance and polish /1:
|
||||
- Memory, personalization, domain fit /1:
|
||||
- Total /10:
|
||||
|
||||
Caps applied:
|
||||
|
||||
Verdict:
|
||||
|
||||
Top corrections:
|
||||
```
|
||||
|
||||
## Final Pass Criteria
|
||||
|
||||
The goal is still incomplete until:
|
||||
|
||||
- Five scorecards are filled from current post-fix evidence.
|
||||
- Every persona total is at least 9/10.
|
||||
- No score cap below 9 remains active.
|
||||
- Route manifest rows for judged routes are Strong or explicitly deferred.
|
||||
- State bundles are attached for all five personas.
|
||||
- T13/T14/T15/T16/T17/T18/T19 are evidenced or explicitly deferred.
|
||||
- The correction register has no open P0 and no unapproved judge-blocking P1.
|
||||
514
docs/audits/2026-07-08-five-persona-judge-scorecards.md
Normal file
514
docs/audits/2026-07-08-five-persona-judge-scorecards.md
Normal file
@@ -0,0 +1,514 @@
|
||||
# Five-Persona UX Judge Scorecards
|
||||
|
||||
Companion artifacts:
|
||||
|
||||
- `docs/audits/2026-07-08-complete-ux-usage-audit.md`
|
||||
- `docs/audits/2026-07-08-ux-route-scenario-manifest.md`
|
||||
- `docs/audits/2026-07-08-ux-state-failure-scenario-matrix.md`
|
||||
- `docs/audits/2026-07-08-ux-non-main-surface-scope.md`
|
||||
- `docs/audits/2026-07-08-ux-correction-register.md`
|
||||
- `docs/audits/2026-07-08-five-persona-judge-runbook.md`
|
||||
- `docs/audits/2026-07-08-source-inventory-consistency-audit.md`
|
||||
- `docs/audits/2026-07-08-state-failure-t12-analysis.md`
|
||||
- `docs/audits/2026-07-08-mobile-executive-t2-t12-analysis.md`
|
||||
- `docs/audits/2026-07-08-runtime-a11y-t10-analysis.md`
|
||||
- `docs/audits/2026-07-08-first-run-onboarding-t1-t2-t12-analysis.md`
|
||||
- `docs/superpowers/plans/2026-07-08-ux-phase-1-corrections.md`
|
||||
|
||||
Purpose: define the final judge gate before any claim that Waggle OS is 9/10 across five personas. These scorecards extend the existing `tests/vision/personas.spec.ts` harness. That harness proves live persona chat, persistence, screenshots, and no cross-persona prompt leakage; it does not yet score the full route/UX rubric.
|
||||
|
||||
Final status (2026-07-13): the fixed-rubric in-product judge run is complete. All five personas score at least 9/10 with no score cap triggered. The historical pre-fix findings and table below are retained as the audit trail; the final table at the end of this document supersedes them. Public launch availability is reported separately and is not silently counted as passing: `waggle-os.ai` is currently unresolved, and signed production distribution plus credential-dependent external-provider smokes remain release gates.
|
||||
|
||||
Execution protocol: use `docs/audits/2026-07-08-five-persona-judge-runbook.md` after the blocking tickets are fixed or explicitly deferred. The runbook is the authoritative checklist for state bundles, screenshots, score caps, and deferral records.
|
||||
|
||||
Phase 1 status update: the approved Phase 1 implementation is complete and verified. The standard cockpit lane now has clean accountless Clerk/CSP behavior, passing mobile Settings and mobile first-run onboarding checks, active Solo/Teams/Enterprise copy cleanup, passing `Ctrl+Shift+N` and Workspace Switcher route behavior, updated visual baselines, and codified thin-route evidence. The full combined browser gate passed 156/156 on port `34150`. Phase 2 has started with partial overlay fixes: Notification Inbox and Create Workspace primary/subdialog contracts now have named dialog/close coverage, custom-template delete uses an in-app confirmation, the sampled 390 x 844 Create Workspace hierarchy prioritizes required setup before optional templates, Context Rail has a labelled complementary contract, Onboarding Tooltips has an explicit non-modal Escape-dismiss contract, and tier-modal close labels are named. These remove some overlay caps, but the final 9/10 gate is still blocked by remaining trust-critical dialogs, screenshot/state refresh, broader runtime accessibility, and T13-T19 non-main evidence unless those are fixed or explicitly deferred.
|
||||
|
||||
## Non-Negotiable Gate
|
||||
|
||||
Do not run the final scoring pass until all are true:
|
||||
|
||||
1. No open P0 findings in the main audit.
|
||||
2. Phase 1 verification lane passes or has explicitly approved visual baseline updates. Current status: passed 156/156 in the combined browser gate on 2026-07-08.
|
||||
3. Route manifest has an evidence owner for every registered route and major overlay.
|
||||
4. State/failure matrix has an evidence owner or approved deferral for each persona's required state bundle.
|
||||
5. Standard browser lane has zero critical app/auth/CSP console errors.
|
||||
6. Mobile Settings, first-run onboarding Profile, billing/profile, Home, Memory, Chat, and the selected overlay path have current 390 px screenshots plus visible element-bounds checks; focused Create Workspace bounds now pass, but document-level overflow alone is not enough.
|
||||
7. No active user-facing Pro upgrade copy remains outside explicit legacy billing servicing.
|
||||
8. Trust-critical destructive flows use in-app confirmation/result states.
|
||||
9. Runtime T10 axe/DOM findings and shell-overlay semantics/close findings on judge routes are fixed or explicitly capped/deferred.
|
||||
10. Public launch funnel, desktop wrapper, utility, hook, Browser Companion extension, developer/substrate, ops/deployment, CI, benchmark, and judging gates have evidence, or the user explicitly defers T13/T14/T15/T16/T17/T18/T19 from the five-persona score.
|
||||
|
||||
If any item fails, judges can still provide feedback, but their score is advisory and cannot satisfy the goal.
|
||||
|
||||
## Scoring Model
|
||||
|
||||
Each persona scores 10 points:
|
||||
|
||||
| Dimension | Points | Judge asks |
|
||||
|---|---:|---|
|
||||
| Functional completion | 2 | Did the route/flow complete without broken state, dead end, or hidden dependency? |
|
||||
| Flow, IA, and discoverability | 2 | Did the next action feel obvious without reading docs? Was the route in the right place? |
|
||||
| Trust, error handling, and recovery | 2 | Were permissions, data consequences, pricing, model state, and recovery clear? |
|
||||
| Visual, accessibility, and responsive quality | 2 | Did it feel designed, readable, keyboardable, and usable on required viewport(s)? |
|
||||
| Performance and polish | 1 | Did it load and respond with no distracting lag, flicker, warnings, or noisy states? |
|
||||
| Memory, personalization, and domain fit | 1 | Did Waggle remember/use context in a way that made the experience meaningfully better? |
|
||||
|
||||
Pass rules:
|
||||
|
||||
- Every persona must score at least 9/10.
|
||||
- No dimension may score below 8/10 when normalized to a 10-point scale.
|
||||
- Any critical console error caps the affected persona at 7/10.
|
||||
- Any blocked primary route caps the affected persona at 6/10.
|
||||
- Any severe mobile clipping/overflow in a required mobile journey caps the affected persona at 7/10.
|
||||
- Any selected overlay that opens but cannot close in the required persona path caps the affected persona at 7/10.
|
||||
- Any selected overlay with no accessible name/landmark or unnamed primary icon-only actions caps the affected persona at 8/10 unless explicitly deferred.
|
||||
- Native browser dialog in a trust-critical step caps that persona at 8/10.
|
||||
- Critical axe finding in a persona primary route caps that persona at 8/10; serious keyboard access findings cap at 8/10 unless explicitly deferred from that persona's route.
|
||||
|
||||
Evidence required for every scorecard:
|
||||
|
||||
- Route list covered.
|
||||
- State bundle covered: account mode, billing tier, disclosure tier, model state, data state, offline/error state, and viewport.
|
||||
- T12 focused supplement checked for current state-slice evidence, native-dialog caps, and persona bundle corrections.
|
||||
- First-run onboarding supplement checked for clean-data console health, mobile Profile bounds, import consequence clarity, and first-task handoff behavior.
|
||||
- Non-main gate decision: T13/T14/T15/T16/T17/T18/T19 evidence attached or explicitly deferred.
|
||||
- Evidence folder from the judge runbook.
|
||||
- Screenshots inspected.
|
||||
- Console status.
|
||||
- Failing or flaky tests relevant to the persona.
|
||||
- Score per dimension.
|
||||
- Free-text verdict: pass, advisory pass, fail.
|
||||
- Top 3 remaining corrections, if any.
|
||||
|
||||
## Persona 1: Solo Founder
|
||||
|
||||
Profile:
|
||||
|
||||
- Maya, solo founder, pre-revenue, 4 months runway.
|
||||
- Wants one clear next move and hates re-explaining context.
|
||||
- Low patience for setup friction.
|
||||
|
||||
State bundle to capture:
|
||||
|
||||
- Account mode: accountless local.
|
||||
- Billing tier: Solo / `FREE`.
|
||||
- UI disclosure tier: `simple` first; `power` only for route evidence if needed.
|
||||
- Model state: no-model recovery plus working-model or skipped-LLM explanation.
|
||||
- Data state: fresh install, no workspace first, then one created workspace.
|
||||
- Offline/error state: accountless Clerk/CSP lane.
|
||||
- Viewport: desktop 1440 x 900 plus mobile Home/Profile spot-check.
|
||||
- Non-main gate decisions: T13/T14 deferred or evidenced if launch/desktop flows enter this score.
|
||||
|
||||
Primary journey:
|
||||
|
||||
1. Start from `/auth` in accountless local-first mode.
|
||||
2. Complete or bypass first-run onboarding.
|
||||
3. Land on `/home`.
|
||||
4. Use Home Start Here to open or create a workspace.
|
||||
5. Send first chat asking for this week's one focus.
|
||||
6. Add runway constraint and verify Waggle can reuse that context.
|
||||
7. Return to Home and see a logical next action.
|
||||
|
||||
Required routes and overlays:
|
||||
|
||||
- `/auth`
|
||||
- Onboarding Wizard
|
||||
- `/home`
|
||||
- `/workspaces`
|
||||
- `/workspaces/:workspaceId/chat`
|
||||
- Settings model gate or model setup affordance
|
||||
- Workspace Switcher if no workspace exists
|
||||
|
||||
Evidence to collect:
|
||||
|
||||
- Desktop screenshots: auth/accountless, clean-data onboarding steps, Home, workspace chat, returned Home.
|
||||
- Mobile screenshots: first-run Welcome and Profile at 390 x 844, with primary action bounds checked.
|
||||
- Console summary: no Clerk/CSP errors in accountless mode.
|
||||
- Transcript artifact showing context persistence or clear explanation of memory behavior.
|
||||
- Route manifest rows for Auth, Home, Workspaces, Workspace.
|
||||
|
||||
Automatic fail triggers:
|
||||
|
||||
- Accountless local mode shows Clerk load errors.
|
||||
- First-run onboarding emits Clerk/CSP console errors.
|
||||
- Mobile onboarding hides the primary action in the first-run Profile step when mobile is in scope.
|
||||
- First useful action is unclear from Home.
|
||||
- Chat cannot accept first message or silently depends on unavailable LLM.
|
||||
- Pro copy appears in the journey.
|
||||
|
||||
Corrections that must land before this judge can pass:
|
||||
|
||||
- T1 local auth/CSP/accountless health is closed for the standard accountless lane.
|
||||
- T2 first-run mobile onboarding primary-action reachability is closed for the codified 390 px check.
|
||||
- T3 Solo/Teams/Enterprise copy cleanup is closed for active Phase 1 surfaces.
|
||||
- T4 shortcut/workspace context is closed for the codified `Ctrl+Shift+N` lane.
|
||||
- T11 route evidence owner is closed for the Phase 1 thin-route shell smoke; deeper state evidence remains.
|
||||
|
||||
Expected 9/10 behavior:
|
||||
|
||||
- Maya understands what to do within 10 seconds of landing on Home.
|
||||
- The app helps her move from broad anxiety to one concrete workspace/chat action.
|
||||
- Memory behavior is honest and useful, not vague marketing copy.
|
||||
|
||||
## Persona 2: Researcher
|
||||
|
||||
Profile:
|
||||
|
||||
- Chen, meticulous researcher validating persistent memory and provenance.
|
||||
- Wants evidence, not vibes.
|
||||
- Tolerates density if the information architecture is trustworthy.
|
||||
|
||||
State bundle to capture:
|
||||
|
||||
- Account mode: accountless or authenticated, but declared.
|
||||
- Billing tier: Solo unless Teams memory/governance is intentionally tested.
|
||||
- UI disclosure tier: `power`.
|
||||
- Model state: working or skipped-LLM with memory UI focus.
|
||||
- Data state: populated memory plus empty/no-result state, including sampled slow Memory, large Memory, and Timeline/Event states.
|
||||
- Offline/error state: missing source, failed export, or trust/destructive recovery path.
|
||||
- Viewport: desktop 1440 x 900; mobile Memory spot-check if scored.
|
||||
- Non-main gate decisions: T19 evidenced or explicitly deferred if browser capture enters this score.
|
||||
|
||||
Primary journey:
|
||||
|
||||
1. Open `/memory`.
|
||||
2. Search memory or inspect available memory records.
|
||||
3. Open memory trust/provenance detail.
|
||||
4. Visit wiki/timeline/evolution-related views.
|
||||
5. Attempt export or delete/archive trust flow.
|
||||
6. Return to chat and ask whether memory is durable versus long context.
|
||||
|
||||
Required routes and overlays:
|
||||
|
||||
- `/memory/:mindScope?`
|
||||
- Memory trust/manage overlays
|
||||
- Wiki tab
|
||||
- Timeline/evolution tabs or `/settings/timeline`
|
||||
- Workspace chat
|
||||
- Native prompt replacements for wiki/export/delete
|
||||
|
||||
Evidence to collect:
|
||||
|
||||
- Screenshots: Memory overview, search result, trust/provenance detail, wiki/timeline state, confirmation modal.
|
||||
- Console summary.
|
||||
- Transcript or UI text explaining memory mechanism honestly.
|
||||
- Route manifest rows for Memory and Timeline.
|
||||
- Current partial evidence: Artifact permanent delete, Memory Center delete/GDPR erase/allow re-import, and Wiki Obsidian/Notion exports now have component coverage and rendered `J3e`/`J3f`/`J3g` evidence for in-app confirmations/forms. The five-persona bundle also covers `memory-slow-list` with a delayed Memory API and loading status, `memory-large-list` with 200 mocked memories, `timeline-large-events` with 360 mocked events, and `wiki-export-obsidian-failure` with branded `Export Failed` copy after a mocked `500`.
|
||||
|
||||
Automatic fail triggers:
|
||||
|
||||
- Broader Notion/export variants remain open; Artifact permanent delete, Memory Center delete/erase/re-import, Wiki export destinations, and one rendered Wiki export-failure path are fixed for the sampled paths.
|
||||
- Search/provenance route shows blank or unexplained empty state.
|
||||
- Long memory titles break layout.
|
||||
- App implies memory is magic without explaining limits.
|
||||
|
||||
Corrections that must land before this judge can pass:
|
||||
|
||||
- T5 approved baseline update for Memory after the fresh classification note.
|
||||
- T7 trust-critical dialogs.
|
||||
- T10 form/accessibility hygiene where memory forms are touched.
|
||||
- T11 route evidence owner.
|
||||
|
||||
Expected 9/10 behavior:
|
||||
|
||||
- Chen can understand what is stored, why it is trusted, where it came from, and how to correct/remove it.
|
||||
- Empty states and provenance states are credible, not decorative.
|
||||
|
||||
## Persona 3: Engineer / Power User
|
||||
|
||||
Profile:
|
||||
|
||||
- Sam, senior engineer and agent wrangler.
|
||||
- Wants keyboard speed, tool clarity, logs, and proof the product is not a chatbot wrapper.
|
||||
- Low tolerance for flaky tests or hidden network dependency.
|
||||
|
||||
State bundle to capture:
|
||||
|
||||
- Account mode: accountless local.
|
||||
- Billing tier: Solo, with Teams-only surfaces evidenced or deferred.
|
||||
- UI disclosure tier: `power` or `admin`.
|
||||
- Model state: local/no-LLM plus working-provider lane if chat is scored.
|
||||
- Data state: one workspace, detected or undetected tools, MCP catalog present.
|
||||
- Offline/error state: marketplace local-only, delayed/large Agents roster, and tool/hook unavailable states.
|
||||
- Viewport: desktop 1440 x 900 plus keyboard-only path.
|
||||
- Non-main gate decisions: T15/T16/T17/T18 evidenced or explicitly deferred for utility, hook, developer, and ops surfaces.
|
||||
|
||||
Primary journey:
|
||||
|
||||
1. Start on `/home`.
|
||||
2. Open Command Center with `Ctrl+K` and navigate to an app.
|
||||
3. Use `Ctrl+Shift+N` to open active workspace chat.
|
||||
4. Open `/launcher` and verify tool/hook state.
|
||||
5. Open `/mcps`, inspect installed/custom MCP server flows.
|
||||
6. Open `/files` and inspect file actions.
|
||||
7. Open `/settings/events` for logs.
|
||||
|
||||
Required routes and overlays:
|
||||
|
||||
- Command Center
|
||||
- `/workspaces/:workspaceId/chat`
|
||||
- `/launcher`
|
||||
- `/mcps`
|
||||
- `/files`
|
||||
- `/settings/events`
|
||||
- Workspace Switcher
|
||||
|
||||
Evidence to collect:
|
||||
|
||||
- Screenshots: Command Center search, chat after shortcut, Launcher, MCP Hub, Files, Events.
|
||||
- Keyboard interaction log for `Ctrl+K`, `Ctrl+Shift+N`, Escape close.
|
||||
- Console summary.
|
||||
- Route manifest rows for Launcher, MCP Hub, Files, Events.
|
||||
- Codified route smoke evidence: `J-route-coverage: priority thin routes render meaningful shells` passed for `/launcher`, `/launcher?watch=1`, `/mcps`, and `/files` on 2026-07-08.
|
||||
|
||||
Automatic fail triggers:
|
||||
|
||||
- `Ctrl+Shift+N` does not open the expected chat route.
|
||||
- Workspace Switcher blocks unrelated navigation.
|
||||
- Marketplace/MCP/Launcher depends on live external sync in the standard audit lane.
|
||||
- Tool output renders broken JSON or unexplained fallback.
|
||||
|
||||
Corrections that must land before this judge can pass:
|
||||
|
||||
- T4 shortcut and Workspace Switcher route contract is closed for the standard browser lane.
|
||||
- T6 marketplace determinism.
|
||||
- T8 performance and payload polish if startup feels heavy.
|
||||
- T11 route coverage for Launcher/MCP/files has shell-level smoke coverage; keep deeper hook/MCP/file interaction states in Sam's evidence bundle.
|
||||
- T16 hook lifecycle if Launcher/tool management is included in Sam's final score.
|
||||
|
||||
Current partial evidence: the five-persona bundle covers `agents-slow-list` with a delayed `/api/agents` response, aria-busy `Loading agents` status, and final `40 agents` plus `Bulk Agent 000` roster proof; it also covers `agents-large-list` with 180 mocked agents and 0 visible overflow. These are accountless/no-LLM sampled state proofs, not substitutes for packaged hook lifecycle or authenticated Team evidence.
|
||||
|
||||
Expected 9/10 behavior:
|
||||
|
||||
- Sam can operate primarily by keyboard, sees real tool/hook state, and trusts logs/error states.
|
||||
- The product feels like an agent OS, not a pile of screens.
|
||||
|
||||
## Persona 4: Team Admin / Security Reviewer
|
||||
|
||||
Profile:
|
||||
|
||||
- Priya, nontechnical but accountable team/product admin.
|
||||
- Needs plain language, billing confidence, governance, vault, backup, approvals.
|
||||
- Cares about not breaking data or exposing secrets.
|
||||
|
||||
State bundle to capture:
|
||||
|
||||
- Account mode: authenticated, or accountless with mocked Teams-tier billing/admin state and limitations declared.
|
||||
- Billing tier: Teams for team/admin surfaces, Solo for gating comparison, legacy Pro collapsed to Solo where relevant; real Team server membership must be evidenced or deferred separately.
|
||||
- UI disclosure tier: `professional` and `admin`.
|
||||
- Model state: not central unless Settings model copy is inspected.
|
||||
- Data state: vault item, approval grant, backup metadata, team governance state, and unlocked Team settings state.
|
||||
- Offline/error state: backup failure, restore failure, and checkout recovery copy.
|
||||
- Viewport: desktop 1440 x 900 plus mobile Settings/Profile spot-check.
|
||||
- Non-main gate decisions: T13/T14/T15 evidenced or explicitly deferred for launch, desktop, and admin/utility paths.
|
||||
|
||||
Primary journey:
|
||||
|
||||
1. Open `/settings` billing/general/model sections.
|
||||
2. Visit `/settings/vault`.
|
||||
3. Add or inspect a secret without revealing value.
|
||||
4. Visit `/approvals` and review/revoke grants.
|
||||
5. Use backup create/restore flow.
|
||||
6. Visit `/team` governance.
|
||||
7. Exercise payment success and payment cancelled recovery.
|
||||
|
||||
Required routes and overlays:
|
||||
|
||||
- `/settings`
|
||||
- `/settings/vault`
|
||||
- `/approvals`
|
||||
- Backup section in Settings or Backup app surface
|
||||
- `/team`
|
||||
- `/payment-success`
|
||||
- `/payment-cancelled`
|
||||
- Erase Data dialog if destructive data flow is inspected
|
||||
|
||||
Evidence to collect:
|
||||
|
||||
- Screenshots: billing copy, active Team billing state, unlocked Team settings state, vault, approval list, backup flow, team governance, payment success/cancelled recovery.
|
||||
- Console summary.
|
||||
- Copy scan: no active Pro upgrade language except explicit legacy billing state.
|
||||
- Confirmation/result-state screenshots for restore/revoke/delete.
|
||||
- Current partial evidence: Approvals revoke-all now has component coverage, rendered `/approvals` `J3d` evidence for an in-app confirmation, and five-persona Team Admin bundle evidence via `approvals-revoke-all-grants`; Artifact permanent delete also has rendered `J3e` evidence; Settings telemetry clear/backup failure/restore success have `settings-trust.test.tsx` and rendered `J3h` evidence; standalone `BackupApp` restore, Automation delete, compliance template delete, and admin-web member removal have focused component evidence. The five-persona Team Admin bundle now covers `approvals-revoke-all-grants`, `billing-team-active-state` with mocked `TEAMS` tier and visible `Waggle Team` / `$49/mo per seat` / `Manage subscription` copy, `team-settings-unlocked-state` with visible Team Server URL/Auth Token/trust-warning copy, `billing-checkout-success-return` with a mocked Team checkout sync, `billing-checkout-cancel-return` with visible `Checkout was cancelled` / `No charge was made` recovery copy, `billing-checkout-unavailable`, `backup-create-failure`, and `backup-restore-failure`, with 0 critical console/page/network failures and 0 visible overflow. Current high-confidence production native-dialog scan is clean.
|
||||
- Codified route smoke evidence: `J-route-coverage` passed for `/payment-success` and `/payment-cancelled`, including redirect to `/settings?tab=billing`, on 2026-07-08; the refreshed route smoke on 2026-07-09 still passes after adding the `checkout=cancelled` marker.
|
||||
|
||||
Automatic fail triggers:
|
||||
|
||||
- Billing copy says Pro as an active tier.
|
||||
- Secret values are exposed unintentionally.
|
||||
- Current known production native dialog scan is clean; remaining risk is uncodified less-common destructive paths, failure-state depth, and focus/keyboard proof rather than known browser-native alert/confirm calls.
|
||||
- Payment cancelled lacks visible no-charge recovery copy.
|
||||
|
||||
Corrections that must land before this judge can pass:
|
||||
|
||||
- T3 pricing/gating copy.
|
||||
- T7 trust-critical dialogs.
|
||||
- T10 form/accessibility hygiene.
|
||||
- T11 route coverage and the five-persona Team Admin bundle now cover payment cancelled, payment success return, mocked active Team billing, and unlocked Team settings states; real authenticated Team server/admin states still need persona screenshots or deferral.
|
||||
|
||||
Expected 9/10 behavior:
|
||||
|
||||
- Priya can tell what plan she is on, what actions are risky, and what happened after each admin action.
|
||||
- The interface feels safe, not scary.
|
||||
|
||||
## Persona 5: Mobile Executive
|
||||
|
||||
Profile:
|
||||
|
||||
- Mobile or tablet user checking status between meetings.
|
||||
- Does not want to configure everything, but needs Home, Settings, Memory, billing/profile, and theme to work.
|
||||
- Sensitive to clipping, tiny targets, and scroll traps.
|
||||
|
||||
State bundle to capture:
|
||||
|
||||
- Account mode: accountless local.
|
||||
- Billing tier: Solo unless Team account view is intentionally sampled.
|
||||
- UI disclosure tier: `simple`, with `power` only as a route-discovery comparison.
|
||||
- Model state: no-model or verified-model banner must fit.
|
||||
- Data state: at least one workspace and some memory.
|
||||
- Offline/error state: overlay close plus readable empty/error state.
|
||||
- Viewport: 390 x 844 primary; optional tablet 1024 x 768.
|
||||
- Non-main gate decisions: T13/T14/T19 deferred or evidenced if launch, desktop, or browser-capture flows enter this mobile score.
|
||||
|
||||
Primary journey:
|
||||
|
||||
1. Set viewport to 390 x 844.
|
||||
2. Open `/home`.
|
||||
3. Open `/settings`.
|
||||
4. Inspect billing/general/model/profile areas.
|
||||
5. Open `/settings/profile`.
|
||||
6. Open `/memory`.
|
||||
7. Open workspace chat and send or type a short message.
|
||||
8. Open Command Center or Workspace Switcher and close it with keyboard/touch equivalent.
|
||||
|
||||
Required routes and overlays:
|
||||
|
||||
- Mobile `/home`
|
||||
- Mobile `/settings`
|
||||
- Mobile `/settings/profile`
|
||||
- Mobile `/memory`
|
||||
- Mobile workspace chat
|
||||
- Command Center or Workspace Switcher
|
||||
- Billing/profile/theme controls
|
||||
|
||||
Evidence to collect:
|
||||
|
||||
- Mobile screenshots for every route above.
|
||||
- Mobile first-run onboarding Welcome/Profile screenshots if the persona starts from a clean install.
|
||||
- Horizontal overflow check.
|
||||
- Critical visible control bounds check, because the fresh mobile smoke found clipped controls without document-level overflow.
|
||||
- Focus/keyboard/touch target notes.
|
||||
- Console summary.
|
||||
|
||||
Automatic fail triggers:
|
||||
|
||||
- Settings remains squeezed two-pane layout at 390 px.
|
||||
- First-run onboarding hides the primary Continue action on the Profile step.
|
||||
- Any primary billing/profile/model control is clipped or unreachable.
|
||||
- Overlay traps scroll/focus.
|
||||
- Selected overlay cannot close by keyboard/touch path.
|
||||
- Selected overlay lacks an accessible name/landmark or leaves primary icon-only controls unnamed. Current update: Notification Inbox, Create Workspace primary/subdialog contracts, Context Rail, Onboarding Tooltips, and tier close controls have focused contract coverage; less common rendered states still need evidence.
|
||||
- Create Workspace returns to a template-first mobile hierarchy in any judged path not covered by the focused 390 x 844 evidence.
|
||||
- Text overlaps or becomes unreadable.
|
||||
|
||||
Corrections that must land before this judge can pass:
|
||||
|
||||
- T2 mobile Settings responsive layout is closed for general, models, billing, and profile in the codified 390 px check.
|
||||
- T2 first-run onboarding responsive layout is closed for the codified mobile Profile reachability check.
|
||||
- T3 pricing/gating copy is closed for active Phase 1 surfaces.
|
||||
- T10 form/accessibility hygiene. Current update: core shell overlay semantics and sampled Create Workspace mobile hierarchy are partially fixed; broader T10 remains open.
|
||||
- T11 mobile route evidence.
|
||||
- T12 mobile state bundle, including selected overlay close evidence.
|
||||
|
||||
Expected 9/10 behavior:
|
||||
|
||||
- The app feels intentionally responsive, not merely shrunken.
|
||||
- Mobile user can inspect and make small changes without fighting layout.
|
||||
|
||||
## Judge Run Protocol
|
||||
|
||||
Preparation:
|
||||
|
||||
1. Build the app from current source.
|
||||
2. Start a fresh-port local server with clean data unless testing return-state memory.
|
||||
3. Run standard verification from the main audit.
|
||||
4. Run or update route manifest evidence.
|
||||
5. Capture required screenshots per persona.
|
||||
6. Run `tests/vision/personas.spec.ts` only in a real-LLM lane, because it is not a no-LLM smoke test.
|
||||
|
||||
Scoring:
|
||||
|
||||
1. Fill the score table for one persona at a time.
|
||||
2. Record exact blockers and route evidence.
|
||||
3. Apply score caps before subjective scoring.
|
||||
4. If a persona scores below 9, create a correction item or map it to an existing T-ticket.
|
||||
5. Do not average away failures; all five must pass.
|
||||
|
||||
Suggested output table:
|
||||
|
||||
| Persona | Functional /2 | Flow /2 | Trust /2 | Visual+A11y /2 | Perf /1 | Memory fit /1 | Total | Verdict | Blockers |
|
||||
|---|---:|---:|---:|---:|---:|---:|---:|---|---|
|
||||
| Solo founder | Not Run | Not Run | Blocked | Not Run | Not Run | Not Run | Not Run | Pre-fix blocked | P0-1, P0-3 |
|
||||
| Researcher | Not Run | Not Run | Blocked | Blocked | Not Run | Not Run | Not Run | Pre-fix blocked | P0-4, P1-1 |
|
||||
| Engineer | Blocked | Blocked | Not Run | Not Run | Not Run | Not Run | Not Run | Pre-fix blocked | P0-5, P0-6 |
|
||||
| Team admin | Not Run | Not Run | Blocked | Not Run | Not Run | Not Run | Not Run | Pre-fix blocked | P0-1, P0-3, P1-1 |
|
||||
| Mobile executive | Not Run | Not Run | Not Run | Blocked | Not Run | Not Run | Not Run | Pre-fix blocked | P0-2, P0-7 |
|
||||
|
||||
## Implementation Backlog Mapping
|
||||
|
||||
| Scorecard blocker | Main ticket |
|
||||
|---|---|
|
||||
| Accountless Clerk/CSP console errors | T1 |
|
||||
| Mobile Settings squeezed/clipped | T2 |
|
||||
| Mobile first-run onboarding primary action hidden | T2/T12 |
|
||||
| Pro copy in active flows | T3 |
|
||||
| `Ctrl+Shift+N` mismatch and overlay trap | T4 |
|
||||
| Visual baselines classified as stale but not approved/updated | T5 |
|
||||
| Marketplace live sync and flaky search | T6 |
|
||||
| Native confirm/alert/prompt | T7 |
|
||||
| Heavy initial payload or delayed first meaningful UI | T8 |
|
||||
| Unknown local model cost semantics | T9 |
|
||||
| Labels/focus/icon-only buttons/noisy warnings | T10 |
|
||||
| Thin route coverage and judge harness gaps | T11 |
|
||||
| Missing state/failure bundle declaration | T12 |
|
||||
| Shell overlay semantics, close behavior, and Create Workspace mobile hierarchy | T10/T12 |
|
||||
| Canonical launch domains do not resolve; download has no releases; checkout, legal, and deploy gates remain open despite fresh localhost rendered evidence | T13 |
|
||||
| Desktop wrapper tray source is narrowed, but packaged tray, installer, update, and sidecar startup evidence is still missing | T14 |
|
||||
| Admin web, CLI launcher, Waggle CLI, legacy memory MCP, and hive-mind CLI still have blocking rendered/admin and built-entry issues; marketplace CLI first-command path is locally fixed | T15 |
|
||||
| AI-tool hook lifecycle has partial rendered Launcher evidence but still lacks real-tool/package invocation proof and clear result/unsupported-output UX | T16 |
|
||||
| Developer API, background worker, and substrate verification evidence missing | T17 |
|
||||
| Ops, deployment, CI, benchmark, and judging evidence missing | T18 |
|
||||
| Browser Companion auth/background save, popup keyboard/focus/Enter save, direct Save page click, restricted-page disabled-state recovery, stable packaged-ID pairing, Memory search provenance, existing chat `auto_recall`/catch-up provenance, and rendered Memory UI after secure save are live-proven, but native toolbar-bubble/native context-menu proof remains incomplete; future recall result shapes need evidence if scored | T19 |
|
||||
|
||||
## Final Judge Run - 2026-07-13
|
||||
|
||||
This table scores the product UX itself. Each persona used a declared account,
|
||||
billing, disclosure, model, data, failure, and viewport state bundle. Captures
|
||||
waited for visible accessible loaders and route-specific legacy loading labels
|
||||
to settle, and animations were disabled for deterministic inspection.
|
||||
|
||||
| Persona | Functional /2 | Flow /2 | Trust /2 | Visual+A11y /2 | Perf /1 | Memory fit /1 | Total | Verdict | Blocking corrections |
|
||||
|---|---:|---:|---:|---:|---:|---:|---:|---|---|
|
||||
| Solo founder | 1.9 | 1.9 | 1.8 | 1.9 | 0.9 | 1.0 | **9.4** | Pass | None in scored lane |
|
||||
| Researcher | 1.9 | 1.8 | 2.0 | 1.9 | 0.9 | 1.0 | **9.5** | Pass | None in scored lane |
|
||||
| Engineer / power user | 1.9 | 1.8 | 1.9 | 1.9 | 0.9 | 0.9 | **9.3** | Pass | None in scored lane |
|
||||
| Team admin / security reviewer | 1.8 | 1.8 | 2.0 | 1.8 | 0.9 | 0.8 | **9.1** | Pass | None in scored lane |
|
||||
| Mobile executive | 1.9 | 1.9 | 1.8 | 1.9 | 0.9 | 0.9 | **9.3** | Pass | None in scored lane |
|
||||
|
||||
Pass-rule checks:
|
||||
|
||||
- Lowest persona total: 9.1/10.
|
||||
- Lowest normalized dimension: 8/10.
|
||||
- Critical console errors: 0 across all five bundles.
|
||||
- Page errors: 0 across all five bundles.
|
||||
- Unexpected critical network failures: 0 across all five bundles.
|
||||
- Visible horizontal overflow findings: 0 across route, failure, and overlay captures.
|
||||
- Score caps triggered: none.
|
||||
|
||||
The five bundles exercise 15 primary route states, 25 failure/slow/large-data
|
||||
states, and 3 selected overlays. The run passed 5/5 in Chromium. Representative
|
||||
screenshots were inspected after the run, including settled Home, Memory,
|
||||
Launcher, Approvals, mobile Settings, and mobile Command Center states.
|
||||
|
||||
The detailed current-head evidence and release boundary are recorded in
|
||||
`docs/audits/2026-07-13-final-goal-verification.md`.
|
||||
360
docs/audits/2026-07-08-launch-funnel-t13-analysis.md
Normal file
360
docs/audits/2026-07-08-launch-funnel-t13-analysis.md
Normal file
@@ -0,0 +1,360 @@
|
||||
# T13 Public Launch Funnel UX Analysis
|
||||
|
||||
Status: focused local fixes implemented for signed-out checkout continuation, checkout cancel recovery, the public-site hydration issue badge, legal placeholder/stale-tier copy, the empty-release download dead-end, mobile download label honesty, and the public-site deployment workflow target. Canonical DNS, real signed installer publication, deployed Vercel/DNS proof, deployed Clerk/Stripe proof, and formal legal sign-off remain open launch gates.
|
||||
|
||||
Scope: `apps/www`, public download, public pricing, auth handoff, Stripe checkout handoff, checkout cancel recovery, account redirect, legal/trust pages, and launch deployment.
|
||||
|
||||
## Why This Matters
|
||||
|
||||
The installed cockpit can score well and still fail the complete-product UX goal if a founder, buyer, reviewer, or mobile executive cannot get from the public site to a trustworthy download, account, or checkout path. T13 therefore remains a final-product gate unless the user explicitly scopes the public launch funnel out of the five-persona score.
|
||||
|
||||
## Current Evidence
|
||||
|
||||
| Check | Result | Notes |
|
||||
|---|---|---|
|
||||
| `npm run test -w apps/www -- --reporter=dot` | Pass | Current focused coverage is 7 files / 19 tests: `BrandPersonasCard`, Pricing checkout links/recovery, Stripe checkout route cancel URL, layout hydration contract, legal launch-copy guard, controlled download path, mobile/desktop download label detection, and public-site deployment workflow guard. |
|
||||
| `npx tsc --noEmit --project apps/www/tsconfig.json` | Pass | No TypeScript errors. |
|
||||
| `npm run build:www` | Pass | Next.js 15.5.18 build succeeds. Routes include static public pages plus dynamic `/account`, `/api/stripe/checkout`, `/api/webhooks/stripe`, `/sign-in`, and `/sign-up`. Build warns that the Next.js ESLint plugin is not detected. |
|
||||
| Build/deploy artifact shape | Improved locally | `Test-Path apps/www/dist` = `False`; `Test-Path apps/www/.next` = `True`; `Test-Path apps/www/out` = `False`. `.github/workflows/deploy-www.yml` now uses Vercel production `pull`, `build`, and `deploy --prebuilt --prod` instead of GitHub Pages static artifact upload. Deployed Vercel/DNS proof remains open. |
|
||||
| Live public domain smoke | Fail | Current external refresh on 2026-07-08: all checked `https://waggle-os.ai/*` URLs failed DNS resolution from this environment; `nslookup waggle-os.ai` returned `Non-existent domain`. |
|
||||
| Local prod route/API smoke | Improved | Current post-fix `next start --hostname localhost --port 34205` returned 200 for `/`, `/?checkout=cancelled`, and `/docs/methodology`; signed-out `GET /api/stripe/checkout?tier=teams&billing=monthly` returned 303 to sign-in with a checkout redirect target. Historical `/pricing?checkout=cancelled` remains a non-route, but the app no longer emits it from Stripe cancel recovery. |
|
||||
| Signed-out checkout API/UI smoke | Improved | Pricing now uses the canonical GET checkout route instead of POST, so signed-out users enter the route's auth redirect flow. POST remains a backward-compat JSON shim for older clients. |
|
||||
| Download target | Improved | Public Download CTAs and footer Product > Download now route to `/download`, a controlled status page that explains Windows/macOS installers are being prepared and links to source/contact instead of an empty GitHub Releases page. Mobile/tablet OS detection now keeps CTAs generic instead of labeling iOS/Android as desktop installers. The release workflow now builds packages before Windows/macOS sidecar packaging, but real signed installer publication remains open. Current production smoke returned 200 for `/download` and found no `releases/latest` target in `/` or `/download`. |
|
||||
| Fresh rendered Browser smoke | Improved | In-app Browser verified `http://localhost:34204/?checkout=cancelled#pricing`: pricing rendered, cancelled-checkout recovery notice appeared, monthly Team CTA and retry link used `/api/stripe/checkout?tier=teams&billing=monthly`, annual toggle updated both links to annual, the Next dev issue badge disappeared after the layout fix, and console warnings/errors were empty. Browser DOM snapshot still failed with the known `incrementalAriaSnapshot` mismatch, so evidence used targeted DOM evaluation plus screenshots. |
|
||||
| Web Interface Guidelines lens | Mixed | Latest guideline source checked on 2026-07-08: <https://raw.githubusercontent.com/vercel-labs/web-interface-guidelines/main/command.md>. Public page has skip link, focus-visible outline, reduced-motion handling, semantic sections, explicit image dimensions in key inspected paths, fixed checkout recovery next steps, and guarded legal placeholder copy. Download/release truth, deployed checkout proof, and formal legal sign-off remain outside the local UI copy fix. |
|
||||
|
||||
Local smoke output summary:
|
||||
|
||||
```text
|
||||
/ -> 200
|
||||
/#pricing -> 200
|
||||
/privacy -> 200
|
||||
/terms -> 200
|
||||
/cookies -> 200
|
||||
/eu-ai-act -> 200
|
||||
/sign-in -> 200
|
||||
/sign-up -> 200
|
||||
/docs/methodology -> 200
|
||||
/account -> 307 location=/sign-in
|
||||
/pricing?checkout=cancelled -> 404
|
||||
/methodology -> 404
|
||||
GET checkout signed-out -> 303 location=http://localhost:3426/sign-in?redirect_url=%2Fapi%2Fstripe%2Fcheckout%3Ftier%3Dteams%26billing%3Dmonthly
|
||||
POST checkout signed-out -> 401 body={"message":"Sign in required","signInUrl":"http://localhost:3426/sign-in?redirect_url=%2Fapi%2Fstripe%2Fcheckout%3Ftier%3Dteams%26billing%3Dmonthly"}
|
||||
```
|
||||
|
||||
Live external refresh summary:
|
||||
|
||||
```text
|
||||
https://waggle-os.ai/ -> DNS resolution failed
|
||||
https://waggle-os.ai/#pricing -> DNS resolution failed
|
||||
https://waggle-os.ai/pricing?checkout=cancelled -> DNS resolution failed
|
||||
https://waggle-os.ai/docs/methodology -> DNS resolution failed
|
||||
https://waggle-os.ai/privacy -> DNS resolution failed
|
||||
https://waggle-os.ai/terms -> DNS resolution failed
|
||||
https://waggle-os.ai/cookies -> DNS resolution failed
|
||||
https://waggle-os.ai/eu-ai-act -> DNS resolution failed
|
||||
https://waggle-os.ai/sign-in -> DNS resolution failed
|
||||
https://waggle-os.ai/sign-up -> DNS resolution failed
|
||||
https://waggle-os.ai/account -> DNS resolution failed
|
||||
https://api.github.com/repos/marolinik/waggle-os/releases/latest -> 404
|
||||
https://api.github.com/repos/marolinik/waggle-os/releases -> []
|
||||
```
|
||||
|
||||
Rendered Browser smoke summary:
|
||||
|
||||
```text
|
||||
Build: npm run build:www -> pass; .next dynamic app generated, Next ESLint plugin warning remains
|
||||
Server: npm run start -w apps/www -- --hostname localhost --port 3491
|
||||
Browser page identity: http://localhost:3491/ -> title "Waggle — The AI workspace that remembers"
|
||||
Console health: homepage 0 warnings/errors; pricing checkout-error state 0; mobile menu state 0
|
||||
Desktop screenshot: output/playwright/www-t13-3491/www-t13-home-desktop.png
|
||||
Pricing screenshot: output/playwright/www-t13-3491/www-t13-pricing-checkout-error-desktop.png
|
||||
Mobile screenshots: output/playwright/www-t13-3491/www-t13-home-mobile.png and www-t13-mobile-menu-open.png
|
||||
Summary JSON: output/playwright/www-t13-3491/www-t13-rendered-summary.json
|
||||
/pricing?checkout=cancelled -> 404
|
||||
/methodology -> 404
|
||||
/docs/methodology -> rendered
|
||||
Signed-out pricing CTA -> stays on / and shows only "Sign in required"
|
||||
GET checkout monthly/annual -> 303 to /sign-in with redirect_url
|
||||
Download CTAs -> https://github.com/marolinik/waggle-os/releases/latest
|
||||
External refresh: waggle-os.ai and www.waggle-os.ai NXDOMAIN; GitHub latest release 404; releases list []
|
||||
```
|
||||
|
||||
Post-fix focused evidence:
|
||||
|
||||
```text
|
||||
Pricing route contract:
|
||||
npm run test -w apps/www -- __tests__/Pricing.test.tsx __tests__/stripe-checkout-route.test.ts --reporter=dot
|
||||
-> 2 files / 3 tests passed
|
||||
|
||||
Hydration contract:
|
||||
npm run test -w apps/www -- __tests__/layout.test.tsx --reporter=dot
|
||||
-> 1 file / 1 test passed
|
||||
|
||||
Public-site suite:
|
||||
npm run test -w apps/www -- --reporter=dot
|
||||
-> 7 files / 19 tests passed
|
||||
|
||||
TypeScript:
|
||||
npx tsc --noEmit --project apps/www/tsconfig.json
|
||||
-> pass
|
||||
|
||||
Build:
|
||||
npm run build:www
|
||||
-> pass; .next output generated; /download route included; Next ESLint plugin warning remains
|
||||
|
||||
Rendered Browser:
|
||||
http://localhost:34204/?checkout=cancelled#pricing
|
||||
-> recovery notice visible; monthly Team CTA + retry href = /api/stripe/checkout?tier=teams&billing=monthly
|
||||
-> annual toggle updates both hrefs to /api/stripe/checkout?tier=teams&billing=annual
|
||||
-> no Next issue badge after layout suppressHydrationWarning; no console warnings/errors
|
||||
|
||||
Local production smoke:
|
||||
next start --hostname localhost --port 34205
|
||||
/ -> 200
|
||||
/?checkout=cancelled -> 200
|
||||
/docs/methodology -> 200
|
||||
/api/stripe/checkout?tier=teams&billing=monthly -> 303 to /sign-in?redirect_url=...
|
||||
|
||||
Legal copy guard:
|
||||
npm run test -w apps/www -- __tests__/legal-copy.test.ts --reporter=dot
|
||||
-> 1 file / 1 test passed
|
||||
rg -n "Day-0|\[Day-0 launch date\]|Pro or Teams|to be filled before public launch|\[to be designated" 'apps/www/app/(legal)'
|
||||
-> no matches
|
||||
|
||||
Download path:
|
||||
npm run test -w apps/www -- __tests__/download-path.test.tsx --reporter=dot
|
||||
-> 1 file / 3 tests passed
|
||||
rg -n "releases/latest|https://github.com/marolinik/waggle-os/releases" apps/www/app apps/www/messages/en.json apps/www/__tests__
|
||||
-> no matches
|
||||
next start --hostname localhost --port 34206
|
||||
/download -> 200, contains installer-status copy, no releases/latest target
|
||||
/ -> 200, no releases/latest target
|
||||
|
||||
Deployment workflow:
|
||||
npm run test -w apps/www -- __tests__/deployment-workflow.test.ts --reporter=dot
|
||||
-> 1 file / 1 test passed
|
||||
.github/workflows/deploy-www.yml now uses Vercel production pull/build/deploy and no longer references GitHub Pages or apps/www/dist.
|
||||
|
||||
Desktop release workflow:
|
||||
npx vitest run packages/server/tests/tauri-config.test.ts --reporter=dot
|
||||
-> 1 file / 19 tests passed
|
||||
.github/workflows/release.yml now runs npm run build:packages before both Windows and macOS sidecar bundle steps.
|
||||
```
|
||||
|
||||
## What Is Already Working
|
||||
|
||||
- Homepage IA is coherent: hero, problem, how it works, memory, proof, features, trust, persona brand moment, open source, pricing, and final CTA.
|
||||
- `page.tsx` includes a skip link and a real `<main id="main">`.
|
||||
- Navbar anchors use absolute section URLs, so legal pages can navigate back to homepage sections.
|
||||
- Global CSS provides `:focus-visible`, heading `scroll-margin-top`, and reduced-motion handling.
|
||||
- Pricing copy now uses Solo/Teams/Enterprise in the main pricing component.
|
||||
- The newer checkout GET route can redirect signed-out users into sign-in with a return target.
|
||||
- Pricing now uses that GET route directly for Team checkout, preserving monthly/annual billing in the URL.
|
||||
- Cancelled checkout returns to the homepage pricing section with an inline recovery notice and retry link.
|
||||
- Public legal pages no longer expose Day-0 launch placeholders, bracketed launch-date placeholders, retired "Pro or Teams" copy, or the named pre-launch address/representative placeholders caught by the launch-copy guard.
|
||||
- Public download CTAs no longer send visitors directly to an empty GitHub Releases page; `/download` is a controlled status page until signed installers exist.
|
||||
- Account page redirects signed-out users before mounting Clerk account UI.
|
||||
- Fresh rendered desktop/mobile localhost smoke shows no current-page console errors or warnings for homepage, mobile menu, or signed-out checkout-error state.
|
||||
|
||||
## Correction Candidates
|
||||
|
||||
### T13-0: Public domain does not currently resolve
|
||||
|
||||
Evidence:
|
||||
- Current external smoke from this environment on 2026-07-08 could not resolve `waggle-os.ai`.
|
||||
- `nslookup waggle-os.ai` returned `Non-existent domain`.
|
||||
- Fresh refresh also shows `www.waggle-os.ai` has no A or CNAME record from this environment.
|
||||
- `apps/www/app/layout.tsx:137-141`, `apps/www/app/docs/methodology/page.tsx:36`, `apps/www/app/robots.ts:17`, and `apps/www/app/sitemap.ts:3` treat `https://waggle-os.ai` as canonical production.
|
||||
|
||||
Impact:
|
||||
- A founder, buyer, reviewer, or mobile executive cannot reach the public acquisition, pricing, legal, download, auth, or account surfaces at the canonical domain.
|
||||
- Local build success does not prove the public launch funnel exists.
|
||||
|
||||
Correction:
|
||||
- Configure DNS for `waggle-os.ai` and deploy the chosen public-site hosting target.
|
||||
- Run public smoke against the canonical domain after DNS propagation.
|
||||
- Keep the local `localhost` smoke as a pre-deploy check, not as final launch evidence.
|
||||
|
||||
Acceptance:
|
||||
- `https://waggle-os.ai/`, legal pages, `/docs/methodology`, `/sign-in`, `/sign-up`, `/account`, checkout handoff, and download CTA resolve from a normal network.
|
||||
- The canonical metadata, sitemap, and robots URL match the deployed host.
|
||||
|
||||
### T13-1: Download path is currently broken
|
||||
|
||||
Status: empty-release dead-end focused fixed locally; real signed installer publication remains open.
|
||||
|
||||
Evidence:
|
||||
- Historical evidence: `DownloadCTA` and footer Product > Download pointed directly to GitHub Releases latest.
|
||||
- GitHub currently reports no releases for `marolinik/waggle-os`; live 2026-07-08 refresh confirms `/releases/latest` returns no latest release and the GitHub API releases list is empty.
|
||||
- Current source evidence: public Download CTAs and the footer Download link route to `/download`.
|
||||
- Current route evidence: `/download` is a controlled status page explaining that Windows and macOS installers are being prepared, with source/contact actions instead of a direct empty release link.
|
||||
- Current scan evidence: no `releases/latest` target remains in `apps/www/app`, `apps/www/messages/en.json`, or `apps/www/__tests__`.
|
||||
- Current source evidence: `apps/www/app/_lib/os-detection.ts` returns `null` for mobile/tablet user agents, `macOS` for desktop Mac, `Windows` for desktop Windows, and `Linux` only for desktop Linux.
|
||||
- `apps/www/messages/en.json:39` says the product platforms are Windows and macOS.
|
||||
|
||||
Impact:
|
||||
- A founder no longer reaches an empty release page, but still cannot download a signed installer until releases are published.
|
||||
- Mobile/tablet visitors keep a generic Download CTA, avoiding a false desktop-installer promise.
|
||||
- A desktop Linux visitor can still see a Linux-specific label even though the public copy says Windows and macOS; this is less damaging now that the CTA leads to a status page, but should be revisited before artifact-specific downloads go live.
|
||||
|
||||
Correction:
|
||||
- Completed locally: point the CTA/footer to a controlled download/status page and add a release-link guard.
|
||||
- Completed locally: harden the tag release workflow so desktop artifacts build workspace packages before sidecar bundling, matching the PR Tauri verification lane.
|
||||
- Remaining launch work: publish real signed Windows/macOS release assets and switch `/download` from status page to artifact-aware download page.
|
||||
- Remaining polish: decide whether desktop Linux should stay generic or be shown as unsupported before signed installers go live.
|
||||
|
||||
Acceptance:
|
||||
- Focused local acceptance met: fresh smoke proves the public CTA leads to a deliberate download landing page, and no supported persona reaches an empty GitHub Releases page from the public CTA.
|
||||
- Full launch acceptance still requires valid Windows/macOS artifacts and final unsupported-OS copy for non-Windows/non-macOS desktops.
|
||||
|
||||
### T13-2: Deployment workflow does not match the current Next app shape
|
||||
|
||||
Status: focused fixed locally; deployed Vercel/DNS smoke remains open.
|
||||
|
||||
Evidence:
|
||||
- Historical evidence: `.github/workflows/deploy-www.yml` ran the www build and uploaded `apps/www/dist` to GitHub Pages.
|
||||
- `apps/www/next.config.mjs:6-13` has no `output: 'export'`.
|
||||
- Current build output is `.next`, not `dist` or `out`.
|
||||
- The built app includes dynamic auth/API routes and middleware.
|
||||
- Current source evidence: `.github/workflows/deploy-www.yml` now installs with `npm ci`, pulls the Vercel production environment, runs public-site tests/typecheck/build, then runs Vercel `build --prod` and `deploy --prebuilt --prod`.
|
||||
- Current test evidence: `deployment-workflow.test.ts` guards that the workflow contains Vercel production deployment commands and does not reference `upload-pages-artifact`, `deploy-pages`, or `apps/www/dist`.
|
||||
|
||||
Impact:
|
||||
- The prior GitHub Pages workflow could not serve the current app as configured.
|
||||
- Local source now has a coherent Next-capable deployment path, but external Vercel secrets, DNS, and deployed smoke are not proved by this local fix.
|
||||
|
||||
Correction:
|
||||
- Completed locally: move the workflow to the existing Vercel production architecture for the dynamic Clerk/Stripe Next app.
|
||||
- Remaining external launch work: configure `VERCEL_TOKEN`, `VERCEL_ORG_ID`, `VERCEL_PROJECT_ID`, production env vars, domain DNS, and run deployed smoke against `https://waggle-os.ai`.
|
||||
|
||||
Acceptance:
|
||||
- Focused local acceptance met: the checked-in deploy workflow no longer targets GitHub Pages/static artifacts for a dynamic Next app.
|
||||
- Full launch acceptance still requires a successful production deploy and public smoke covering `/`, legal pages, auth/account routing, checkout route behavior, webhook reachability, and download CTA.
|
||||
|
||||
### T13-3: Signed-out Team checkout dead-ends in the pricing UI
|
||||
|
||||
Status: focused fixed locally.
|
||||
|
||||
Evidence:
|
||||
- `apps/www/app/api/stripe/checkout/route.ts:213-240` has a GET flow that redirects signed-out users to sign-in.
|
||||
- `apps/www/app/api/stripe/checkout/route.ts:242-284` keeps a POST compatibility flow returning JSON.
|
||||
- `apps/www/app/api/stripe/checkout/route.ts:273-276` returns `{ message: 'Sign in required', signInUrl }` for signed-out POST.
|
||||
- Historical source evidence: the old pricing UI used POST, read only `message` on non-OK responses, and ignored `signInUrl`.
|
||||
- Historical rendered smoke clicked Annual then Get Team while signed out; the page stayed at `/`, showed only a small `Sign in required` alert, and exposed no sign-in recovery link in the pricing state.
|
||||
- Current source evidence: `apps/www/app/_components/Pricing.tsx` renders the Team CTA as a link to `/api/stripe/checkout?tier=teams&billing={monthly|annual}`.
|
||||
- Current rendered Browser evidence: monthly and annual Team links update correctly and signed-out users enter the route-level GET flow.
|
||||
|
||||
Impact:
|
||||
- A buyer clicking Get Team before auth sees implementation-shaped error text instead of continuing to sign-in/sign-up and checkout.
|
||||
|
||||
Correction:
|
||||
- Completed locally: pricing CTA migrated to the canonical GET redirect flow.
|
||||
- Remaining launch evidence: verify return-to-checkout after a real Clerk sign-in/sign-up session and real Stripe checkout session.
|
||||
|
||||
Acceptance:
|
||||
- Focused local acceptance met: signed-out Get Team starts the auth route instead of showing a dead-end POST error; billing period is preserved in the route URL.
|
||||
- Full launch acceptance still requires a real signed-in checkout run against deployed auth/Stripe config.
|
||||
|
||||
### T13-4: Checkout cancel recovery points to a dead route
|
||||
|
||||
Status: focused fixed locally.
|
||||
|
||||
Evidence:
|
||||
- Historical source evidence: `cancel_url` pointed to `/pricing?checkout=cancelled`, but pricing is a section on `/`.
|
||||
- Historical local and rendered Browser smokes confirmed `/pricing?checkout=cancelled -> 404`.
|
||||
- Current source evidence: the Stripe cancel URL is `/?checkout=cancelled#pricing`.
|
||||
- Current test evidence: `stripe-checkout-route.test.ts` verifies the cancel URL passed to Stripe.
|
||||
- Current rendered Browser evidence: `/?checkout=cancelled#pricing` renders pricing, shows a cancelled-checkout recovery notice, and exposes a retry link that tracks the selected billing period.
|
||||
|
||||
Impact:
|
||||
- A buyer who cancels Stripe checkout can land on a 404 instead of a recoverable pricing state.
|
||||
|
||||
Correction:
|
||||
- Completed locally: use `/?checkout=cancelled#pricing` plus an inline recovery notice and retry action.
|
||||
|
||||
Acceptance:
|
||||
- Focused local acceptance met: cancelled checkout returns to a visible pricing recovery state, not a 404.
|
||||
- Full launch acceptance still requires a real Stripe cancellation redirect on the deployed site.
|
||||
|
||||
### T13-5: Legal and trust pages are not launch-ready
|
||||
|
||||
Status: placeholder/stale-tier copy focused fixed locally; formal legal approval remains open.
|
||||
|
||||
Evidence:
|
||||
- Historical evidence: legal pages contained "Day-0 placeholder text", `[Day-0 launch date]`, launch/address placeholders, and Privacy said "upgrade to Pro or Teams".
|
||||
- Current source evidence: terms/privacy/cookies/EU AI Act pages use July 8, 2026 effective/updated dates, current Solo/Team language, non-placeholder contact/representative wording, and no named launch placeholder patterns.
|
||||
- Current test evidence: `legal-copy.test.ts` guards against Day-0 placeholder text, launch-date placeholders, pre-launch address placeholders, retired "Pro or Teams" copy, and bracketed representative placeholders.
|
||||
|
||||
Impact:
|
||||
- Team admins, enterprise reviewers, and privacy-conscious founders lose trust before installing when public legal pages expose placeholders or retired tier language.
|
||||
|
||||
Correction:
|
||||
- Completed locally: replaced the named placeholders/stale-tier copy and added a legal-copy guard.
|
||||
- Remaining launch/legal process: obtain formal Egzakta legal approval for the current text, registered details, and representative wording before treating these pages as legally final.
|
||||
|
||||
Acceptance:
|
||||
- Focused local acceptance met: `legal-copy.test.ts` passes and the targeted `rg` launch-placeholder scan returns no matches.
|
||||
- Full launch acceptance still requires legal sign-off.
|
||||
|
||||
### T13-6: Production smoke needs a stable browser lane
|
||||
|
||||
Evidence:
|
||||
- Current HTTP route/API smoke passes for core routes when using `--hostname localhost`.
|
||||
- Earlier rendered smoke found `127.0.0.1` binding failures and Clerk development/session-loop warning noise.
|
||||
- The fresh Browser smoke exercises homepage, mobile menu, signed-out pricing CTA, checkout redirect API, and route recovery. It does not exercise Clerk modal browser behavior, signed-in checkout, or real Stripe return.
|
||||
|
||||
Impact:
|
||||
- A route-only smoke can miss the exact UI failures buyers hit: modal auth, return-to-checkout, console warning loops, mobile nav, and visual layout.
|
||||
|
||||
Correction:
|
||||
- Add a repeatable public-site Playwright/browser smoke using the known-good `localhost` host binding.
|
||||
- Cover desktop and mobile: homepage, mobile menu, download CTA, sign-in/sign-up pages, account redirect, signed-out checkout, checkout cancel, legal pages, and methodology.
|
||||
|
||||
Acceptance:
|
||||
- Fresh screenshots and route/API logs are attached with no unexpected 404, 500, timeout, or auth-loop noise.
|
||||
|
||||
### T13-7: Coverage is too narrow for a launch funnel
|
||||
|
||||
Evidence:
|
||||
- `apps/www/__tests__` currently covers only `BrandPersonasCard`.
|
||||
- Historical coverage gap: no checked-in route E2E, checkout-recovery test, legal placeholder guard, release/download target guard, or deployment artifact guard was found.
|
||||
- Current local improvement: checkout recovery, legal placeholder/stale-tier, release/download target, mobile download-label, and deployment workflow guards now exist. Deployed-domain smoke, real checkout, signed installer publication, and formal legal sign-off remain open.
|
||||
|
||||
Impact:
|
||||
- The public site can regress in the exact flows needed for acquisition and purchase while tests stay green.
|
||||
|
||||
Correction:
|
||||
- Add focused tests/guards:
|
||||
- Download CTA target and platform labels.
|
||||
- Signed-out checkout auth continuation.
|
||||
- Checkout cancel recovery route.
|
||||
- Legal placeholder/stale-tier grep.
|
||||
- Deployment artifact/hosting mode consistency.
|
||||
- Public route smoke in CI or release checklist.
|
||||
|
||||
Acceptance:
|
||||
- `npm run test -w apps/www`, www typecheck, www build, and the public funnel smoke all pass from a clean checkout.
|
||||
|
||||
## Five-Persona Impact
|
||||
|
||||
| Persona | Cap Until Fixed | Why |
|
||||
|---|---:|---|
|
||||
| Solo founder/operator | 5/10 | Canonical public domain does not resolve; local Download no longer dead-ends, but there is still no signed installer artifact to obtain. |
|
||||
| Team admin/security reviewer | 5/10 | Public legal/pricing/account pages are unreachable at the canonical domain; deployment target, formal legal sign-off, and real signed-in checkout evidence remain launch blockers. |
|
||||
| Mobile executive | 5/10 | Mobile cannot inspect the canonical public site; mobile download labels are honest locally, but signed download/release truth still needs fixing after deploy. |
|
||||
| Engineer/power user | 6/10 | Local site is credible, but NXDOMAIN plus no signed release artifact and deploy mismatch make the product look unreleasable. |
|
||||
| Privacy/compliance reviewer | 6/10 | Public legal pages no longer expose the named placeholder/stale-tier copy locally, but canonical-domain reachability and formal legal sign-off still block launch trust. |
|
||||
|
||||
## Approval Recommendation
|
||||
|
||||
Keep T13 outside Phase 1 implementation, but do not treat it as optional for the final 9/10 complete-UX goal. After the installed-app P0s are approved and fixed, run T13 as a launch-readiness slice with this order:
|
||||
|
||||
1. Make `waggle-os.ai` resolve and deploy the selected public-site target.
|
||||
2. Publish signed installer artifacts behind the controlled `/download` path.
|
||||
3. Fix deploy target or hosting architecture.
|
||||
4. Verify real deployed Clerk sign-in/sign-up return-to-checkout and Stripe cancel/success redirects.
|
||||
5. Complete formal legal sign-off for public legal/trust copy.
|
||||
6. Add the public funnel smoke and remaining minimal guards.
|
||||
|
||||
T13 can be deferred only if the user explicitly says the five-persona judge score is limited to the installed desktop cockpit and excludes the public acquisition/payment/legal funnel.
|
||||
131
docs/audits/2026-07-08-mobile-executive-t2-t12-analysis.md
Normal file
131
docs/audits/2026-07-08-mobile-executive-t2-t12-analysis.md
Normal file
@@ -0,0 +1,131 @@
|
||||
# Focused Mobile Executive T2/T12 Analysis
|
||||
|
||||
Status: original analysis plus current focused verification. The original screenshot pass recorded why T2 failed; the current focused Settings journey verifies that P0-2 is fixed in the present build.
|
||||
|
||||
Purpose: add 390 x 844 rendered evidence for the Mobile Executive judge path and sharpen the mobile acceptance criteria. The original pass shows why a simple document-level horizontal overflow check is not enough; the current focused run confirms the Settings portion of T2 now passes.
|
||||
|
||||
Guideline source refreshed during this pass: Vercel Web Interface Guidelines, `https://raw.githubusercontent.com/vercel-labs/web-interface-guidelines/main/command.md`. The rules most relevant here are safe-area and overflow discipline, readable touch targets, visible focus, form clarity, URL/state clarity, and not relying on screenshots alone when layout can clip individual controls.
|
||||
|
||||
## Fresh Mobile Smoke Evidence
|
||||
|
||||
Environment:
|
||||
|
||||
```powershell
|
||||
$env:WAGGLE_PORT='3419'
|
||||
$env:WAGGLE_TRUST_LOCALHOST='1'
|
||||
$env:WAGGLE_DISABLE_MARKETPLACE_SYNC='1'
|
||||
$env:WAGGLE_DATA_DIR="$env:TEMP\waggle-mobile-smoke-3419"
|
||||
$env:EMBEDDING_PROVIDER='mock'
|
||||
$env:VITE_CLERK_PUBLISHABLE_KEY=''
|
||||
$env:CLERK_SECRET_KEY=''
|
||||
npm run build
|
||||
npx tsx packages/server/src/local/start.ts --skip-litellm
|
||||
```
|
||||
|
||||
Viewport: Playwright mobile/touch context at 390 x 844.
|
||||
|
||||
Screenshot directory:
|
||||
|
||||
```text
|
||||
output/playwright/mobile-executive-3419/
|
||||
```
|
||||
|
||||
Current focused verification:
|
||||
|
||||
```powershell
|
||||
$env:WAGGLE_E2E_PORT='34195'
|
||||
$env:WAGGLE_E2E_BASE_URL='http://localhost:34195'
|
||||
npx playwright test tests/e2e/user-journeys.spec.ts --project=chromium -g "Settings is usable at 390px" --reporter=line
|
||||
```
|
||||
|
||||
Result: pass, 1/1. The focused route checks `/settings`, `/settings?tab=models`, `/settings?tab=billing`, and `/settings/profile` at 390 x 844. It asserts no document-level horizontal overflow and no visible control overflow for buttons, tabs, tab panels, inputs, selects, and textareas.
|
||||
|
||||
Screenshots captured:
|
||||
|
||||
- `home.png`
|
||||
- `settings.png`
|
||||
- `settings-tab-models.png`
|
||||
- `settings-tab-billing.png`
|
||||
- `settings-profile.png`
|
||||
- `memory.png`
|
||||
- `workspace-chat.png`
|
||||
- `overlay-command-center-open.png`
|
||||
- `overlay-command-center-closed.png`
|
||||
- `overlay-workspace-switcher-open.png`
|
||||
- `overlay-workspace-switcher-closed.png`
|
||||
|
||||
Build/runtime notes:
|
||||
|
||||
- `npm run build` passed.
|
||||
- Tailwind ambiguous motion-token warnings are now fixed by named motion utilities.
|
||||
- The `shape-selection.ts` dynamic/static import warning is now fixed by a static adapter import guarded by `build-warning-hygiene.test.ts`.
|
||||
- The Vite large-chunk warning is now fixed: current startup JS is `index-D-wAFouW.js` at 421.96 kB minified and 114.08 kB gzip, with PostHog split into a lazy `posthog-t8jwqJJL.js` chunk at 208.95 kB.
|
||||
- The local server degraded embeddings to mock and skipped LiteLLM as intended for the analysis lane.
|
||||
|
||||
## Route Results
|
||||
|
||||
| Route or overlay | Result | Document overflow | Visible layout result | Console/status notes |
|
||||
|---|---|---:|---|---|
|
||||
| `/home` | Rendered expected Home content. | No | No critical route layout failure found in this smoke. Decorative offscreen elements are present but did not break the Home task. | Original screenshot pass emitted T1 Clerk/CSP errors; current focused console-health checks pass 3/3 on port `34196`. |
|
||||
| `/settings` | Rendered expected Settings content. | No | Original pass failed T2 because the persistent side rail and Settings rail left a narrow content column. Current focused run on port `34195` passes visible-control bounds. | Existing T1 Clerk/CSP errors were emitted in the original screenshot pass. |
|
||||
| `/settings?tab=models` | Rendered expected Models content. | No | Original pass failed T2 because disclosure controls overflowed and provider/model cards were squeezed. Current focused run on port `34195` passes visible-control bounds. | Existing T1 Clerk/CSP errors were emitted in the original screenshot pass. |
|
||||
| `/settings?tab=billing` | Rendered expected Billing content. | No | Original pass failed T2/T10 because billing copy/cards were squeezed and the Annual toggle exceeded the viewport. Current focused run on port `34195` passes visible-control bounds. | Existing T1 Clerk/CSP errors were emitted in the original screenshot pass. |
|
||||
| `/settings/profile` | Rendered expected Profile content. | No | Original pass had no visible route-level overflow; current focused run on port `34195` also passes visible-control bounds. | Existing T1 Clerk/CSP errors were emitted in the original screenshot pass. |
|
||||
| `/memory` | Rendered expected Memory content. | No | Fails mobile polish: the Memory tab strip extends past the viewport; this belongs to T10/T12 unless it blocks the chosen mobile judge path. | Original screenshot pass emitted T1 Clerk/CSP errors; current focused console-health checks pass 3/3 on port `34196`. |
|
||||
| `/workspaces/default-workspace/chat` | Rendered expected workspace/chat shell. | No | Fails mobile polish: workspace tabs extend past the viewport; message send and keyboard/touch flow were not exercised. | Original screenshot pass emitted T1 Clerk/CSP errors; current focused console-health checks pass 3/3 on port `34196`. |
|
||||
| Command Center overlay | Opened. Escape close check failed in this run. | No | Original pass found long command labels/subtitles overflowing and a missing dialog description warning; current Command Center focused branch passes elsewhere. | Original screenshot pass emitted T1 Clerk/CSP errors; current focused console-health checks pass 3/3 on port `34196`. |
|
||||
| Workspace Switcher overlay | Opened and closed with Escape. | No | Good signal for one mobile overlay close path, but route-changing close behavior remains part of T4 until codified. | Original screenshot pass emitted T1 Clerk/CSP errors; current focused console-health checks pass 3/3 on port `34196`. |
|
||||
|
||||
Key interpretation: `document.documentElement.scrollWidth` stayed equal to the 390 px viewport for the route screenshots, but individual controls visibly overflowed or became unreadably narrow. Phase 1 must test critical element bounds and screenshots, not only document scroll width.
|
||||
|
||||
## Findings Added By This Pass
|
||||
|
||||
### M1: Settings mobile failure is stronger than horizontal page overflow
|
||||
|
||||
Ticket mapping: T2, with T10/T12 evidence impact.
|
||||
|
||||
The original Settings failure was not only "the page scrolls sideways." The document can report no horizontal overflow while controls still clip inside constrained flex columns. The current `J-mobile: Settings is usable at 390px width` test now asserts that critical visible elements stay within the viewport:
|
||||
|
||||
- Settings section tab list.
|
||||
- Settings header disclosure segmented control.
|
||||
- Models provider cards and model selector/change controls.
|
||||
- Billing plan/toggle controls and primary plan copy.
|
||||
- Profile fields and save controls.
|
||||
|
||||
### M2: Memory and workspace chat mobile tab strips need evidence ownership
|
||||
|
||||
Ticket mapping: T10/T12, and T11 if route evidence is codified.
|
||||
|
||||
Both `/memory` and workspace chat rendered, but their tab strips extended beyond the viewport. This may be acceptable if they become intentionally scrollable with clear affordance, but it cannot be ignored in the Mobile Executive judge bundle.
|
||||
|
||||
### M3: Command Center mobile close and label fit are not proven
|
||||
|
||||
Ticket mapping: T10/T12; also affects the Engineer path if Command Center is selected as a required overlay.
|
||||
|
||||
Command Center opened on mobile, but the script still found it visible after Escape. Long command labels/subtitles also overflowed, and the browser logged a missing dialog description warning. If the Mobile Executive judge uses Workspace Switcher instead, this can be deferred; if it uses Command Center, the score remains capped.
|
||||
|
||||
### M4: T1 console health is now verified for sampled accountless routes
|
||||
|
||||
Ticket mapping: T1.
|
||||
|
||||
The original screenshot pass emitted Clerk/CSP console errors on every mobile route and overlay. Current focused accountless console-health checks pass 3/3 on port `34196`, covering first-run, initial load, and full-product critical console guards. Keep this in regression and add explicit Clerk-enabled state evidence later.
|
||||
|
||||
### M5: Workspace Switcher has one good mobile close path
|
||||
|
||||
Ticket mapping: T4/T12.
|
||||
|
||||
Workspace Switcher opened and closed with Escape in this mobile pass. This does not close T4 because the previous route-changing navigation issue still needs codified regression coverage, but it is good evidence for the overlay-close part of the Mobile Executive route sequence.
|
||||
|
||||
## Correction Requirements
|
||||
|
||||
Keep these in mobile verification:
|
||||
|
||||
1. Run Settings mobile coverage for `/settings`, `/settings?tab=models`, `/settings?tab=billing`, and `/settings/profile`.
|
||||
2. Fail if any critical visible control extends outside the viewport, even when document-level scroll width is clean.
|
||||
3. Capture or inspect screenshots for Home, Settings general/models/billing/profile, Memory, workspace chat, Command Center, and Workspace Switcher.
|
||||
4. Record whether Command Center or Workspace Switcher is the selected Mobile Executive overlay path; the selected overlay must open and close without trapping focus/scroll.
|
||||
5. Keep T1 console capture attached to mobile evidence; sampled accountless console health is currently green, while explicit Clerk-enabled state evidence remains separate.
|
||||
|
||||
## Current Recommendation
|
||||
|
||||
Keep Phase 1 scoped to T1, T2, T3, T4, T5, and T11. The Settings portion of T2 and sampled accountless T1 console health are now green in focused verification, so remaining Mobile Executive risk shifts to Memory/workspace chat tab-strip evidence ownership and the selected overlay path. Command Center mobile close/label fit is now covered elsewhere by the focused Command Center branch in `tests/e2e/user-journeys.spec.ts`.
|
||||
76
docs/audits/2026-07-08-ops-deploy-ci-judging-t18-analysis.md
Normal file
76
docs/audits/2026-07-08-ops-deploy-ci-judging-t18-analysis.md
Normal file
@@ -0,0 +1,76 @@
|
||||
# T18 Ops, Deployment, CI, Benchmark, and Judging UX Analysis
|
||||
|
||||
Date: 2026-07-08
|
||||
Scope: GitHub Actions, Docker, Compose, Render, LiteLLM config, infra test lane, benchmark harness, and judging artifacts.
|
||||
Mode: analysis plus focused public-site deployment workflow fix.
|
||||
|
||||
## Bottom Line
|
||||
|
||||
T18 is not a product screen, but it is still part of UX: it is the experience of shipping, operating, validating, and proving the product.
|
||||
|
||||
The current state is mixed. Config syntax is healthy, secrets are not tracked in the checked env files, the benchmark harness now passes from its package-local command (29 files / 325 tests), production Compose now fails closed for Postgres and MinIO credentials, Render is explicitly aligned to the hosted local-sidecar mode, and the public-site workflow targets Vercel prebuilt deployment instead of a nonexistent GitHub Pages static artifact. The remaining blockers are release-confidence issues: deployed Vercel/DNS proof is still missing, CI browser E2E is advisory, the 19-suite live infra lane is not in CI and could not run here because Docker Desktop's engine is unavailable, and current `judging/` files are historical rather than the July five-persona scoring evidence.
|
||||
|
||||
## User Jobs
|
||||
|
||||
- Release reviewers can trust CI as an honest gate.
|
||||
- Operators can validate Compose, Docker, Render, and LiteLLM config without leaking local secrets into logs.
|
||||
- Hosted deployment mode is explicit: local sidecar demo or team Postgres server.
|
||||
- Infra-dependent tests have a known runnable lane.
|
||||
- Benchmark and judge harness commands work from documented entrypoints.
|
||||
- Historical judge screenshots and reports are not mistaken for current 9/10 evidence.
|
||||
|
||||
## Command Evidence
|
||||
|
||||
| Check | Result | Notes |
|
||||
|---|---:|---|
|
||||
| YAML parse for `docker-compose.yml`, `docker-compose.production.yml`, `render.yaml`, `litellm-config.yaml`, and all 7 `.github/workflows/*.yml` | Pass | Syntax is valid for the checked deployment, workflow, and model-router files. |
|
||||
| `docker --version`; `docker compose version` | Pass | Docker CLI 28.4.0 and Compose v2.39.2 are installed. |
|
||||
| `docker compose ps --format json` | Fail | Docker Desktop Linux engine pipe was not reachable, so live Compose services could not be inspected or started here. |
|
||||
| 127.0.0.1 port preflight for 5434 and 6381 | Closed | The Postgres/Redis ports required by `vitest.infra.config.ts` were not reachable locally. |
|
||||
| `docker compose ... config --no-interpolate` targeted scan | Pass | Safer shareable evidence path because variable references stay literal instead of printing local secret values. |
|
||||
| Secret tracking check for `.env`, `.env.local`, `AI API KEYS.txt`, and `apps/www/.env.local` | Pass for tracked files | Only checked `.env.example` files are tracked; local secret-bearing files are ignored. |
|
||||
| `Test-Path apps/www/dist`; `Test-Path apps/www/.next`; deploy workflow scan | `False`; `True`; Improved | `apps/www` is a dynamic Next app producing `.next`; `.github/workflows/deploy-www.yml` now uses Vercel `pull`, `build`, and `deploy --prebuilt --prod` and no longer uploads `apps/www/dist` to GitHub Pages. |
|
||||
| `npx tsc --noEmit --project benchmarks/harness/tsconfig.json` | Pass | Benchmark harness TypeScript compiles. |
|
||||
| `npm run test --prefix benchmarks/harness -- --reporter=dot` | Pass, 29 files / 325 tests | Package-local command now delegates to the root Vitest aliases/setup. |
|
||||
| `npx vitest run benchmarks/harness/tests --config vitest.config.ts --reporter=dot` | Pass, 29 files / 325 tests | Root-run benchmark tests pass, but output is noisy with turn and benchmark logs. |
|
||||
| `judging/FINAL-REPORT.md` and `judging/round3/*` inspection | Historical only | These are June 2026 judge rounds, not current post-fix July scorecards. |
|
||||
|
||||
## Source Findings
|
||||
|
||||
| ID | Severity | Finding | Evidence | Correction Needed |
|
||||
|---|---:|---|---|---|
|
||||
| T18-1 | P1 | Public-site deploy target needed to match the dynamic Next app. | Historical Pages workflow uploaded nonexistent `apps/www/dist`; current workflow now uses Vercel production pull/build/deploy and `deployment-workflow.test.ts` guards against reintroducing `upload-pages-artifact`, `deploy-pages`, or `apps/www/dist`. | Local workflow mismatch is fixed. Remaining T13/T18 work is external: Vercel secrets/project linkage, production env, DNS, and deployed smoke evidence. |
|
||||
| T18-2 | Resolved | Render deploy mode was ambiguous. | `render.yaml` now explicitly documents and tests the hosted local-sidecar mode, keeps `/data` persistence and Stripe sidecar routes, and removes unused Postgres/Redis provisions. The Postgres/Redis-backed team server remains the Dockerfile/Compose path. | Keep the two deployment modes documented separately. |
|
||||
| T18-3 | Resolved locally | The broad CI browser E2E job is advisory, leaving no merge-blocking rendered UX gate. | `.github/workflows/ci.yml` now adds a separate blocking `e2e-smoke` job; `npm run test:e2e:smoke` passed 5/5 locally on a fresh build and sidecar. The broad exploratory E2E job remains advisory. | Keep the smoke slice small and stable; the broad suite remains evidence-producing rather than merge-blocking. |
|
||||
| T18-4 | P1 | Live infra suites are not represented in CI and were not runnable in this audit environment. | `vitest.infra-suites.ts` lists 19 Postgres/Redis suites. Docker engine was unavailable and ports 5434/6381 were closed. | Add a Docker-provisioned CI lane or a documented local lane with migration/start/stop commands and current evidence; otherwise defer infra evidence explicitly. |
|
||||
| T18-5 | Resolved | Production Compose kept default credentials. | Postgres and MinIO credentials now use required `${VAR:?set VAR}` interpolation; the deployment test asserts the fail-closed contract, and `docker compose -f docker-compose.production.yml config --no-interpolate` preserves the required placeholders for secret-safe review. | Keep production secrets in the deployment environment and never add convenience fallbacks back to this file. |
|
||||
| T18-6 | P1 | Shareable ops evidence can leak secrets if reviewers use the obvious command. | `docker compose config` interpolates ignored local env values. `--no-interpolate` is safer for evidence logs; sanitized env validation also passes locally. | Keep the secret-safe command pair in the release runbook and attach sanitized output for the deployment packet. |
|
||||
| T18-7 | Resolved | Benchmark package-local test command failed even though root-run tests passed. | `npm run test --prefix benchmarks/harness -- --reporter=dot` passes 29 files / 325 tests through the root config. | Keep the package-local delegation script as the canonical harness entrypoint. |
|
||||
| T18-8 | P1 | Judging artifacts are stale for the current goal. | `judging/FINAL-REPORT.md` is a June 2026 mission report; the July scorecards/runbook still require a current human-scored pass against the post-fix source. | Generate and review new five-persona artifacts only after external release scope is decided; keep historical reports clearly labeled as historical. |
|
||||
| T18-9 | P2 external evidence | Provider freshness and runtime routing are hermetically proven; a paid external-provider request is not yet attached. | The desktop runtime builds secret-free LiteLLM config from complete live provider catalogs rather than a model inventory. It covers provider pagination, refreshes UI catalogs on app focus, restarts on key save/retry, and hot-loads an exact model id released after startup when that id is selected for default, Chat, or fleet execution. Key-save -> unseen model -> generated config -> exact-id completion is deterministic. | Add one credentialed provider smoke in the release lane and retain the hermetic proof as the deterministic CI gate. |
|
||||
|
||||
## Persona Impact
|
||||
|
||||
| Persona | Current T18 cap | Why |
|
||||
|---|---:|---|
|
||||
| Engineer / power user | 7/10 | CI, benchmark, infra, and command-shape gaps reduce trust that green means shippable. |
|
||||
| Team admin / security reviewer | 7/10 | Default production credentials, ambiguous hosted deploy mode, and secret-log risks are trust blockers. |
|
||||
| Solo founder | 8/10 | Public deploy and checkout recovery can fail before the founder reaches the desktop app. |
|
||||
| Researcher | 8/10 | Historical judge artifacts cannot be reused as evidence for current memory/UX quality. |
|
||||
| Mobile executive | 8/10 | Less directly affected, but public deploy and judge evidence still gate the complete-system claim. |
|
||||
|
||||
## Acceptance For Closing T18
|
||||
|
||||
- `apps/www` deployment target is coherent with the actual Next app output and dynamic routes; production Vercel/DNS smoke is still required under T13 before final scoring.
|
||||
- Render deploy mode is decided and verified: hosted sidecar demo or team Postgres server.
|
||||
- Production Compose has fail-closed secrets or a clear sample-vs-production split.
|
||||
- Secret-safe validation commands are documented and used for shareable ops evidence.
|
||||
- CI includes a blocking rendered smoke lane, and the broad exploratory E2E job is explicitly advisory before scoring.
|
||||
- `npm run test:infra` has a Docker/migration lane with current evidence, or the 19 infra suites are explicitly deferred.
|
||||
- Benchmark package-local command shape is fixed or the root-run command is documented as canonical.
|
||||
- Current five-persona judging artifacts are generated after approved fixes and replace historical evidence for scoring.
|
||||
- LiteLLM/provider hermetic routing remains green, and one credentialed external-provider smoke is attached or explicitly outside the current score.
|
||||
|
||||
## Packet Decision
|
||||
|
||||
Keep T18 as `Phase 2 Pending` / launch-tooling gate. It should not block Phase 1 implementation, but it blocks the final "complete UX, all parts functional, five judges at 9/10" claim unless the user explicitly defers ops/deployment/CI/benchmark/judging from the score.
|
||||
280
docs/audits/2026-07-08-route-evidence-t11-analysis.md
Normal file
280
docs/audits/2026-07-08-route-evidence-t11-analysis.md
Normal file
@@ -0,0 +1,280 @@
|
||||
# T11 Route Evidence Gap Analysis
|
||||
|
||||
Status: route-existence ownership verified. No product code changed.
|
||||
|
||||
Scope: installed `apps/web` route registry, app-id route mapping, direct route/test references, command-query destinations, major overlays, and route evidence needed before five-persona scoring.
|
||||
|
||||
## Why This Matters
|
||||
|
||||
The shell can have many green component and API tests while still leaving a user-visible route unproven. T11 is the guardrail against scoring only the familiar paths. A route is not judge-ready until it has an evidence owner that proves the actual URL, expected state, viewport, and recovery behavior.
|
||||
|
||||
## Current Route Registry
|
||||
|
||||
Authoritative source: `apps/web/src/App.tsx`.
|
||||
|
||||
Registered production routes:
|
||||
|
||||
```text
|
||||
/auth
|
||||
/
|
||||
/home
|
||||
/workspaces
|
||||
/workspaces/:workspaceId/:tab?
|
||||
/memory/:mindScope?
|
||||
/artifacts
|
||||
/files
|
||||
/agents
|
||||
/automations
|
||||
/skills
|
||||
/room
|
||||
/waggle-dance
|
||||
/approvals
|
||||
/connectors
|
||||
/mcps
|
||||
/marketplace
|
||||
/launcher
|
||||
/team
|
||||
/settings
|
||||
/settings/vault
|
||||
/settings/profile
|
||||
/settings/mission-control
|
||||
/settings/timeline
|
||||
/settings/events
|
||||
/settings/usage
|
||||
/benchmarks
|
||||
/platform
|
||||
/payment-success
|
||||
/payment-cancelled
|
||||
*
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- `/payment-cancelled` is a router redirect to `/settings?tab=billing`, not a full page component.
|
||||
- `/benchmarks` and `/platform` are real routed AppShell children, even though they are command-palette-oriented surfaces.
|
||||
- `/motion-spec` is dev-only and remains excluded from the product score unless developer visual tooling is brought into scope.
|
||||
- `routeFor` covers 28 app ids, including killed/retargeted ids, but it is not the same as rendered URL evidence.
|
||||
|
||||
## Current Command Evidence
|
||||
|
||||
| Check | Result | Interpretation |
|
||||
|---|---|---|
|
||||
| `npm run test -- apps/web/src/test/p1a-routes.test.ts --run` | Fail, no files found | Root Vitest excludes `apps/**`; the obvious command does not run app route tests. |
|
||||
| `npm run test -w apps/web -- src/test/p1a-routes.test.ts --run` | Pass, 1 file / 29 tests | Proves app-id route mapping, search result retargeting, query serialization, nav active-route matching, and dock route invariants. |
|
||||
| `npm run test -w apps/web -- src/components/os/apps/BenchmarkApp.test.tsx src/components/os/apps/PlatformApp.test.tsx src/test/pr7a-billing.test.tsx --run` | Pass, 3 files / 17 tests | Proves component behavior for Benchmark, Platform, Settings billing deep-link, and PaymentSuccess states, but not direct URL rendering for `/benchmarks`, `/platform`, or `/payment-cancelled`. |
|
||||
| Fresh built-app route smoke, port 3407 | Mixed | `npm run build` passed, then a one-off Playwright smoke against `http://127.0.0.1:3407` proved `/benchmarks`, `/platform`, and `/payment-cancelled` render meaningful shell content; `/payment-cancelled` redirects to `/settings?tab=billing`. All three routes still emit the existing T1 CSP/Clerk console errors, so this is route-existence evidence, not judge-ready route health. Screenshots: `output/playwright/route-smoke-3407/benchmarks.png`, `platform.png`, `payment-cancelled.png`. |
|
||||
| Fresh thin-route smoke, port 3411 | Mixed | `npm run build` passed, then a one-off Playwright smoke proved `/launcher`, `/launcher?watch=1`, `/waggle-dance`, `/artifacts`, `/settings/profile`, `/settings/timeline`, `/payment-success`, `/automations`, `/mcps`, `/settings/usage`, and `/files` return 200 and render meaningful shell content. All routes still emit T1 CSP/Clerk console errors. Additional findings: leaving Launcher while detection is in flight can log `[adapter] detectTools failed: Failed to fetch`, and `/settings/usage` emits a visible 403 resource error while showing a Team-tier gate. Screenshots: `output/playwright/thin-route-smoke-3411/*.png`. |
|
||||
| Current all-route built-preview smoke, port 3457 | Mixed | `npm run build` passed, a fresh sidecar on `127.0.0.1:3333` returned healthy, and a Playwright smoke against built preview `http://127.0.0.1:3457` navigated 33 desktop routes plus 11 mobile route spot-checks. All navigations returned 200, `/payment-cancelled` redirected to `/settings?tab=billing`, and no route had document-level horizontal overflow. This remains supplemental screenshot/overflow evidence now that `J-route-coverage` owns codified route-existence regression coverage. Artifacts: `output/playwright/route-evidence-3457/all-route-smoke.json`, `all-route-smoke-summary.json`, and 44 screenshots under `output/playwright/route-evidence-3457/screenshots/`. |
|
||||
| `J-route-coverage` Playwright tests, port `34200` | Pass, 2 tests | Codifies route-existence owners for `/benchmarks`, `/platform`, `/payment-cancelled`, `/launcher`, `/launcher?watch=1`, `/waggle-dance`, `/artifacts`, `/settings/profile`, `/settings/timeline`, `/payment-success`, `/automations`, `/mcps`, `/settings/usage`, and `/files`. `/payment-cancelled` is asserted to redirect to `/settings?tab=billing` and show billing/plan recovery copy. | Proves rendered shell/content and redirect behavior, not deeper form/action/error states such as MCP install, file upload, payment provider round-trip, Launcher hook lifecycle, or Usage cost semantics. |
|
||||
| Direct route-string reference count over `tests/e2e`, `tests/visual`, `tests/vision`, `apps/web/src/test` | Mixed | Confirms several zero/thin route evidence owners. Counts below are references, not proof by themselves. |
|
||||
|
||||
Current warnings:
|
||||
|
||||
- The app-local test commands emit Node `punycode` deprecation warnings.
|
||||
- `pr7a-billing.test.tsx` emits React Router future-flag warnings in the Settings billing deep-link test.
|
||||
- The current all-route built-preview smoke emits a Clerk development-key warning on every sampled route, even though the route exists and renders. This keeps T1 open for standard judge console health.
|
||||
|
||||
## Current All-Route Built-Preview Smoke
|
||||
|
||||
Run date: 2026-07-08.
|
||||
|
||||
Artifacts:
|
||||
|
||||
- `output/playwright/route-evidence-3457/all-route-smoke-summary.json`
|
||||
- `output/playwright/route-evidence-3457/all-route-smoke.json`
|
||||
- `output/playwright/route-evidence-3457/screenshots/*.png`
|
||||
|
||||
Scope:
|
||||
|
||||
- Desktop 1440 x 900: `/auth`, `/`, `/home`, `/workspaces`, `/workspaces/default-workspace/chat`, `/workspaces/default-workspace/files`, `/memory`, `/artifacts`, `/files`, `/agents`, `/automations`, `/skills`, `/room`, `/waggle-dance`, `/approvals`, `/connectors`, `/mcps`, `/marketplace`, `/launcher`, `/launcher?watch=1`, `/team`, `/settings`, `/settings/vault`, `/settings/profile`, `/settings/mission-control`, `/settings/timeline`, `/settings/events`, `/settings/usage`, `/benchmarks`, `/platform`, `/payment-success`, `/payment-cancelled`, and the catch-all route.
|
||||
- Mobile 390 x 844: `/home`, `/settings`, `/settings?tab=models`, `/settings?tab=billing`, `/settings/profile`, `/memory`, `/workspaces/default-workspace/chat`, `/launcher`, `/mcps`, `/files`, and `/payment-cancelled`.
|
||||
|
||||
What the smoke proves:
|
||||
|
||||
- No sampled route failed navigation.
|
||||
- All sampled routes returned 200 through the preview server.
|
||||
- `/payment-cancelled` redirects to `/settings?tab=billing`.
|
||||
- Every sampled route produced meaningful body text and a screenshot.
|
||||
- No sampled route had document-level horizontal overflow.
|
||||
|
||||
What still remains outside route-existence T11:
|
||||
|
||||
- Route-existence ownership is now codified in `tests/e2e/user-journeys.spec.ts`; the all-route smoke remains supplemental screenshot/overflow evidence.
|
||||
- Auth-enabled route health remains tied to T1/T12 rather than this accountless route-existence lane.
|
||||
- `/launcher?watch=1` still logs `[adapter] detectTools failed: ... /api/tools/detect: Failed to fetch`, so Launcher watch mode needs T16 runtime evidence.
|
||||
- `/settings/usage` still logs a 403 resource error while rendering the Team-tier gate, so Usage & Cost semantics remain tied to T9.
|
||||
- The catch-all route intentionally renders the branded not-found page, but it currently logs the attempted bad route as a console error. The final route smoke should either demote this expected event or explicitly exclude it from critical console failure counts.
|
||||
- The DOM heuristic found runtime accessible-name gaps across judge routes, including chat composer (`ChatApp.tsx:1594`), Launcher refresh/prompt (`LauncherApp.tsx:332`, `:381`), Artifacts search/create (`ArtifactCenterApp.tsx:183`, `:231`), Agents search (`AgentsApp.tsx:284`), Skills search (`CapabilitiesApp.tsx:475`), Settings daily budget (`SettingsApp.tsx:506`), Vault refresh/add-secret controls (`VaultApp.tsx:257`, `:331`, `:380`), Profile identity fields (`UserProfileApp.tsx:327` through `:353`), WaggleDance refresh (`WaggleDanceApp.tsx:55`), Approvals refresh (`ApprovalsApp.tsx:199`), and Mission Control refresh (`CockpitApp.tsx:83`). These mostly close under T10; keep them there rather than reopening T11 route ownership.
|
||||
|
||||
## Direct Route Reference Matrix
|
||||
|
||||
Counts were generated with direct fixed-string search across `tests/e2e`, `tests/visual`, `tests/vision`, and `apps/web/src/test`.
|
||||
|
||||
| Route | Direct refs | Current interpretation |
|
||||
|---|---:|---|
|
||||
| `/auth` | 7 | Covered enough for route ownership; explicit auth-enabled confidence remains tied to T1/T12. |
|
||||
| `/` | 3 | Covered as shell index/redirect, but redirect flash remains judged through rendered shell evidence. |
|
||||
| `/home` | 81 | Strong route evidence owner. |
|
||||
| `/workspaces` | 105 | Strong references, but workspace destructive/manage flows still need state-specific proof. |
|
||||
| `/workspaces/:workspaceId/:tab?` | 68 | Strong references; `Ctrl+Shift+N` and active workspace fallback still tracked in T4. |
|
||||
| `/memory` | 135 | Strong references, with visual/native-dialog issues tracked elsewhere. |
|
||||
| `/artifacts` | 2 | Codified `J-route-coverage` now renders the Artifact/Library shell; delete/archive/empty/error state owner still needed. |
|
||||
| `/files` | 6 | Codified `J-route-coverage` now renders the storage/files shell; upload/preview/path/error states remain T12. |
|
||||
| `/agents` | 11 | Mixed. Agent center/builder has component coverage; route-level form/error evidence still needed. |
|
||||
| `/automations` | 4 | Codified `J-route-coverage` now renders Automation Center shell; builder, validation, pause/resume/logs need routed evidence. |
|
||||
| `/skills` | 47 | Mixed. Main issue is copy/install determinism rather than route existence. |
|
||||
| `/room` | 12 | Mixed. Parallel-agent empty/running/completed states need route evidence. |
|
||||
| `/waggle-dance` | 1 | Codified `J-route-coverage` now renders signal-sharing shell; value clarity and live signal states still need evidence. |
|
||||
| `/approvals` | 7 | Mixed. Tier-gated and revoke-all consequence evidence still needed. |
|
||||
| `/connectors` | 37 | Mixed. Good references, but credential/revoke/error/no-secret states need proof. |
|
||||
| `/mcps` | 4 | Codified `J-route-coverage` now renders MCP Hub installed/catalog/custom shell; MCP install/verify/scope/revoke states need routed evidence. |
|
||||
| `/marketplace` | 53 | Mixed. Search/browse is known flaky because standard audit can hit live external sync. |
|
||||
| `/launcher` | 1 | Codified `J-route-coverage` now renders `/launcher` and `/launcher?watch=1`; launch/prompt/hook lifecycle states still need evidence. |
|
||||
| `/team` | 9 | Mixed. Team admin route exists; Solo/Team tier gating and governance states need proof. |
|
||||
| `/settings` | 66 | Mixed. Mobile layout and native dialog issues remain P0/P1. |
|
||||
| `/settings/vault` | 7 | Mixed. Secret save/error/no-leak keyboard states need proof. |
|
||||
| `/settings/profile` | 2 | Codified `J-route-coverage` now renders profile form shell; save/error and mobile state evidence remain separate T10/T12 work. |
|
||||
| `/settings/mission-control` | 10 | Mixed. Visual baseline and local model pricing semantics remain open. |
|
||||
| `/settings/timeline` | 3 | Codified `J-route-coverage` now renders the timeline/activity shell; workspace timeline, filters, long-list, and date formatting states need proof. |
|
||||
| `/settings/events` | 6 | Mixed. Logs route needs long-line/filter/empty visual proof. |
|
||||
| `/settings/usage` | 4 | Codified `J-route-coverage` now renders usage/cost shell; unknown local model cost semantics remain open. |
|
||||
| `/benchmarks` | 0 | Codified `J-route-coverage` now renders the route; discovery/value evidence is still missing. |
|
||||
| `/platform` | 0 | Codified `J-route-coverage` now renders the route; judged-scope decision/discovery evidence is still missing. |
|
||||
| `/payment-success` | 3 | Codified `J-route-coverage` now renders the no-checkout fallback; completed checkout-return state is not proven. |
|
||||
| `/payment-cancelled` | 0 | Codified `J-route-coverage` now proves redirect to `/settings?tab=billing` plus billing recovery copy. |
|
||||
| Bad route / catch-all | 1 | Thin but present through invalid-route stress coverage. |
|
||||
|
||||
## Correction Candidates
|
||||
|
||||
### T11-1: Codify route-level evidence for zero-hit routes
|
||||
|
||||
Routes:
|
||||
|
||||
- `/benchmarks`
|
||||
- `/platform`
|
||||
- `/payment-cancelled`
|
||||
|
||||
Evidence:
|
||||
|
||||
- `App.tsx` registers all three routes.
|
||||
- Direct test reference count found zero route references for all three.
|
||||
- Benchmark and Platform have component tests, but those do not prove AppShell URL rendering, command palette discoverability, route chrome, or status-bar context.
|
||||
- Payment cancelled is a redirect; the original ad hoc smoke proved the redirect, and the current codified route test now proves that `/payment-cancelled` lands on Settings billing and explains the recovery action.
|
||||
- 2026-07-08 ad hoc fresh built-app smoke on port 3407 proves current URL rendering: `/benchmarks` and `/platform` return 200 with meaningful shell content, while `/payment-cancelled` returns 200 and redirects to `/settings?tab=billing`.
|
||||
- 2026-07-08 all-route built-preview smoke on port 3457 reproves those three routes in the same pass as the rest of the registered route table.
|
||||
- 2026-07-08 codified `J-route-coverage: thin utility routes render or redirect clearly` passed again on port `34200` and owns regression evidence for `/benchmarks`, `/platform`, and `/payment-cancelled`.
|
||||
- The older ad hoc smokes reproduced pre-fix T1 CSP/Clerk console errors. Current T1 console-health closure is tracked separately; this T11 route smoke proves route existence and recovery copy, not full per-route console health.
|
||||
|
||||
Acceptance:
|
||||
|
||||
- A user-journey or route-smoke test navigates to `/benchmarks`, `/platform`, and `/payment-cancelled` with skip params.
|
||||
- `/benchmarks` and `/platform` render the expected route chrome and content.
|
||||
- `/payment-cancelled` redirects to billing and leaves the user with clear next action copy.
|
||||
- The route smoke is codified so the current ad hoc evidence is not lost between judge runs.
|
||||
|
||||
Status: route-existence coverage is now codified. Deeper payment provider round-trip and recovery-state screenshots remain part of Team Admin/T12 evidence.
|
||||
|
||||
### T11-2: Upgrade thin route owners from reference count to user-state proof
|
||||
|
||||
Priority thin routes:
|
||||
|
||||
- `/launcher`
|
||||
- `/waggle-dance`
|
||||
- `/artifacts`
|
||||
- `/settings/profile`
|
||||
- `/settings/timeline`
|
||||
- `/payment-success`
|
||||
- `/automations`
|
||||
- `/mcps`
|
||||
- `/settings/usage`
|
||||
|
||||
Evidence:
|
||||
|
||||
- These routes have 1-4 direct references, or only static/component coverage.
|
||||
- Several are primary judge paths: Engineer uses Launcher/MCP/files/logs; Team admin uses payment recovery; Researcher uses timeline; Mobile executive uses profile/settings.
|
||||
- 2026-07-08 ad hoc thin-route smoke on port 3411 proves the priority thin URLs render, but it also shows route health is still capped by global T1 console errors, Launcher detection race/noise, and Usage/Cost 403 resource noise.
|
||||
- 2026-07-08 all-route built-preview smoke on port 3457 reproves the priority thin routes, expands the mobile route sample, and confirms no document-level horizontal overflow; it also confirms label/name gaps on Profile, Vault, Launcher, and other persona routes.
|
||||
- 2026-07-08 codified `J-route-coverage: priority thin routes render meaningful shells` passed again on port `34200` and owns route-existence evidence for `/launcher`, `/launcher?watch=1`, `/waggle-dance`, `/artifacts`, `/settings/profile`, `/settings/timeline`, `/payment-success`, `/automations`, `/mcps`, `/settings/usage`, and `/files`.
|
||||
|
||||
Acceptance:
|
||||
|
||||
- Each thin route gets an evidence owner: route smoke, visual snapshot, persona screenshot, or explicit deferral.
|
||||
- Evidence states the user mode, tier, viewport, and data state.
|
||||
- Thin route rows in the manifest are changed only after evidence exists.
|
||||
- Launcher detection and Usage/Cost 403 noise are resolved, documented as expected, or excluded from the final judge lane.
|
||||
|
||||
Status: route-existence ownership is codified. Persona-critical interaction states remain open under T10/T12/T16 as applicable.
|
||||
|
||||
### T11-3: Separate route existence from state coverage
|
||||
|
||||
Evidence:
|
||||
|
||||
- `p1a-routes.test.ts` proves `routeFor` and nav data invariants.
|
||||
- It does not mount `App.tsx`, render screens, test API-backed empty/error states, or exercise mobile layout.
|
||||
|
||||
Acceptance:
|
||||
|
||||
- The route manifest names both kinds of evidence:
|
||||
- route mapping/unit evidence; and
|
||||
- rendered route/state evidence.
|
||||
- Judge scorecards cannot cite `p1a-routes.test.ts` alone for a user-visible route.
|
||||
|
||||
### T11-4: Fix or document the app-test command shape
|
||||
|
||||
Evidence:
|
||||
|
||||
- Root `npm run test -- apps/web/src/test/p1a-routes.test.ts --run` exits with "No test files found" because the root Vitest include/exclude pattern excludes `apps/**`.
|
||||
- `npm run test -w apps/web -- src/test/p1a-routes.test.ts --run` works.
|
||||
|
||||
Impact:
|
||||
|
||||
- A future reviewer can think route tests are missing or broken when they used the root command.
|
||||
|
||||
Correction:
|
||||
|
||||
- Use app-local commands in the Phase 1 plan and T11 verification notes, or add a root script that intentionally targets app tests.
|
||||
|
||||
Acceptance:
|
||||
|
||||
- The T11 verification checklist uses commands that actually run app tests.
|
||||
|
||||
### T11-5: Overlay evidence must be tied to routes
|
||||
|
||||
Overlays:
|
||||
|
||||
- Command Center
|
||||
- Workspace Switcher
|
||||
- Persona Switcher
|
||||
- Spawn Agent
|
||||
- Onboarding Wizard
|
||||
- Login Briefing
|
||||
- Upgrade/Trial modals
|
||||
- Notification Inbox
|
||||
- Context Rail
|
||||
- Erase Data dialog
|
||||
|
||||
Evidence:
|
||||
|
||||
- The route manifest lists these overlays, but their evidence is not consistently attached to persona journeys.
|
||||
- Workspace Switcher is already a P0 because it can block route traversal.
|
||||
- Command Center has component tests, but route discovery paths for command-only routes still need URL proof.
|
||||
|
||||
Acceptance:
|
||||
|
||||
- Each judge persona cites overlay evidence where that overlay is in the path.
|
||||
- Route-changing overlays close predictably on selection, Escape, outside click, and route-changing nav.
|
||||
- Command Center can reach command-only surfaces or those surfaces are explicitly deferred.
|
||||
|
||||
## Five-Persona Impact
|
||||
|
||||
| Persona | T11 risk |
|
||||
|---|---|
|
||||
| Solo founder/operator | Home/workspace evidence is strong, but first-run and accountless confidence still depends on T1/T12 console and onboarding evidence. |
|
||||
| Researcher | Memory is strong; timeline, artifacts, export/delete, and Browser Companion-adjacent capture evidence are thinner. |
|
||||
| Engineer/power user | Launcher, MCP Hub, files, events, bad-route recovery, command-only surfaces, and T16/T17 adjunct surfaces need stronger routed evidence. |
|
||||
| Team admin/security reviewer | Payment cancelled/success, Vault, Approvals, Team governance, and Settings billing recovery need route/state proof. |
|
||||
| Mobile executive | Home/settings route coverage exists, but profile, usage, timeline, and mobile state evidence are thin. |
|
||||
|
||||
## Approval Recommendation
|
||||
|
||||
T11 is now closed for route-existence ownership: the route manifest exists, zero/thin routes have codified `J-route-coverage` owners, and the focused route coverage run passed 2/2 on port `34200`. Broader per-route state bundles stay in T12/Phase 2, with Launcher runtime proof in T16 and Usage semantics in T9.
|
||||
177
docs/audits/2026-07-08-runtime-a11y-t10-analysis.md
Normal file
177
docs/audits/2026-07-08-runtime-a11y-t10-analysis.md
Normal file
@@ -0,0 +1,177 @@
|
||||
# Focused Runtime Accessibility T10 Analysis
|
||||
|
||||
Status: original runtime analysis plus follow-up implementation notes. The axe table below records the pre-fix built-app smoke; the 2026-07-08 T10 update records the focused controls now covered by regression tests, plus the new zero-violation runtime axe gate for the sampled core routes.
|
||||
|
||||
Purpose: add rendered accessibility evidence for high-traffic routed surfaces. The earlier Web Guidelines supplement is a static source scan; this file records what axe-core and DOM heuristics found when the built app actually rendered under the standard E2E skip harness.
|
||||
|
||||
Guideline source refreshed during this pass: Vercel Web Interface Guidelines, `https://raw.githubusercontent.com/vercel-labs/web-interface-guidelines/main/command.md`. The rules most relevant here are named icon buttons, named form controls, keyboard-reachable scroll regions, semantic headings/landmarks, visible focus, and long label handling.
|
||||
|
||||
## Runtime Evidence
|
||||
|
||||
Environment:
|
||||
|
||||
```powershell
|
||||
$env:WAGGLE_PORT='3423'
|
||||
$env:WAGGLE_TRUST_LOCALHOST='1'
|
||||
$env:WAGGLE_DISABLE_MARKETPLACE_SYNC='1'
|
||||
$env:WAGGLE_DATA_DIR="$env:TEMP\waggle-a11y-smoke-3423"
|
||||
$env:EMBEDDING_PROVIDER='mock'
|
||||
$env:VITE_CLERK_PUBLISHABLE_KEY=''
|
||||
$env:CLERK_SECRET_KEY=''
|
||||
npm run build
|
||||
npx tsx packages/server/src/local/start.ts --skip-litellm
|
||||
```
|
||||
|
||||
Rendered URL shape:
|
||||
|
||||
```text
|
||||
?skipOnboarding=true&skipBoot=true&skipBriefing=true&tier=power
|
||||
```
|
||||
|
||||
Evidence artifacts:
|
||||
|
||||
```text
|
||||
output/playwright/a11y-runtime-3423/runtime-a11y-summary-skip.json
|
||||
output/playwright/a11y-runtime-3423/runtime-a11y-summary.json
|
||||
output/playwright/route-evidence-3457/all-route-smoke-summary.json
|
||||
output/playwright/route-evidence-3457/all-route-smoke.json
|
||||
```
|
||||
|
||||
`runtime-a11y-summary.json` is retained as a harness caveat: without the skip parameters, desktop direct routes can render first-run/auth state rather than the intended app surface. `runtime-a11y-summary-skip.json` is the authoritative route-content evidence from this pass.
|
||||
|
||||
Implementation note: axe was injected with Playwright `bypassCSP: true`. A normal `page.addScriptTag()` was correctly blocked by the app CSP, which is consistent with the known T1 lane. This bypass was used only to inspect accessibility; it is not product behavior.
|
||||
|
||||
Color contrast was disabled in the axe run because Waggle already has dedicated `npm run ux:contrast` and `npm run ux:color-guard` gates.
|
||||
|
||||
## Post-Fix Runtime Gate
|
||||
|
||||
The ad hoc audit is now codified as `tests/e2e/runtime-a11y.spec.ts`.
|
||||
|
||||
Latest evidence:
|
||||
|
||||
```powershell
|
||||
$env:WAGGLE_E2E_PORT='34193'
|
||||
$env:WAGGLE_E2E_BASE_URL='http://localhost:34193'
|
||||
npx playwright test tests/e2e/runtime-a11y.spec.ts --project=chromium --reporter=line
|
||||
```
|
||||
|
||||
Result: pass, 2/2. The gate runs axe-core against Home, Settings, Profile, Vault, Mission Control, Memory, workspace chat, Agents, Waggle Dance, Launcher, MCP Hub, Files, and Approvals at desktop 1440 x 900 and mobile 390 x 844. It currently expects zero axe violations for this sampled route set.
|
||||
|
||||
## Routes Sampled
|
||||
|
||||
Each route below returned HTTP 200 and matched expected route text in both desktop 1440 x 900 and mobile 390 x 844 contexts when loaded with the skip harness.
|
||||
|
||||
| Route | Desktop axe result | Mobile axe result | Runtime concern |
|
||||
|---|---|---|---|
|
||||
| `/home` | `region` moderate | `region` moderate | Some status/topbar content is outside landmarks; ask input is named but lacks form metadata. |
|
||||
| `/settings` | `button-name` critical, `select-name` critical, `region` moderate | Same | A tooltip/icon button lacks a discernible name; Prompt Shape select lacks an associated accessible name; Settings form metadata remains weak. |
|
||||
| `/settings/profile` | `select-name` critical, `region` moderate | Same | Profile selects lack associated accessible names; several profile fields lack `name`/`autocomplete` metadata. |
|
||||
| `/settings/vault` | Not in original axe sample | Not in original axe sample | Added to the codified post-fix gate after the Vault metadata slice. |
|
||||
| `/settings/mission-control` | Not in original axe sample | Not in original axe sample | Added to the codified post-fix gate; expansion exposed ComplianceDashboard unnamed action buttons and heading order, now fixed. |
|
||||
| `/memory` | `region` moderate | Same | Original concern: Memory search/list controls had weak metadata and top/status content outside landmarks repeated. Post-fix Memory Center metadata/focus coverage is recorded below. |
|
||||
| `/workspaces/default-workspace/chat` | `image-alt` critical, `aria-allowed-role` minor, `region` moderate | Same | Workspace tab panel semantics and file/preview/icon imagery need source-level verification; chat composer lacks form metadata. |
|
||||
| `/agents` | Not in original axe sample | Not in original axe sample | Added to the codified post-fix gate after the agent-card control slice. |
|
||||
| `/waggle-dance` | Not in original axe sample | Not in original axe sample | Added to the codified post-fix gate after the WaggleDance refresh-control slice. |
|
||||
| `/launcher` | `button-name` critical, `region` moderate | Same | Refresh icon button lacks a name; prompt textarea lacks form metadata. |
|
||||
| `/mcps` | `region` moderate | Same | No named-control violations in this smoke; landmark issue repeats. |
|
||||
| `/files` | `scrollable-region-focusable` serious, `heading-order` moderate, `region` moderate | Same | Files has a keyboard-inaccessible scroll region and heading order issue. |
|
||||
| `/approvals` | `button-name` critical, `region` moderate | Same | Refresh/revoke icon buttons lack accessible names. |
|
||||
|
||||
Implementation update 2026-07-08 T10:
|
||||
|
||||
- Settings now names the telemetry toggle, associates the Prompt Shape select and high-traffic model/team/KVARK fields, and adds form metadata for daily budget, mutation gate, URLs, and tokens.
|
||||
- Profile identity, writing-style, brand, and language controls now have explicit labels, stable `name` values, and autocomplete metadata; the Analyze Style action now has a token focus ring.
|
||||
- Launcher now names the refresh icon button and the optional launch prompt textarea.
|
||||
- Approvals now names refresh and per-grant revoke icon buttons.
|
||||
- Cockpit, WaggleDance, ComplianceDashboard, and custom AgentCard delete controls now expose accessible names; AgentCard now separates selection and delete into distinct labelled buttons instead of nesting the delete action inside a selectable card.
|
||||
- ComplianceDashboard report template/date controls and ComplianceTemplateModal create/edit fields now expose associated labels, stable name/autocomplete metadata, and token `focus-visible:ring-2` focus rings.
|
||||
- All Workspaces and Wiki search inputs now have stable names/autocomplete metadata and visible focus-ring replacements.
|
||||
- Files storage overview and file-browser scroll panes now expose named, keyboard-focusable regions with visible focus rings.
|
||||
- Files new-folder and inline rename fields now expose accessible names, stable name/autocomplete metadata, disabled spellcheck for file names, and token focus rings.
|
||||
- Files bulk Move and file Properties dialogs now expose named icon-only close controls with token focus rings, and the file row context menu preserves the file-specific Properties action.
|
||||
- Chat composer and Vault add-secret controls now have accessible names plus stable `name`/autocomplete metadata; the Chat composer also has a token focus ring, and Vault refresh/edit/reveal/delete controls are named.
|
||||
- Memory Center list/detail controls now expose stable search/select/editor metadata, focus rings for chips and actions, and named re-import/detail-editor paths.
|
||||
- MemoryCard selection checkboxes now expose memory-specific accessible names plus stable name/value metadata.
|
||||
- Memory Trust Manage search and correction editor now expose stable metadata and token focus rings.
|
||||
- TimelineTab sidebar search, filter toggle, and minimum-importance slider now expose accessible names/label association, stable metadata, and token focus rings.
|
||||
- TimelineApp event-type filtering now pairs its stable metadata with `autocomplete="off"`.
|
||||
- EvolutionTab proposal review note now associates its label with the textarea, exposes stable metadata, and uses a token focus ring.
|
||||
- EvolutionTab New Run modal now names the close icon and exposes associated labels, stable metadata, select/textarea autocomplete, and token focus rings for target, baseline, and schema controls.
|
||||
- ConnectorCard row actions and Jira credential setup now expose stable email/token metadata, correct email/password semantics, disabled spellcheck, and visible focus rings.
|
||||
- ExtensionCard marketplace inline connector-token paste now exposes connector-specific labels, stable name/autocomplete metadata, disabled spellcheck, and focus-ring coverage.
|
||||
- InstallAuditPanel marketplace audit type filter now exposes stable name/autocomplete metadata and a token focus ring.
|
||||
- ModelPilotCard budget threshold slider now exposes an accessible name, stable name, and token focus ring while preserving update behavior.
|
||||
- TelemetryApp daily budget input now exposes stable name/autocomplete metadata.
|
||||
- SkillEditorDrawer markdown textarea now exposes stable name/autocomplete metadata with spellcheck disabled.
|
||||
- Agent template custom-agent, group-builder, and group-detail task-runner controls now expose stable labels/names/autocomplete metadata, visible focus rings, and `aria-pressed` strategy state.
|
||||
- Agent Center templates search now exposes contextual accessible names plus stable name/autocomplete metadata for persona/group search.
|
||||
- Automation Center template workspace and assist-mode controls now expose stable name/autocomplete metadata and token focus rings.
|
||||
- AgentCard and GroupCard now separate selection and delete into distinct labelled buttons, remove invalid nested interactive structures, and replace broad transitions with explicit transition properties.
|
||||
- SuggestedAgentCards now gives persona media explicit dimensions and replaces the browse affordance's broad transition with explicit color/transform transitions.
|
||||
- CreateWorkspaceDialog now scopes chip, template, storage, persona, and agent-group transitions to explicit color/transform properties.
|
||||
- WorkspaceSwitcher and PersonaSwitcher now scope switcher row/card transitions to explicit color properties.
|
||||
- ConnectorCard, BrandTile, and McpCatalog now scope connector row, brand tile, distribution, and category-filter transitions to explicit properties.
|
||||
- TelemetryApp, SurfaceToggle, and workspace TasksTab now scope meter, switch-knob, and delete-action transitions to explicit properties.
|
||||
- ArtifactCenterApp, DashboardApp, HomeCockpit, and MarketplaceApp now close the remaining broad-transition backlog with explicit card/chip transition properties.
|
||||
- SpawnAgentDialog launch task/new-workspace fields, McpCatalog catalog search, ArtifactCenterApp detail editor controls, ModelGate cloud-key/local-pull controls, inline CapabilityRequestCard connector-token entry, and TelegramDigestCard credential fields now expose associated labels or accessible names plus stable `name`/autocomplete metadata.
|
||||
- AgentCenterRow now gives list-row persona media explicit dimensions in the rendered Agent Center route coverage.
|
||||
- ReadyStep now gives the onboarding completion logo explicit dimensions.
|
||||
- BootScreen and StatusBar now give persistent brand/logo media explicit dimensions.
|
||||
- LoginBriefing, the ChatApp empty state, and SpawnAgentDialog persona picker/review media now give mascot/persona images explicit dimensions.
|
||||
- First-run onboarding profile name/role, workspace-name, and first-task controls now expose stable name/autocomplete metadata.
|
||||
- EraseDataDialog now associates the destructive confirmation label with the phrase field, adds stable name/autocomplete metadata, and exposes a token focus-visible ring.
|
||||
- McpScopeDialog target-workspace select now exposes stable name/autocomplete metadata and a token focus-visible ring.
|
||||
- CreateWorkspaceDialog visible setup fields, template search, template-creator AI/name fields, and folder-picker new-folder field now expose stable name/autocomplete metadata and accessible labels/names.
|
||||
- AllWorkspacesApp search now pairs its stable metadata with an input-level token focus ring.
|
||||
- WikiTab search now pairs stable metadata with an input-level token focus ring, and the Obsidian/Notion export target fields expose target-specific metadata and token focus rings.
|
||||
- ArtifactCenterApp detail Kind select now pairs its stable metadata with a token focus ring.
|
||||
- LauncherApp optional launch prompt now pairs its stable metadata with a token focus ring.
|
||||
- The sampled runtime axe defects are closed: Model Pilot info is named and expanded-state aware, shared avatars default to decorative `alt=""` unless callers provide text, workspace tab panels use an allowed role host, file preview placeholders expose image labels, storage headings preserve order, toast close controls are named, and the persistent status bar is a labeled header landmark.
|
||||
- Focused regression coverage: `settings-trust.test.tsx`, `timeline-app.test.tsx`, `wiki-export-trust.test.tsx`, `UserProfileApp.test.tsx`, `VaultApp.test.tsx`, `lane-c-input-power.test.tsx`, `launcher-a11y.test.tsx`, `p7-b1-approvals-error.test.tsx`, `p7-b5-error-threading.test.tsx`, `memory-center-trust.test.tsx`, `pr35-memory-trust-manage.test.tsx`, `CockpitApp.test.tsx`, `ComplianceDashboard.test.tsx`, `compliance-template-trust.test.tsx`, `AgentCard.test.tsx`, `GroupCard.test.tsx`, `SuggestedAgentCards.test.tsx`, `phase3b-agent-center.test.tsx`, `phase3c-agent-builder.test.tsx`, `phase3c-skill-builder.test.tsx`, `phase3c-automation-builder.test.tsx`, `phase3b-automation-center.test.tsx`, `AgentTemplateForms.test.tsx`, `phase4b-mcp-hub.test.tsx`, `phase4b-connector-hub.test.tsx`, `phase4b-marketplace-extend.test.tsx`, `StorageAndFilesApp.test.tsx`, `KnowledgeGraphViewer.test.tsx`, `HarvestTab.test.tsx`, `wave-w-briefing-entrance.test.tsx`, `wave-u-chat-action-row.test.tsx`, `SpawnAgentDialog.test.tsx`, `ModelGate.test.tsx`, `artifact-center-trust.test.tsx`, `pr4-inline-capability.test.tsx`, `TelegramDigestCard.test.tsx`, `WhoAreYouStep.test.tsx`, `WorkspaceCreateStep.test.tsx`, `FirstTaskStep.test.tsx`, `EraseDataDialog.test.tsx`, `shell-overlay-contracts.test.tsx`, the `/files` and Command Center branches of `tests/e2e/user-journeys.spec.ts`, and `tests/e2e/runtime-a11y.spec.ts`.
|
||||
|
||||
## Command Center Runtime Result
|
||||
|
||||
Using the exact existing E2E shortcut shape, `Control+k`:
|
||||
|
||||
- Desktop: Command Center opens, focus lands in the search input, Escape closes it, no visible element overflow.
|
||||
- Mobile 390 x 844: Command Center opens and Escape closes it, but long subtitles overflow the dialog width.
|
||||
- Both desktop and mobile log: `Warning: Missing Description or aria-describedby={undefined} for {DialogContent}.`
|
||||
- The dialog text still includes active `Pinned - Pro` copy, which is already covered by T3.
|
||||
|
||||
Implementation update 2026-07-08 T10/T12: Command Center now includes a hidden dialog description, catalog subtitles render as their own truncating line, and `J-mobile: Command Center is described and fits at 390px` proves description, Escape close, and zero visible row overflow on a fresh production build.
|
||||
|
||||
This refines the earlier mobile smoke: the failed close result came from an ad hoc uppercase shortcut path. The close path passes with the current E2E shortcut shape. The focused label-fit/dialog-description contract is now fixed, and the expanded route axe gate is green.
|
||||
|
||||
## Source Owners To Verify During Implementation
|
||||
|
||||
These are likely owners from source inspection; implementation must re-read the files immediately before editing.
|
||||
|
||||
| Runtime finding | Likely source owner |
|
||||
|---|---|
|
||||
| Settings unnamed tooltip/icon button and unnamed Prompt Shape select | Fixed for the sampled route gate: telemetry toggle and Prompt Shape are named, and runtime axe is green. |
|
||||
| Profile unnamed select and weak form metadata | Focused update landed for identity, writing-style, brand, and language controls; runtime axe includes `/settings/profile` and is green. |
|
||||
| Launcher unnamed refresh icon button | Fixed for the sampled route gate: refresh and optional prompt controls are named, and runtime axe is green. |
|
||||
| Approvals unnamed refresh/revoke icon buttons | Fixed for the sampled route gate: refresh and per-grant revoke controls are named, and runtime axe is green. |
|
||||
| All-route smoke: additional placeholder-only or unassociated visible fields | Chat composer metadata/focus, Vault add-secret, Profile preference/brand controls and Analyze Style focus, Agent/Skill/Automation Builder fields, SkillEditorDrawer markdown editor, Automation Center template controls, Agent template custom-agent/group-builder/group-detail controls, Agent Center templates search, ExtensionCard inline connector-token paste, InstallAuditPanel audit filter, ModelPilotCard budget-threshold slider, TelemetryApp daily budget, Spawn Agent launch controls, Artifact Center search/create/detail controls, Agent Center search, Skills Hub search, ConnectorCard credential controls, Memory Center search/detail controls, MemoryCard selection checkbox labels/metadata, Memory Trust search/correction controls and focus rings, TimelineTab search/filter controls, TimelineApp event-type select metadata, EvolutionTab proposal review note and New Run modal controls, Knowledge Graph search/scope controls, Harvest import controls, Custom MCP form controls, MCP catalog search and scope select, ModelGate key/pull controls, inline capability connector-token entry, Telegram digest credentials, first-run onboarding profile/workspace/first-task controls, EraseDataDialog destructive confirmation, Create Workspace visible setup/template/folder-picker fields, Compliance Dashboard report options, Compliance Template form fields, Wiki search/export target controls, and Files new-folder/rename fields are fixed. Remaining representative owners are other less-traveled form surfaces outside the sampled route gate. |
|
||||
| All-route smoke: additional unnamed icon-only controls | Vault refresh/edit/reveal/delete, WaggleDance refresh, Cockpit refresh, ComplianceDashboard report actions, AgentCard custom-delete/separate selection, GroupCard custom-delete/separate selection, Skill Builder reorder/remove controls, Files toolbar actions and move/properties dialog close controls, Mission Control refresh/pause/resume/stop controls, ConnectorCard row actions, Knowledge Graph toolbar/legend controls, and Harvest refresh/source actions are fixed. Remaining representative owners include broader modal controls outside the current gate. |
|
||||
| Workspace `role="tabpanel"` axe warning | Fixed in `WorkspaceDesktopApp.tsx`; the tab panel now sits on a `section` instead of `main`, and the runtime axe gate is green. |
|
||||
| Workspace/chat image or preview alt warning | Fixed through shared `AvatarImage` default alt text and file preview placeholder labeling; the runtime axe gate is green. |
|
||||
| Files scrollable region and heading order | Fixed for sampled `/files` route: Storage and Files scroll panes are named/focusable, storage card headings preserve order, and runtime axe is green. |
|
||||
| Command Center missing dialog description and mobile subtitle overflow | Focused update landed in `apps/web/src/components/os/overlays/CommandCenter.tsx`; unit coverage and `J-mobile: Command Center is described and fits at 390px` prove the dialog description and mobile row-fit contract. |
|
||||
| Landmark `region` warning across many routes | Fixed in `StatusBar.tsx`; the persistent top chrome is now a labeled `header` landmark, and runtime axe is green. |
|
||||
|
||||
## Correction Candidates
|
||||
|
||||
| ID | Correction | Phase recommendation | Closure evidence |
|
||||
|---|---|---|---|
|
||||
| T10-H | Codify a small runtime a11y smoke using axe-core for the five-persona route set. | Fixed 2026-07-08 for sampled routes | `tests/e2e/runtime-a11y.spec.ts` covers Home, Settings, Profile, Vault, Mission Control, Memory, Chat, Agents, Waggle Dance, Launcher, MCP, Files, and Approvals in desktop/mobile and passes 2/2 on port `34193`. |
|
||||
| T10-I | Add accessible names to icon-only buttons found at runtime. | Partially implemented 2026-07-08; Mission Control, ConnectorCard, Knowledge Graph, Harvest source controls, GroupCard delete, TimelineTab filter toggle, and Files dialog close controls fixed 2026-07-09 | Focused tests cover Settings telemetry, Launcher refresh, Approvals refresh/revoke, Vault actions, WaggleDance refresh, Cockpit refresh, ComplianceDashboard report actions, AgentCard custom-delete controls, GroupCard custom-delete controls, Files toolbar actions and move/properties dialog close controls, Mission Control refresh/pause/resume/stop controls, ConnectorCard row actions, Knowledge Graph toolbar/legend controls, Harvest refresh/source actions, and TimelineTab filter toggle; runtime axe confirms sampled `button-name` closure. Continue with remaining unsampled icon-only controls. |
|
||||
| T10-J | Associate labels with native selects and add form metadata to high-traffic fields. | Partially implemented 2026-07-08; builder, route-search, ConnectorCard, ExtensionCard inline token paste, ModelPilotCard budget slider, AllWorkspaces search focus, Artifact Center detail Kind focus, Launcher prompt focus, TelemetryApp daily budget, SkillEditorDrawer markdown editor, Memory Center, MemoryCard selection checkbox labels/metadata, Memory Trust Manage, TimelineTab, EvolutionTab review note/New Run modal select/textarea metadata, Knowledge Graph, Harvest, Custom MCP, MCP scope select, agent template, Automation Center template controls, Agent Center templates search, InstallAuditPanel audit filter, Spawn Agent, ModelGate, Telegram, inline capability, first-run onboarding, EraseDataDialog, Create Workspace, compliance report/template, Chat composer focus, and Files inline-field/action metadata fixed 2026-07-09 | Focused tests cover Settings, Profile identity/preferences/brand and Analyze Style focus, Launcher prompt controls/focus, Chat composer metadata/focus, Vault add-secret controls, Agent/Skill/Automation Builder controls, SkillEditorDrawer markdown editor, Automation Center template controls, Agent template creator/detail controls, Agent Center templates search, ExtensionCard inline connector-token paste, InstallAuditPanel audit filter, ModelPilotCard budget-threshold slider, AllWorkspaces search focus, TelemetryApp daily budget, Spawn Agent launch controls, Artifact Center search/create/detail controls and detail Kind focus, Agent Center search, Skills Hub search, ConnectorCard credential controls, Memory Center search/detail controls, MemoryCard selection checkbox labels/metadata, Memory Trust search/correction controls and focus rings, TimelineTab search/filter controls, EvolutionTab proposal review note/New Run modal select/textarea controls, Knowledge Graph search/scope controls, Harvest import controls, Custom MCP form controls, MCP catalog search/scope controls, ModelGate key/pull controls, inline capability connector-token entry, Telegram digest credential controls, first-run onboarding profile/workspace/first-task controls, EraseDataDialog destructive confirmation, Create Workspace visible setup/template/folder-picker controls, Compliance Dashboard report options, Compliance Template form controls, and Files new-folder/rename fields; runtime axe is green for the sampled route set. Continue with remaining less-traveled forms. |
|
||||
| T10-K | Fix keyboard access for scrollable route regions. | Fixed 2026-07-08 for sampled Files route | Focused tests cover the Storage overview and Files browser scroll panes as named `tabIndex=0` regions; runtime axe confirms sampled `scrollable-region-focusable` closure. |
|
||||
| T10-L | Resolve workspace tab panel semantics and preview/image accessible text. | Fixed 2026-07-08 for sampled workspace route | Workspace/chat axe `aria-allowed-role` and `image-alt` findings are gone in `tests/e2e/runtime-a11y.spec.ts`. |
|
||||
| T10-M | Add or correct Command Center dialog description and mobile long-label handling. | Focused fixed 2026-07-08 | Unit tests cover `DialogDescription` and truncating catalog subtitles; the mobile E2E route proves the dialog is described, closes with Escape, and has no visible row overflow at 390 x 844. |
|
||||
| T10-N | Decide shell landmark strategy for StatusBar/top chrome. | Fixed 2026-07-08 | `StatusBar` is now a labeled `header` landmark; repeated axe `region` warnings are gone in the runtime gate. |
|
||||
|
||||
## Current Recommendation
|
||||
|
||||
Keep Phase 1 unchanged except for Settings controls touched by T2. The first T10 follow-up slices now include a repeatable runtime axe gate with zero sampled desktop/mobile violations across 13 routes, plus focused fixes for the highest-noise named-control findings, Files scroll-region keyboard access, toolbar controls, and inline new-folder/rename fields, workspace semantics/image text, shell landmarks, Chat/Profile/Vault metadata/action focus, Agent/Skill/Automation Builder metadata, SkillEditorDrawer markdown-editor metadata, Automation Center template metadata/focus, Agent template creator/detail metadata/focus, Agent Center templates search metadata, ExtensionCard inline connector-token metadata, InstallAuditPanel audit-filter metadata, TelemetryApp daily-budget metadata, Spawn Agent launch metadata, Artifact/Agent/Skills route search and detail metadata, ConnectorCard setup metadata/actions, Memory Center search/detail metadata, MemoryCard selection checkbox labels/metadata, Memory Trust search/correction metadata, TimelineTab search/filter metadata, EvolutionTab review-note/New Run modal metadata, Knowledge Graph toolbar/search/scope controls, Harvest import/source controls, Custom MCP form and catalog-search/scope metadata, ModelGate key/pull metadata, inline capability connector-token metadata, Telegram digest credential metadata, first-run onboarding profile/workspace/first-task metadata, EraseDataDialog destructive confirmation metadata/focus, Create Workspace visible setup/template/folder-picker metadata, Compliance Dashboard report-option metadata, Compliance Template form metadata/focus, Mission Control/Agents/WaggleDance icon controls, scoped production transition-all closure, AgentCenterRow media stability, ReadyStep media stability, BootScreen/StatusBar media stability, LoginBriefing/Chat/SpawnAgentDialog media stability, and Command Center description/mobile fit. Final judge scoring still needs any remaining unsampled form metadata outside the covered surfaces, remaining unsampled icon-only controls, and modal focus-return evidence.
|
||||
119
docs/audits/2026-07-08-shell-overlays-t10-t12-analysis.md
Normal file
119
docs/audits/2026-07-08-shell-overlays-t10-t12-analysis.md
Normal file
@@ -0,0 +1,119 @@
|
||||
# Shell Overlay T10/T12 Analysis
|
||||
|
||||
Status: focused supplement with partial Phase 2 implementation update. The original evidence remains as the pre-fix baseline; the update below records the current post-fix state for the shell overlay contract slice.
|
||||
|
||||
Purpose: verify the shell overlays and interruption surfaces that users hit while navigating, switching context, spawning agents, creating workspaces, handling notifications, and seeing tier gates.
|
||||
|
||||
## Evidence
|
||||
|
||||
- Built current web bundle with `npm run build`: pass. Vite emitted the existing Tailwind arbitrary-value warnings, one dynamic/static import warning for `shape-selection.ts`, and the large main chunk warning.
|
||||
- Started a fresh sidecar on `http://127.0.0.1:3437` with isolated `WAGGLE_DATA_DIR`, mock embeddings, marketplace sync disabled, and `--skip-litellm`.
|
||||
- Ran Playwright desktop smoke at 1440 x 980:
|
||||
- `output/playwright/shell-overlays-3437/summary.json`
|
||||
- screenshots `00-baseline.png` through `07-upgrade-modal.png`
|
||||
- Ran Playwright mobile smoke at 390 x 844:
|
||||
- `output/playwright/shell-overlays-3437/mobile-summary.json`
|
||||
- screenshots `mobile-00-baseline.png` through `mobile-03-create-workspace.png`
|
||||
- Source inspected:
|
||||
- `apps/web/src/components/os/overlays/NotificationInbox.tsx`
|
||||
- `apps/web/src/components/os/overlays/KeyboardShortcutsHelp.tsx`
|
||||
- `apps/web/src/components/os/overlays/ContextRail.tsx`
|
||||
- `apps/web/src/components/os/overlays/CreateWorkspaceDialog.tsx`
|
||||
- `apps/web/src/components/os/overlays/PersonaSwitcher.tsx`
|
||||
- `apps/web/src/components/os/overlays/SpawnAgentDialog.tsx`
|
||||
- `apps/web/src/components/os/overlays/WorkspaceSwitcher.tsx`
|
||||
- `apps/web/src/components/os/overlays/UpgradeModal.tsx`
|
||||
- `apps/web/src/components/os/overlays/TrialExpiredModal.tsx`
|
||||
- `apps/web/src/components/os/overlays/OnboardingTooltips.tsx`
|
||||
|
||||
## Runtime Results
|
||||
|
||||
| Surface | Desktop result | Mobile result | Verdict |
|
||||
|---|---|---|---|
|
||||
| Keyboard Shortcuts | `role="dialog"`, labelled, focus contained, Escape closes | Not sampled in mobile smoke | Pass for desktop shell path. |
|
||||
| Persona Switcher | `role="dialog"`, labelled, focus contained, Escape closes | Not sampled in mobile smoke | Pass for desktop shell path. |
|
||||
| Spawn Agent | Opens and closes with Escape; task/model flow renders | Opens without horizontal overflow | Mostly pass; Radix dialog focus works, but model/task controls still need T10 form detail in the wider accessibility pass. |
|
||||
| Workspace Switcher | `role="dialog"`, labelled, focus contained | Entry to Create Workspace works | Pass for isolated open/close; T4 route-trap issue remains separate. |
|
||||
| Upgrade Modal | `role="dialog"`, labelled, focus contained, Escape closes, named close action. Rendered event path covered by `J3c`. | Not sampled in mobile smoke | Pass for the sampled tier-interruption contract; broader billing/upgrade flow remains outside this overlay slice. |
|
||||
| Notification Inbox | Pre-fix: opened visually with no `role="dialog"`/landmark, Escape left it open, 2 unnamed icon buttons. Post-fix: named dialog, focus trap, Escape close, named mark-all/close actions, and no horizontal entrance motion that can overflow mobile. | Mobile Executive bundle now captures Notification Inbox at 390 x 844 with 0 visible overflow and screenshot evidence. | Pass for the codified overlay contract; keep broader notification content states in T12. |
|
||||
| Create Workspace | Pre-fix: no labelled dialog, Escape left it open, unnamed icon buttons, native template-delete `confirm()`. Post-fix: named main dialog plus named Folder Picker / Template Creator subdialogs, focus traps, Escape close, named close/template/share actions, and in-app template-delete confirmation. | Post-fix 390 x 844 rendered path prioritizes name, storage, optional template disclosure, and a visible Create action without horizontal overflow. | Partial: sampled T10/T7/mobile hierarchy contracts are fixed; final score still needs screenshot refresh and broader route/state evidence. |
|
||||
| Context Rail | Post-fix component evidence: labelled `complementary` panel, named close action, and expandable item state. | Source-only for mobile | Pass for source/component-level landmark contract; full rendered entry states still need route-specific evidence. |
|
||||
| Onboarding Tooltips | Post-fix component evidence: named non-modal dialog, `aria-modal="false"`, and global Escape dismissal that records the dismissed state. | Source-only for mobile | Pass for the semantic decision and keyboard dismiss contract; final score still needs broader first-run/mobile state evidence. |
|
||||
| Trial Expired Modal | Source-only: labelled dialog/focus trap exists; post-fix component evidence covers named close action. | Source-only | Pass for the close-label contract; full trial-expired account state remains T12/billing evidence. |
|
||||
|
||||
The desktop and mobile runs reproduced the same T1 Clerk/CSP console errors already recorded in the first-run supplement. They are not new overlay findings, but they keep the standard local audit lane noisy until T1 is fixed.
|
||||
|
||||
## Findings
|
||||
|
||||
### P1-19: Shell overlays do not share a consistent accessibility/close contract
|
||||
|
||||
Implementation update 2026-07-08:
|
||||
|
||||
- `NotificationInbox.tsx` now uses `useFocusTrap`, `role="dialog"`, `aria-modal`, a labelled title, Escape close, and explicit labels for "Mark all notifications as read" and "Close notifications".
|
||||
- `CreateWorkspaceDialog.tsx` now uses `useFocusTrap`, `role="dialog"`, `aria-modal`, a labelled title, Escape close, and explicit labels for close, template select/duplicate/edit/delete, the share toggle, Folder Picker, and Template Creator.
|
||||
- `ContextRail.tsx` now exposes the rail as a labelled complementary panel and gives the close icon a name.
|
||||
- `OnboardingTooltips.tsx` now has an explicit non-modal dialog contract and Escape dismissal without trapping focus.
|
||||
- `UpgradeModal.tsx` and `TrialExpiredModal.tsx` now give their close icon controls accessible names.
|
||||
- Codified component evidence: `npm run test -w apps/web -- src/test/shell-overlay-contracts.test.tsx --reporter=dot` passed 1 file / 8 tests after adding Create Workspace hierarchy and subdialog assertions.
|
||||
- Codified rendered evidence: `tests/e2e/user-journeys.spec.ts` includes `J3b: notification and create-workspace overlays have dialog close contracts`, `J3c: tier interruption modal exposes a named close contract`, and `J-mobile: create workspace prioritizes primary setup at 390px width`. `J3b` passed previously on port `34151`; focused `J3c` passed on port `34153`; the earlier full journey suite passed 18/18 on port `34155`; focused `J-mobile` passed on port `34157`; the expanded full journey suite passed 19/19 on port `34158`. Current five-persona evidence adds Mobile Executive Notification Inbox and Command Center overlay screenshots with 0 visible overflow in the 5/5 run on port `34267`.
|
||||
- Browser-plugin spot check attempted on port `34154`, but the in-app Browser DOM snapshot path failed with `incrementalAriaSnapshot is not a function`; Playwright remains the reliable rendered evidence lane for this slice.
|
||||
- Remaining P1-19 scope: route-specific rendered evidence for Context Rail/Onboarding Tooltips/trial-expired states is still open.
|
||||
|
||||
Evidence:
|
||||
|
||||
- `NotificationInbox.tsx:38-60` renders a fixed overlay with click-to-close but no dialog/menu role, no labelled container, no focus trap, and no Escape close handler. Runtime: `notifications-after-escape.stillVisible = true`.
|
||||
- `NotificationInbox.tsx:57` and `:60` expose icon-only actions without accessible names for "mark all read" and "close"; the tooltip is not a reliable button name.
|
||||
- `CreateWorkspaceDialog.tsx:757-766` renders the main creation overlay without `role="dialog"`, `aria-modal`, `aria-labelledby`, or the shared `useFocusTrap`; runtime: `create-workspace-after-escape.stillVisible = true`.
|
||||
- `CreateWorkspaceDialog.tsx:857`, `:867`, `:910`, `:918`, and neighboring icon-only actions are visible controls without accessible names; runtime counted 16 visible unnamed icon buttons while the dialog was open.
|
||||
- `ContextRail.tsx:62-75` renders a fixed side rail and close button without a labelled landmark or accessible close name.
|
||||
- `OnboardingTooltips.tsx:97-137` renders a centered overlay with "Dismiss all" and "Next/Got it" actions but no explicit modal/non-modal semantics or keyboard close path.
|
||||
- `UpgradeModal.tsx:90` and `TrialExpiredModal.tsx:58-62` have modal focus behavior, but their close icon buttons are unnamed.
|
||||
|
||||
Correction:
|
||||
|
||||
- Define one shared overlay contract: modal overlays use `role="dialog"`, `aria-modal`, `aria-labelledby`/description where useful, `useFocusTrap`, Escape close, focus restore, and named close/action icon buttons.
|
||||
- Non-modal panels use a labelled landmark such as `aside aria-label`, remain keyboard reachable, do not trap focus, and provide an explicit close button name.
|
||||
- Codify Notification Inbox, Create Workspace, Context Rail, Onboarding Tooltips, Upgrade Modal, and Trial Expired Modal in a focused overlay smoke.
|
||||
|
||||
Closure evidence:
|
||||
|
||||
- Desktop and 390 x 844 screenshots for the changed overlays.
|
||||
- DOM evidence that all high-frequency overlays have an accessible name and expected role/landmark.
|
||||
- Keyboard evidence: open, Tab/Shift+Tab, Escape or named close, focus return.
|
||||
- No visible unnamed icon-only buttons in the sampled overlay DOM.
|
||||
|
||||
### P1-20: Create Workspace is too dense and template-first on mobile
|
||||
|
||||
Implementation update 2026-07-08:
|
||||
|
||||
- Template deletion no longer uses native `confirm()`. It now opens an in-app confirmation dialog that names the template and says existing workspaces are not changed.
|
||||
- The mobile Create Workspace path now starts with required setup, keeps templates behind a "Start from template" disclosure, and keeps the Create action in a visible footer on the 390 x 844 rendered path.
|
||||
- This closes the sampled T7 template-management and mobile hierarchy portions of P1-20, but the final score still needs refreshed screenshots and broader route/state evidence.
|
||||
|
||||
Evidence:
|
||||
|
||||
- Historical desktop screenshot `06-create-workspace.png` showed a long, dense creation modal that began with template category filters, search, template cards, then finally the workspace name and storage controls.
|
||||
- Historical mobile screenshot `mobile-03-create-workspace.png` showed the first 390 x 844 viewport dominated by template controls, with storage continuation and Create/Cancel requiring internal scrolling.
|
||||
- Current focused evidence: component contract asserts the name field precedes optional templates and that template search is absent until "Start from template"; rendered `J-mobile` asserts required setup, visible Create action, progressive template search, and no horizontal overflow at 390 x 844.
|
||||
|
||||
Correction:
|
||||
|
||||
- Make the primary creation path obvious first: name, storage type/path, and Create/Cancel should be reachable without hunting.
|
||||
- Move template search/grid into a collapsed or secondary "Start from template" area on narrow screens, or use progressive disclosure with a clear selected-template summary.
|
||||
- Keep destructive template management branded and named; no native `confirm()`.
|
||||
|
||||
Closure evidence:
|
||||
|
||||
- 390 x 844 screenshot refresh where the required fields and primary action are visible or clearly sticky/reachable.
|
||||
- Keyboard-only create path proof.
|
||||
- Template management proof with in-app confirmation.
|
||||
|
||||
## What Already Looks Good
|
||||
|
||||
- Keyboard Shortcuts, Persona Switcher, Workspace Switcher, Upgrade Modal, Login Briefing, and Trial Expired Modal use the shared focus trap or Radix dialog patterns in source.
|
||||
- Spawn Agent has a coherent two-step flow, clear "Review & Launch" affordance, and useful no-key/no-model copy. It should remain in T10 only for detailed form semantics and model-selection evidence.
|
||||
- Upgrade Modal copy is aligned with Solo/Team and visually clear; the sampled close-label gap is now fixed.
|
||||
|
||||
## Recommendation
|
||||
|
||||
Keep Phase 1 unchanged. Treat the core shell overlay contract as Phase 2 partially fixed: Notification Inbox, Command Center, Create Workspace primary/subdialog contracts, Context Rail, Onboarding Tooltips, tier close labels, and the sampled Create Workspace mobile hierarchy now have focused coverage. Continue Phase 2 with route-specific overlay state evidence for less common states and the broader trust-critical native dialog queue.
|
||||
219
docs/audits/2026-07-08-source-inventory-consistency-audit.md
Normal file
219
docs/audits/2026-07-08-source-inventory-consistency-audit.md
Normal file
@@ -0,0 +1,219 @@
|
||||
# Source Inventory Consistency Audit - 2026-07-08
|
||||
|
||||
Status: analysis supplement. No product code was changed.
|
||||
|
||||
Purpose: compare the UX audit packet against the current source inventory so the final "complete UX" claim does not silently miss reachable routes, command destinations, apps, packages, or non-main user surfaces.
|
||||
|
||||
Deeper T19 follow-up: `docs/audits/2026-07-08-browser-companion-t19-analysis.md`.
|
||||
|
||||
## Commands
|
||||
|
||||
```powershell
|
||||
Get-Content apps/web/src/App.tsx
|
||||
Get-Content apps/web/src/lib/routes.ts
|
||||
Get-Content apps/web/src/lib/dock-tiers.ts
|
||||
Get-Content apps/web/src/lib/command-catalog.ts
|
||||
Get-ChildItem apps -Directory
|
||||
Get-ChildItem packages -Directory
|
||||
Get-ChildItem apps/www/app -Recurse -File
|
||||
Get-ChildItem apps/browser-ext -Recurse -File
|
||||
rg -n "browser-ext|WAGGLE_BROWSER_EXT|WAGGLE_DEV_ALLOW_ANY_EXTENSION|chrome-extension" packages apps tests docs --glob '!docs/audits/2026-07-08-*.md'
|
||||
node --check apps/browser-ext/popup.js
|
||||
node --check apps/browser-ext/background.js
|
||||
node --check apps/browser-ext/content.js
|
||||
npx tsc --noEmit --project packages/server/tsconfig.json
|
||||
Test-NetConnection 127.0.0.1 -Port 3333
|
||||
```
|
||||
|
||||
## In-App Route Registry Check
|
||||
|
||||
Source: `apps/web/src/App.tsx`.
|
||||
|
||||
Result: every production shell route is represented in `docs/audits/2026-07-08-ux-route-scenario-manifest.md`.
|
||||
|
||||
Routes verified:
|
||||
|
||||
```text
|
||||
/auth
|
||||
/ -> /home
|
||||
/home
|
||||
/workspaces
|
||||
/workspaces/:workspaceId/:tab?
|
||||
/memory/:mindScope?
|
||||
/artifacts
|
||||
/files
|
||||
/agents
|
||||
/automations
|
||||
/skills
|
||||
/room
|
||||
/waggle-dance
|
||||
/approvals
|
||||
/connectors
|
||||
/mcps
|
||||
/marketplace
|
||||
/launcher
|
||||
/team
|
||||
/settings
|
||||
/settings/vault
|
||||
/settings/profile
|
||||
/settings/mission-control
|
||||
/settings/timeline
|
||||
/settings/events
|
||||
/settings/usage
|
||||
/benchmarks
|
||||
/platform
|
||||
/payment-success
|
||||
/payment-cancelled
|
||||
*
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- `/motion-spec` is registered only under `import.meta.env.DEV`, outside the AppShell subtree. It is a development reference surface, not part of the production five-persona score, unless the user explicitly asks to judge developer-only visual tooling.
|
||||
- The index route is represented in the manifest as `/ -> /home`; the automated string check did not count the separate index element because it has no `path` string.
|
||||
|
||||
## Dock And Command Destination Check
|
||||
|
||||
Sources:
|
||||
|
||||
- `apps/web/src/lib/routes.ts`
|
||||
- `apps/web/src/lib/dock-tiers.ts`
|
||||
- `apps/web/src/lib/command-catalog.ts`
|
||||
|
||||
Result: dock/app route destinations are represented in the manifest. Two command-query destinations need explicit evidence ownership because they are real user-facing command outcomes:
|
||||
|
||||
```text
|
||||
apps/web/src/lib/command-catalog.ts:75 - /launcher?watch=1
|
||||
apps/web/src/lib/command-catalog.ts:78 - /settings?tab=billing
|
||||
```
|
||||
|
||||
Correction owner: T11 route evidence. These are not new top-level routes, but the final judge packet should prove the watch-mode Launcher state and billing-tab deep link, or explicitly defer them.
|
||||
|
||||
The same source check also reinforces T3:
|
||||
|
||||
```text
|
||||
apps/web/src/lib/command-catalog.ts:113 - heading "Pinned - Pro" is active Command Center copy
|
||||
apps/web/src/lib/dock-tiers.ts:81 - stale Pro comment in approvals copy context
|
||||
```
|
||||
|
||||
## Apps Inventory
|
||||
|
||||
Current `apps/` inventory:
|
||||
|
||||
```text
|
||||
apps/browser-ext - no package.json; Chrome MV3 extension
|
||||
apps/web - package.json; main installed cockpit UI
|
||||
apps/www - package.json; public launch funnel
|
||||
```
|
||||
|
||||
Audit consequence:
|
||||
|
||||
- `apps/web` is the main installed product surface.
|
||||
- `apps/www` is already T13.
|
||||
- `apps/browser-ext` was underrepresented in the July UX packet and should be a separate non-main gate because it is a real user-facing capture surface.
|
||||
|
||||
## Package Inventory
|
||||
|
||||
Current `packages/` inventory contains 28 package workspaces:
|
||||
|
||||
```text
|
||||
admin-web
|
||||
agent
|
||||
cli
|
||||
core
|
||||
hive-mind-cli
|
||||
hive-mind-core
|
||||
hive-mind-hooks-claude-code
|
||||
hive-mind-hooks-claude-desktop
|
||||
hive-mind-hooks-codex
|
||||
hive-mind-hooks-codex-desktop
|
||||
hive-mind-hooks-core
|
||||
hive-mind-hooks-cursor
|
||||
hive-mind-hooks-hermes
|
||||
hive-mind-hooks-openclaw
|
||||
hive-mind-mcp-server
|
||||
hive-mind-shim-core
|
||||
hive-mind-wiki-compiler
|
||||
launcher
|
||||
marketplace
|
||||
memory-mcp
|
||||
optimizer
|
||||
sdk
|
||||
server
|
||||
shared
|
||||
waggle-dance
|
||||
weaver
|
||||
wiki-compiler
|
||||
worker
|
||||
```
|
||||
|
||||
Audit consequence:
|
||||
|
||||
- The July packet's T15/T16/T17 grouping covers these packages by role, but future AGENTS/docs language that says "27 packages" is stale against current source.
|
||||
- `hive-mind-hooks-core` is a package workspace and should stay in T16/T17 verification command scope, even though it is not a launchable AI-tool adapter.
|
||||
|
||||
## Public Site Inventory
|
||||
|
||||
Current `apps/www/app` route/API files:
|
||||
|
||||
```text
|
||||
/
|
||||
/account
|
||||
/sign-in/[[...sign-in]]
|
||||
/sign-up/[[...sign-up]]
|
||||
/docs/methodology
|
||||
/design/personas
|
||||
/privacy
|
||||
/terms
|
||||
/cookies
|
||||
/eu-ai-act
|
||||
/api/stripe/checkout
|
||||
/api/webhooks/stripe
|
||||
```
|
||||
|
||||
Audit consequence:
|
||||
|
||||
- T13 already covers homepage, account/auth, methodology, legal pages, checkout, and webhook/deploy shape.
|
||||
- `/design/personas` should be treated as a public/supporting route if public-site route evidence is expanded. It does not affect Phase 1.
|
||||
|
||||
## Browser Companion Inventory
|
||||
|
||||
Current files:
|
||||
|
||||
```text
|
||||
apps/browser-ext/manifest.json
|
||||
apps/browser-ext/popup.html
|
||||
apps/browser-ext/popup.js
|
||||
apps/browser-ext/background.js
|
||||
apps/browser-ext/content.js
|
||||
apps/browser-ext/README.md
|
||||
```
|
||||
|
||||
Source contract:
|
||||
|
||||
- `apps/browser-ext/README.md` presents the surface as "Waggle Companion", a Chrome MV3 extension for saving pages/selections to workspace memory.
|
||||
- `packages/server/src/local/routes/browser-ext.ts` exposes `GET /api/browser-ext/session-token` and `GET /api/browser-ext/health`.
|
||||
- `packages/server/src/local/cors-config.ts` gates extension origins through `WAGGLE_BROWSER_EXT_IDS` or the dev-only `WAGGLE_DEV_ALLOW_ANY_EXTENSION=1`.
|
||||
- `apps/browser-ext/background.js` calls `/api/browser-ext/session-token`, `/api/browser-ext/health`, and `/api/memory/frames`.
|
||||
|
||||
Current evidence gap:
|
||||
|
||||
- The T19 follow-up loaded the unpacked extension in Chromium and captured a disconnected popup screenshot at `output/playwright/browser-companion-disconnected-state.png`.
|
||||
- JS syntax, manifest parse, and `packages/server` typecheck pass.
|
||||
- The secure-default live smoke proves content-script extraction from a normal page, MV3 service-worker no-Origin header handling, token bootstrap during save, background save through `chrome.runtime.sendMessage`, and `/api/memory/frames` imported-frame confirmation. The popup keyboard/click smoke proves Tab order across popup actions, visible Save selection focus, Enter-to-save selection, Save page UI click, frame creation, `/api/memory/search` `source: import` provenance for both captures, and restricted-page disabled-state/recovery behavior.
|
||||
- Remaining evidence gaps: native toolbar-bubble proof, native context-menu click flow, CORS-denied screenshot with a real extension origin, signed release-package proof if scored, and any future scored recall result shape. Stable packaged-ID sidecar pairing is now covered by `output/playwright/browser-companion-toolbar-3333/packaged-id-pairing-summary.json`, and existing chat `auto_recall`/catch-up imported provenance is covered by `packages/agent/tests/orchestrator-recall-hardening.test.ts`.
|
||||
|
||||
Recommended ticket: T19, Browser Companion Extension UX Gate.
|
||||
|
||||
Suggested acceptance:
|
||||
|
||||
- Load unpacked extension in Chromium with a fresh sidecar and extension origin allowlist.
|
||||
- Verify connected and disconnected popup states.
|
||||
- Verify save selected text and save whole page produce visible extension feedback and a memory frame in Waggle.
|
||||
- Verify CORS-denied state tells the user how to start or configure Waggle.
|
||||
- Verify popup accessibility basics beyond the proven keyboard/focus path if screen-reader announcement behavior enters scoring.
|
||||
- Either add automated smoke evidence or explicitly defer the extension from the five-persona score.
|
||||
|
||||
## Phase Impact
|
||||
|
||||
This supplement does not change Phase 1. It adds T19 as a final-product gate after the in-app P0 blockers are cleared.
|
||||
140
docs/audits/2026-07-08-state-failure-t12-analysis.md
Normal file
140
docs/audits/2026-07-08-state-failure-t12-analysis.md
Normal file
@@ -0,0 +1,140 @@
|
||||
# Focused T12 State and Failure Bundle Analysis
|
||||
|
||||
Status: implementation follow-up in progress; this file now records guarded scorecard contracts and current browser evidence.
|
||||
|
||||
Purpose: the state/failure matrix defines the dimensions; this supplement records the current command/source evidence and converts T12 into the concrete bundle contract each judge persona must satisfy before a 9/10 score is credible.
|
||||
|
||||
Guideline source refreshed during this pass: Vercel Web Interface Guidelines, `https://raw.githubusercontent.com/vercel-labs/web-interface-guidelines/main/command.md`. The rules most relevant to T12 are destructive actions using confirmation/undo instead of immediate action, URL-state clarity for stateful UI, inline error/focus behavior, visible focus, long-content handling, and large-list performance.
|
||||
|
||||
## Current Command Evidence
|
||||
|
||||
| Evidence | Result | What it proves | What it does not prove |
|
||||
|---|---:|---|---|
|
||||
| `npm run test -w apps/web -- src/test/p1b-authgate-surfaces.test.tsx src/test/p4-onboarding-status.test.ts src/test/wave-t-onboarding-boot-gate.test.ts src/test/pr7a-billing.test.tsx src/test/phase5b-backup.test.tsx src/test/p7-b4-files-error.test.tsx src/test/phase4b-connector-hub.test.tsx src/test/phase4b-mcp-hub.test.tsx src/test/p7-b1-approvals-error.test.tsx --run` | Pass, 9 files / 79 tests | Auth-gate surfaces, onboarding status/boot gates, billing/payment states, backup status classification, files errors, connector hub, MCP hub, and approvals error cases have healthy focused component/unit coverage. | It is not a rendered route, viewport, console, or five-persona state-bundle run. |
|
||||
| `tests/e2e/failure-injection/network-drop.spec.ts` source inspection | Existing coverage found | Chat SSE hard drop, truncated stream after one token, and retry after dropped stream are represented as browser failure-injection scenarios. | The spec was not rerun in this focused T12 pass, and it covers chat stream failure rather than sidecar/offline/Stripe/marketplace failure broadly. |
|
||||
| Fresh built-app route smoke, port 3407 | Mixed | `/payment-cancelled` redirects to `/settings?tab=billing`, which is useful Team Admin payment-recovery state evidence; `/benchmarks` and `/platform` also render meaningful shell content. | The smoke is ad hoc, not codified; all three routes emit the existing T1 CSP/Clerk console errors, and payment recovery copy still needs judge inspection. |
|
||||
| Fresh thin-route smoke, port 3411 | Mixed | Engineer, Researcher, Team Admin, and Mobile Executive state bundles now have ad hoc rendered evidence for Launcher/watch, WaggleDance, Artifacts, Profile, Timeline, Payment Success fallback, Automations, MCP Hub, Usage & Cost, and Files. | The smoke is ad hoc, not codified; all routes emit T1 CSP/Clerk errors, Launcher can log a detect-in-flight adapter error, Usage & Cost logs a 403 resource error, and primary workflow states are still untested. |
|
||||
| `J-route-coverage` Playwright tests, port 34139 | Pass, 2 tests | Codifies route-level state inputs for Engineer (`/launcher`, `/launcher?watch=1`, `/mcps`, `/files`), Researcher (`/artifacts`, `/settings/timeline`), Team Admin (`/payment-success`, `/payment-cancelled`), and Mobile Executive/Profile (`/settings/profile`, `/settings/usage`). | This is rendered shell/recovery-copy proof only; it does not exercise destructive actions, real checkout provider return, Launcher hook lifecycle, MCP install/verify, file upload, or failure-state bundles. |
|
||||
| Fresh Mobile Executive smoke, port 3419 | Mixed | A 390 x 844 rendered lane now has screenshots for Home, Settings general/models/billing/profile, Memory, workspace chat, Command Center, and Workspace Switcher. It proves the routes render and that Workspace Switcher can close with Escape on mobile. | The smoke is ad hoc, not codified; every route still emits T1 CSP/Clerk errors; Settings has visible clipped/squeezed controls despite clean document scroll width; Memory/chat tab strips overflow; Command Center remained visible after Escape and logged a missing dialog description warning. |
|
||||
| `npx playwright test tests/e2e/five-persona-state-bundles.spec.ts --project=chromium --reporter=list`, port 34245 | Pass, 5 tests | Codifies rendered state bundles plus one failure probe for all five judge personas and writes `state-bundle.md`/`.json` plus screenshots under `output/playwright/five-persona-state-bundles/`. Current capture has 0 critical console errors, 0 page errors, 0 critical network failures, 0 visible overflow, and overlay close proof where required. | It is still a no-LLM/accountless smoke; it does not prove authenticated Teams, real checkout, packaged desktop, hook lifecycle, file upload, every destructive/failure recovery path, or final judge scoring. |
|
||||
| `npx playwright test tests/e2e/five-persona-state-bundles.spec.ts --project=chromium --grep "mobile-executive" --reporter=list`, port 34244 | Pass, 1 test | Rebuilt after the workspace tab and chat agent-strip mobile fixes; regenerated Mobile Executive evidence now reports 0 visible horizontal overflow items on Home, Settings, Memory, and mobile chat backend-offline failure at 390 x 844. | It only rechecks the Mobile Executive bundle; broader authenticated/mobile Team paths still need targeted evidence. |
|
||||
| `npx playwright test tests/e2e/five-persona-state-bundles.spec.ts --project=chromium --reporter=list --retries=0`, port 34270 with matched `WAGGLE_E2E_BASE_URL` | Pass, 5 tests | Regenerates all five persona bundles with route, failure, and overlay evidence. Mobile Executive captures Notification Inbox plus Command Center screenshots; Engineer captures Marketplace unavailable; Team Admin captures backup restore failure. The run enforces 0 critical console/page/network failures and 0 visible overflow across routes, failures, and overlays. | It is still an accountless/no-LLM bundle; authenticated Teams, real checkout, packaged desktop, hook lifecycle, file upload, Stripe/model/health failure states, and final judge scoring remain outside this proof. |
|
||||
| `npx playwright test tests/e2e/five-persona-state-bundles.spec.ts --project=chromium --grep "engineer-power-user\|team-admin-security-reviewer" --reporter=list --retries=0`, port 34269 with matched `WAGGLE_E2E_BASE_URL` | Pass, 2 tests | Regenerates Engineer and Team Admin bundles with `marketplace-unavailable` and `backup-restore-failure` probes. Both bundles report 0 visible overflow, 0 critical console errors, 0 page errors, and 0 critical network failures, with screenshots under `output/playwright/five-persona-state-bundles/`. | It is still an accountless/no-LLM bundle; Stripe unavailable/cancelled, local model unavailable, health degradation, authenticated Teams, packaged desktop, hook lifecycle, file upload, and final judge scoring remain outside this proof. |
|
||||
| `npx playwright test tests/e2e/five-persona-state-bundles.spec.ts --project=chromium --grep "engineer-power-user\|team-admin-security-reviewer\|mobile-executive" --reporter=list --retries=0`, port 34272 with matched `WAGGLE_E2E_BASE_URL` | Pass, 3 tests | Focused verification for the newly expanded T12-C probes: Engineer now captures Cockpit `mission-control-health-degraded`, Team Admin captures `billing-checkout-unavailable`, and Mobile Executive captures `local-model-runtime-unavailable`. The run reports 0 critical console/page/network failures and 0 visible overflow for the touched bundles. | It is still an accountless/no-LLM bundle; authenticated Teams, real checkout provider success/cancel, packaged desktop, hook lifecycle, file upload, and final judge scoring remain outside this proof. |
|
||||
| `npx playwright test tests/e2e/five-persona-state-bundles.spec.ts --project=chromium --reporter=list --retries=0`, port 34273 with matched `WAGGLE_E2E_BASE_URL` | Pass, 5 tests | Regenerates all five persona bundles after the expanded failure probes. The current bundle includes chat backend offline, Memory API unavailable, Launcher sidecar offline, Cockpit health degraded, Marketplace unavailable, billing checkout unavailable, backup create failure, backup restore failure, local model runtime unavailable, mobile chat backend offline, Notification Inbox, Command Center, and Workspace Switcher evidence, with 0 critical console/page/network failures and 0 visible overflow. | It is still an accountless/no-LLM bundle; authenticated Teams, real checkout provider success/cancel, packaged desktop, hook lifecycle, file upload, scale/performance states, and final judge scoring remain outside this proof. |
|
||||
| `npx playwright test tests/e2e/five-persona-state-bundles.spec.ts --project=chromium --grep "engineer-power-user" --reporter=list --retries=0`, port 34274 with matched `WAGGLE_E2E_BASE_URL` | Pass, 1 test | Focused verification for the Files upload failure lane. The Engineer bundle now captures `files-upload-failure`: a failed `/api/workspaces/:id/files/upload` shows branded `Upload failed` recovery copy and does not render the failed filename as a successful file row. The bundle reports 0 critical console/page/network failures and 0 visible overflow. | It is still an accountless/no-LLM focused slice; authenticated Teams, real checkout provider success/cancel, packaged desktop, hook lifecycle, scale/performance states, and final judge scoring remain outside this proof. |
|
||||
| `npx playwright test tests/e2e/five-persona-state-bundles.spec.ts --project=chromium --reporter=list --retries=0`, port 34275 with matched `WAGGLE_E2E_BASE_URL` | Pass, 5 tests | Regenerates the full five-persona bundle after the Files upload-failure fix. The current full bundle includes all previous sampled probes plus `files-upload-failure`, and reports 0 critical console/page/network failures and 0 visible overflow. | It is still an accountless/no-LLM bundle; authenticated Teams, real checkout provider success/cancel, packaged desktop, hook lifecycle, scale/performance states, and final judge scoring remain outside this proof. |
|
||||
| `npm run test -- packages/server/tests/local/files-upload-multipart.test.ts --reporter=dot` | Pass, 1 test | Regresses the real browser multipart boundary that previously returned Fastify `415 FST_ERR_CTP_INVALID_MEDIA_TYPE`; upload now returns `201`, the file entry shape is `/successful-upload.md`, and `/files/list` returns the uploaded file. | This is server-route proof only; it does not prove the rendered Files UI without the Playwright bundle. |
|
||||
| `npx playwright test tests/e2e/five-persona-state-bundles.spec.ts --project=chromium --grep "engineer-power-user" --reporter=list --retries=0`, port 34278 with matched `WAGGLE_E2E_BASE_URL` | Pass, 1 test | Focused verification for the successful Files upload workflow. The Engineer bundle now captures `files-upload-success`: the upload response is `ok`, returns `{ name: 'successful-upload.md', path: '/successful-upload.md' }`, and the Files UI renders the uploaded filename with 0 visible overflow. | It is still an accountless/no-LLM focused slice; authenticated Teams, real checkout provider success/cancel, packaged desktop, hook lifecycle, scale/performance states, and final judge scoring remain outside this proof. |
|
||||
| `npx playwright test tests/e2e/five-persona-state-bundles.spec.ts --project=chromium --reporter=list --retries=0`, port 34279 with matched `WAGGLE_E2E_BASE_URL` | Pass, 5 tests | Regenerates the full five-persona bundle after the multipart upload fix. The current Engineer bundle includes `files-upload-failure` and `files-upload-success`; the full run reports 0 critical console/page/network failures and 0 visible overflow. | It is still an accountless/no-LLM bundle; authenticated Teams, real checkout provider success/cancel, packaged desktop, hook lifecycle, scale/performance states, and final judge scoring remain outside this proof. |
|
||||
| `npx playwright test tests/e2e/five-persona-state-bundles.spec.ts --project=chromium --grep "engineer-power-user" --reporter=list --retries=0`, port 34280 with matched `WAGGLE_E2E_BASE_URL` | Pass, 1 test | Focused verification for a sampled Files large-list state. The Engineer bundle now captures `files-large-list`: `/api/workspaces/:id/files/list` is mocked with 240 files, the Files UI renders `bulk-file-000.md`, shows `240 items`, and reports 0 visible overflow. | It is still one large-list slice; broader memory/event/marketplace/agent/file scale and slow-data states remain outside this proof. |
|
||||
| `npx playwright test tests/e2e/five-persona-state-bundles.spec.ts --project=chromium --reporter=list --retries=0`, port 34281 with matched `WAGGLE_E2E_BASE_URL` | Pass, 5 tests | Regenerates the full five-persona bundle after the Files large-list probe. The current Engineer bundle includes `files-upload-failure`, `files-upload-success`, and `files-large-list`; the full run reports 0 critical console/page/network failures and 0 visible overflow. | It is still an accountless/no-LLM bundle; authenticated Teams, real checkout provider success/cancel, packaged desktop, hook lifecycle, broader scale/performance states, and final judge scoring remain outside this proof. |
|
||||
| `npx playwright test tests/e2e/five-persona-state-bundles.spec.ts --project=chromium --grep "engineer-power-user" --reporter=list --retries=0`, port 34282 with matched `WAGGLE_E2E_BASE_URL` | Pass, 1 test | Focused verification for a sampled Marketplace large-catalog state. The Engineer bundle now captures `marketplace-large-catalog`: `/api/marketplace`, `/api/connectors`, and `/api/mcps` are mocked as a 240-entry catalog, and the Marketplace UI renders `Bulk Skill 000` plus grouped counts for `Skills 120`, `Connectors 60`, and `MCPs 60` with 0 visible overflow. | It is still one marketplace scale slice; broader memory/event/agent scale, slow-data states, and deeper performance evidence remain outside this proof. |
|
||||
| `npx playwright test tests/e2e/five-persona-state-bundles.spec.ts --project=chromium --reporter=list --retries=0`, port 34283 with matched `WAGGLE_E2E_BASE_URL` | Pass, 5 tests | Regenerates the full five-persona bundle after the Marketplace large-catalog probe. The current Engineer bundle reports 7 failure/scale probes, including `marketplace-unavailable`, `marketplace-large-catalog`, `files-upload-failure`, `files-upload-success`, and `files-large-list`; the full run reports 0 critical console/page/network failures and 0 visible overflow. | It is still an accountless/no-LLM bundle; authenticated Teams, real checkout provider success/cancel, packaged desktop, hook lifecycle, remaining broader scale/performance states, and final judge scoring remain outside this proof. |
|
||||
| `npx playwright test tests/e2e/five-persona-state-bundles.spec.ts --project=chromium --grep "researcher" --reporter=list --retries=0`, port 34284 with matched `WAGGLE_E2E_BASE_URL` | Pass, 1 test | Focused verification for a sampled Memory large-list state. The Researcher bundle now captures `memory-large-list`: bare `/api/memory?...limit=200` is mocked with 200 unique memories, and the Memory Center list renders `200 memories` plus `Bulk Memory 000` with 0 visible overflow and 0 critical console/page/network failures. | It is still one memory scale slice; broader event/agent scale, slow-data states, and deeper performance evidence remain outside this proof. |
|
||||
| `npx playwright test tests/e2e/five-persona-state-bundles.spec.ts --project=chromium --reporter=list --retries=0`, port 34285 with matched `WAGGLE_E2E_BASE_URL` | Pass, 5 tests | Regenerates the full five-persona bundle after the Memory large-list probe. The current Researcher bundle reports 2 failure/scale probes, including `memory-list-unavailable` and `memory-large-list`; the full run reports 0 critical console/page/network failures and 0 visible overflow. | It is still an accountless/no-LLM bundle; authenticated Teams, real checkout provider success/cancel, packaged desktop, hook lifecycle, remaining broader scale/performance states, and final judge scoring remain outside this proof. |
|
||||
| `npx playwright test tests/e2e/five-persona-state-bundles.spec.ts --project=chromium --grep "engineer-power-user" --reporter=list --retries=0`, port 34286 with matched `WAGGLE_E2E_BASE_URL` | Pass, 1 test | Focused verification for a sampled Agents large-list state. The Engineer bundle now captures `agents-large-list`: `/api/agents` is mocked with 180 live agents, and the Agents UI renders `180 agents` plus `Bulk Agent 000` with 0 visible overflow and 0 critical console/page/network failures. | It is still one agent scale slice; broader event scale, slow-data states, and deeper performance evidence remain outside this proof. |
|
||||
| `npx playwright test tests/e2e/five-persona-state-bundles.spec.ts --project=chromium --reporter=list --retries=0`, port 34287 with matched `WAGGLE_E2E_BASE_URL` | Pass, 5 tests | Regenerates the full five-persona bundle after the Agents large-list probe. The current Engineer bundle reports 8 failure/scale probes, including `agents-large-list`, `marketplace-unavailable`, `marketplace-large-catalog`, `files-upload-failure`, `files-upload-success`, and `files-large-list`; the full run reports 0 critical console/page/network failures and 0 visible overflow. | It is still an accountless/no-LLM bundle; authenticated Teams, real checkout provider success/cancel, packaged desktop, hook lifecycle, remaining event scale/slow-data states, and final judge scoring remain outside this proof. |
|
||||
| `npx playwright test tests/e2e/five-persona-state-bundles.spec.ts --project=chromium --grep "researcher" --reporter=list --retries=0`, port 34289 with matched `WAGGLE_E2E_BASE_URL` | Pass, 1 test | Focused verification for a sampled Timeline/Event large-list state. The Researcher bundle now captures `timeline-large-events`: `/api/events?...limit=500` is mocked with 360 timeline events, and the Timeline UI renders `360 events` plus `Used bulk_tool_000` with 0 visible overflow and 0 critical console/page/network failures. | It is still one event scale slice; slow-data states and deeper performance evidence remain outside this proof. |
|
||||
| `npx playwright test tests/e2e/five-persona-state-bundles.spec.ts --project=chromium --reporter=list --retries=0`, port 34290 with matched `WAGGLE_E2E_BASE_URL` | Pass, 5 tests | Regenerates the full five-persona bundle after the Timeline/Event large-list probe. The current Researcher bundle reports `memory-list-unavailable`, `memory-large-list`, and `timeline-large-events`; the full run reports 0 critical console/page/network failures and 0 visible overflow. | It is still an accountless/no-LLM bundle; authenticated Teams, real checkout provider success/cancel, packaged desktop, hook lifecycle, slow-data states, deeper performance evidence, and final judge scoring remain outside this proof. |
|
||||
| `npx playwright test tests/e2e/five-persona-state-bundles.spec.ts --project=chromium --grep "researcher" --reporter=list --retries=0`, port 34291 with matched `WAGGLE_E2E_BASE_URL` | Pass, 1 test | Focused verification for a sampled slow-data state. The Researcher bundle now captures `memory-slow-list`: `/api/memory?...limit=200` is deliberately delayed by 2 seconds, the Memory UI exposes an aria-busy `Loading memories` status while waiting, then renders `40 memories` plus `Bulk Memory 000` with 0 visible overflow and 0 critical console/page/network failures. | It is still one slow-data slice; deeper app-payload/performance evidence remains outside this proof. |
|
||||
| `npx playwright test tests/e2e/five-persona-state-bundles.spec.ts --project=chromium --reporter=list --retries=0`, port 34292 with matched `WAGGLE_E2E_BASE_URL` | Pass, 5 tests | Regenerates the full five-persona bundle after the slow Memory probe. The current Researcher bundle reports `memory-list-unavailable`, `memory-slow-list`, `memory-large-list`, and `timeline-large-events`; the full run reports 0 critical console/page/network failures and 0 visible overflow. | It is still an accountless/no-LLM bundle; authenticated Teams, real checkout provider success/cancel, packaged desktop, hook lifecycle, broader slow-data/performance states, and final judge scoring remain outside this proof. |
|
||||
| `node node_modules/playwright/cli.js test tests/e2e/five-persona-state-bundles.spec.ts --project=chromium --grep researcher --reporter=list` | Pass, 1 test | Focused verification for a sampled Researcher export-failure state. The Researcher bundle now captures `wiki-export-obsidian-failure`: `/api/wiki/pages` returns a compiled page, `/api/wiki/export/obsidian` returns `500`, and the Wiki UI renders `Export Failed` plus `Disk permission denied for C:/Research Vault`, with 0 visible overflow and 0 critical console/page/network failures. | It is still one export-failure slice; Notion failure, export permission variants, authenticated memory governance, and final judge scoring remain outside this proof. |
|
||||
| `node node_modules/playwright/cli.js test tests/e2e/five-persona-state-bundles.spec.ts --project=chromium --reporter=list` | Pass, 5 tests | Regenerates the full five-persona bundle after adding the Researcher export-failure probe. The Researcher bundle now reports 5 failure/workflow/scale/slow-data probes, including `wiki-export-obsidian-failure`; the full run reports 0 critical console/page/network failures and 0 visible overflow. | It is still an accountless/no-LLM bundle; authenticated Teams, real checkout provider success/cancel, packaged desktop, hook lifecycle, broader performance states, and final judge scoring remain outside this proof. |
|
||||
| `npx playwright test tests/e2e/five-persona-state-bundles.spec.ts --project=chromium --grep "engineer-power-user" --reporter=list --retries=0`, port 34294 with matched `WAGGLE_E2E_BASE_URL` | Pass, 1 test | Focused verification for a sampled Agents slow-data state. The Engineer bundle now captures `agents-slow-list`: `/api/agents` is deliberately delayed by 2 seconds, the Agents UI exposes an aria-busy `Loading agents` status while waiting, then renders `40 agents` plus `Bulk Agent 000` with 0 visible overflow and 0 critical console/page/network failures. | It is still one agent slow-data slice; authenticated Teams, real checkout provider success/cancel, packaged desktop, hook lifecycle, broader performance states, and final judge scoring remain outside this proof. |
|
||||
| `npx playwright test tests/e2e/five-persona-state-bundles.spec.ts --project=chromium --reporter=list --retries=0`, port 34295 with matched `WAGGLE_E2E_BASE_URL` | Pass, 5 tests | Regenerates the full five-persona bundle after the Agents slow-list probe. The current Engineer bundle reports 9 failure/workflow/scale/slow-data probes, including `agents-slow-list`, `agents-large-list`, `marketplace-unavailable`, `marketplace-large-catalog`, `files-upload-failure`, `files-upload-success`, and `files-large-list`; the full run reports 0 critical console/page/network failures and 0 visible overflow. | It is still an accountless/no-LLM bundle; authenticated Teams, real checkout provider success/cancel, packaged desktop, hook lifecycle, broader performance states, and final judge scoring remain outside this proof. |
|
||||
| `npx playwright test tests/e2e/five-persona-state-bundles.spec.ts --project=chromium --grep "team-admin-security-reviewer" --reporter=list --retries=0`, port 34297 with matched `WAGGLE_E2E_BASE_URL` | Pass, 1 test | Focused verification for Team Admin checkout return states after the cancelled-checkout banner fix. The bundle now captures `billing-checkout-success-return` with mocked `/api/stripe/sync` returning `TEAMS`, and `billing-checkout-cancel-return` redirecting to Billing with `Checkout was cancelled` plus `No charge was made` copy. | It is local/accountless rendered return proof; real deployed Stripe/Clerk checkout success/cancel, authenticated Teams, packaged desktop, hook lifecycle, and final judge scoring remain outside this proof. |
|
||||
| `npx playwright test tests/e2e/five-persona-state-bundles.spec.ts --project=chromium --reporter=list --retries=0`, port 34298 with matched `WAGGLE_E2E_BASE_URL` | Pass, 5 tests | Regenerates the full five-persona bundle after the Team Admin checkout return probes. The Team Admin bundle reports 5 failure/workflow probes, including `billing-checkout-unavailable`, `billing-checkout-success-return`, `billing-checkout-cancel-return`, `backup-create-failure`, and `backup-restore-failure`; the full run reports 0 critical console/page/network failures and 0 visible overflow. | It is still an accountless/no-LLM bundle; real authenticated Teams server, real deployed/provider checkout success/cancel, packaged desktop, hook lifecycle, broader performance states, and final judge scoring remain outside this proof. |
|
||||
| `npx playwright test tests/e2e/user-journeys.spec.ts --project=chromium --grep "J-route-coverage: thin utility routes render or redirect clearly" --reporter=list --retries=0`, port 34299 with matched `WAGGLE_E2E_BASE_URL` | Pass, 1 test | Confirms the broader thin-route route-smoke still accepts the `/payment-cancelled` redirect into Billing after the new `checkout=cancelled` marker. | Route-smoke only; detailed success/cancel copy is owned by the five-persona Team Admin bundle above. |
|
||||
| `npx playwright test tests/e2e/five-persona-state-bundles.spec.ts --project=chromium --grep "team-admin-security-reviewer" --reporter=list --retries=0`, port 34300 with matched `WAGGLE_E2E_BASE_URL` | Pass, 1 test | Focused verification for local mocked Teams-state evidence. The Team Admin bundle now captures `billing-team-active-state` with `/api/tier` mocked to `TEAMS` and visible `Waggle Team`, `$49/mo per seat`, and `Manage subscription` copy, plus `team-settings-unlocked-state` with visible `Team Server URL`, `Auth Token`, and trust warning copy. | It is local rendered Teams-tier state proof, not real authenticated Team server membership, deployed checkout, packaged desktop, hook lifecycle, or final judge scoring. |
|
||||
| `npx playwright test tests/e2e/five-persona-state-bundles.spec.ts --project=chromium --reporter=list --retries=0`, port 34301 with matched `WAGGLE_E2E_BASE_URL` | Pass, 5 tests | Regenerates the full five-persona bundle after adding the Team active billing and Team settings unlocked probes. The Team Admin bundle reports 7 failure/workflow probes, including `billing-team-active-state`, `team-settings-unlocked-state`, checkout unavailable/success/cancel, backup creation failure, and backup restore failure; the full run reports 0 critical console/page/network failures and 0 visible overflow. | It is still an accountless/no-LLM bundle with mocked Teams tier; real authenticated Team server, real deployed/provider checkout success/cancel, packaged desktop, hook lifecycle, broader performance states, and final judge scoring remain outside this proof. |
|
||||
| `node node_modules/playwright/cli.js test tests/e2e/five-persona-state-bundles.spec.ts --project=chromium --grep team-admin-security-reviewer --reporter=list` | Pass, 1 test | Focused verification for sampled Team Admin approval-grant revocation. The bundle now captures `approvals-revoke-all-grants`: `/approvals` is mocked with one saved grant, the in-app approval modal confirms the destructive revoke-all action, and the final route renders `No saved grants` plus revoked-state toast copy with 0 critical console/page/network failures and 0 visible overflow. | It is local accountless rendered proof for the sampled revoke-all path; broader authenticated approval audit history, focus/keyboard behavior, and less common approval policy states remain outside this proof. |
|
||||
| `node node_modules/playwright/cli.js test tests/e2e/five-persona-state-bundles.spec.ts --project=chromium --reporter=list` | Pass, 5 tests | Regenerates the full five-persona bundle after adding the Team Admin approval-grant revoke-all probe. The Team Admin bundle now reports 8 failure/workflow probes, including `approvals-revoke-all-grants`, Teams state, checkout unavailable/success/cancel, backup creation failure, and backup restore failure; the full run reports 0 critical console/page/network failures and 0 visible overflow. | It is still an accountless/no-LLM bundle with mocked Teams tier; real authenticated Team server, real deployed/provider checkout success/cancel, packaged desktop, hook lifecycle, broader performance states, and final judge scoring remain outside this proof. |
|
||||
| Native dialog fixed-string scan for `confirm(`, `alert(`, and `prompt(` | Current high-confidence production scan clean | T7/T12 no longer has known production browser-native dialog calls after the focused fixes. | Remaining broad scan hits are markdown sanitizer test payloads only; T7/T12 still needs persona-state, failure-state, and accessibility evidence. |
|
||||
|
||||
Observed warning noise in the focused state slice:
|
||||
|
||||
- Node `[DEP0040] punycode` deprecation warnings.
|
||||
- Expected but noisy `[useWorkspaces] fetch failed: Unauthorized` logs in auth-gate tests.
|
||||
- Repeated React `act(...)` warnings from provider/test setup.
|
||||
- React Router v7 future-flag warnings.
|
||||
|
||||
Interpretation: the tested state slices are encouraging, but the output is still too noisy for a final judge lane. T10 should reduce or isolate expected warning noise so new state failures stand out.
|
||||
|
||||
## Native Dialog Evidence
|
||||
|
||||
High-confidence browser-native dialog hits that affect T7/T12:
|
||||
|
||||
| Surface | Evidence | Judge risk |
|
||||
|---|---|---|
|
||||
| Workspace/template management | `apps/web/src/components/os/overlays/CreateWorkspaceDialog.tsx:877`, `:925` | Solo founder or admin sees a browser confirm inside a branded creation/manage flow. |
|
||||
| Approvals | Historical: `apps/web/src/components/os/apps/ApprovalsApp.tsx:158`. Post-fix: shared in-app `ApprovalModal` plus focused component/rendered evidence. | Fixed for the sampled Team-admin revoke-all path; broader approval states still need persona-bundle evidence. |
|
||||
| Backup app | Historical: `apps/web/src/components/os/apps/BackupApp.tsx:79`. Post-fix: shared in-app `ApprovalModal` plus focused component evidence. | Fixed for the standalone restore-selection path; broader backup/persona-bundle evidence remains. |
|
||||
| Automation Center | Historical: `apps/web/src/components/os/apps/AutomationCenterApp.tsx:184`. Post-fix: shared in-app `ApprovalModal` plus focused component evidence. | Fixed for sampled automation delete path; broader automation persona-bundle evidence remains. |
|
||||
| Artifact Center | Historical: `apps/web/src/components/os/apps/ArtifactCenterApp.tsx:165`. Post-fix: shared in-app `ApprovalModal` plus focused component/rendered evidence. | Fixed for the sampled Researcher/engineer permanent-delete path; archive and broader artifact states still need persona-bundle evidence. |
|
||||
| Compliance templates | Historical: `apps/web/src/components/os/apps/cockpit/ComplianceTemplateModal.tsx:155`. Post-fix: shared in-app `ApprovalModal` plus focused component evidence. | Fixed for sampled compliance-template delete path; broader compliance persona-bundle evidence remains. |
|
||||
| Memory Center | Historical: `apps/web/src/components/os/apps/memory/MemoryCenterTab.tsx:301`, `:311`, `:356`. Post-fix: shared in-app `ApprovalModal` plus focused component/rendered evidence for delete, erase, and allow re-import. | Fixed for sampled Researcher permanent-delete, GDPR erasure, and suppression-lift paths; broader persona-bundle evidence remains. |
|
||||
| Settings telemetry/backup | Historical: `apps/web/src/components/os/apps/SettingsApp.tsx:399`, `:893`, `:901`, `:911`, `:921`, `:922`, `:925`. Post-fix: shared in-app `ApprovalModal` plus inline backup status evidence. | Fixed for sampled Team-admin telemetry clear, backup failure, and restore-success paths; broader backup/persona-bundle evidence remains. |
|
||||
| Wiki export | Historical: `apps/web/src/components/os/apps/memory/WikiTab.tsx:188`, `:214`. Post-fix: in-app Obsidian/Notion export form dialogs plus focused component/rendered evidence; the Researcher bundle now includes an Obsidian export-failure state. | Fixed for sampled Researcher export destination paths and one rendered Obsidian export-failure path; broader Notion/permission variants remain. |
|
||||
|
||||
Earlier false positives in `MemoryTrustManage.tsx` were local memory-review callbacks and are no longer present in the production scan.
|
||||
|
||||
## State Bundle Contract
|
||||
|
||||
Each judge run must write a `state-bundle.md` before scoring. A persona cannot score 9/10 from default-state screenshots alone.
|
||||
|
||||
| Persona | Required bundle fields | Minimum state evidence |
|
||||
|---|---|---|
|
||||
| Solo founder | Accountless local, Solo billing, simple disclosure, no-model recovery plus first working/skipped chat lane, fresh/no-workspace then one workspace, accountless Clerk/CSP error lane, desktop plus mobile Home spot-check | `/auth`, onboarding/skip, `/home`, workspace create/list, first chat, return Home, console summary. |
|
||||
| Researcher | Declared auth mode, Solo or declared Team, power disclosure, memory-focused model state, populated memory plus no-results, missing-source/export/delete failure state, desktop plus optional mobile Memory | Memory overview, search hit, no-results, provenance/trust detail, wiki/timeline, branded confirmation/result, chat explanation, T19 decision if browser capture is included. |
|
||||
| Engineer / power user | Accountless local, Solo with Teams-only deferrals, power/admin disclosure, local/no-LLM plus one working-provider lane if chat is scored, workspace plus detected/undetected tools and MCP catalog, marketplace/tool unavailable states, desktop keyboard path | Command Center, `Ctrl+Shift+N`, Launcher, MCP Hub, Files, Events, console summary, T15/T16/T17/T18 decisions if utilities/hooks/developer lanes are in scope. |
|
||||
| Team admin / security reviewer | Authenticated or declared accountless limitation, Teams plus Solo gating comparison and legacy Pro-to-Solo state, professional/admin disclosure, vault/approval/backup/team data, backup or restore failure copy, desktop plus mobile Settings/Profile spot-check | Billing, active mocked Team billing state, unlocked Team settings state, Vault secret hidden, Approvals, backup/restore, Team governance, payment success, payment cancelled, branded confirmation/result, T13/T14/T15 decisions. |
|
||||
| Mobile executive | Accountless local, Solo unless Team view sampled, simple disclosure with power comparison only if needed, no-model or verified-model banner fitting mobile, one workspace plus some memory, overlay close/error state, 390 x 844 primary viewport | Mobile Home, Settings general/billing/models/profile, Memory, chat, overlay open/closed, horizontal overflow check, focus/touch notes. |
|
||||
|
||||
## Current Strengths
|
||||
|
||||
- The focused 79-test state slice passed across auth gate, onboarding, billing, backup, files, connector, MCP, and approvals error cases.
|
||||
- Current follow-up: `p7-b1-approvals-error.test.tsx` now covers Approvals revoke-all without native `confirm()`, rendered `J3d` passes on port `34159`, and the expanded user-journey suite passed 20/20 on port `34160` before the Artifact `J3e` addition.
|
||||
- Current follow-up: `artifact-center-trust.test.tsx` now covers Artifact permanent delete without native `confirm()`, rendered `J3e` passes on port `34161`, and the expanded user-journey suite passes 21/21 on port `34162`.
|
||||
- Current follow-up: `memory-center-trust.test.tsx` now covers Memory Center delete, GDPR erase, and allow re-import without native `confirm()`, rendered `J3f` passes on port `34164`, and the expanded user-journey suite passes 22/22 on port `34165`.
|
||||
- Current follow-up: `wiki-export-trust.test.tsx` now covers Wiki Obsidian and Notion export destinations without native `prompt()`, rendered `J3g` passes on port `34167`, and the path remains included in the latest 24/24 user-journey suite on port `34173`.
|
||||
- Current follow-up: `settings-trust.test.tsx` now covers Settings telemetry clear, backup failure, and restore success without native dialogs, rendered `J3h` passes on port `34171`, and the expanded user-journey suite passes 24/24 on port `34173`.
|
||||
- Current follow-up: `p1b-authgate-surfaces.test.tsx` now covers standalone `BackupApp` restore without native `confirm()`.
|
||||
- Current follow-up: `phase3b-automation-center.test.tsx` now covers Automation delete without native `confirm()`.
|
||||
- Current follow-up: `compliance-template-trust.test.tsx` now covers compliance template delete without native `confirm()`.
|
||||
- Current follow-up: `admin-pages.test.ts` now covers admin-web member removal without native `confirm()`.
|
||||
- Current follow-up: the five-persona scorecards now require explicit per-persona T12 state bundles and are guarded by `tests/five-persona-state-bundle-contract.test.ts`.
|
||||
- Current follow-up: `tests/e2e/five-persona-state-bundles.spec.ts` now writes rendered state-bundle evidence and screenshots for all five judge personas under `output/playwright/five-persona-state-bundles/`.
|
||||
- Current follow-up: `MemoryCenterApp` lets the Memory view tablist wrap on mobile, guarded by `p3-memory-center-app.test.tsx`; the regenerated Mobile Executive bundle reports 0 Memory-route overflow items at 390 x 844.
|
||||
- Current follow-up: five-persona browser bundles now include sampled failure/workflow/scale/slow-data probes for chat backend offline, Memory API unavailable, Memory slow-list handling, Memory large-list handling, Timeline/Event large-list handling, Launcher sidecar offline, Cockpit health degraded, Agents slow-list handling, Agents large-list handling, Marketplace unavailable, Marketplace large-catalog handling, Files upload failure, Files upload success, Files large-list handling, billing checkout unavailable, Team active billing state, Team settings unlocked state, billing checkout success return, billing checkout cancel return, backup creation failure, backup restore failure, approval grant revoke-all, local model runtime unavailable, and mobile chat backend offline. The same E2E gate now fails on critical console errors, critical network failures, page errors, and visible horizontal overflow.
|
||||
- Current follow-up: `WorkspaceDesktopApp` wraps workspace tabs on mobile, and `ChatApp` wraps the composer agent strip with a shorter mobile model chip; the regenerated Mobile Executive chat failure bundle reports 0 visible overflow at 390 x 844.
|
||||
- Current follow-up: the five-persona bundle now records explicit overlay evidence and screenshots. Mobile Executive captures Notification Inbox and Command Center; `NotificationInbox` no longer uses horizontal entrance motion that can transiently overflow the 390 px viewport.
|
||||
- Chat stream failure-injection specs exist for network drop and retry behavior.
|
||||
- The existing state matrix correctly separates billing tier from UI disclosure tier, which is essential for Solo-vs-Team scoring.
|
||||
- The judge runbook already requires state-bundle files, console capture, route evidence, screenshots, and deferral records.
|
||||
|
||||
## Current Gaps
|
||||
|
||||
- Scorecards now require declared state bundles and current rendered evidence folders tie route, account mode, billing tier, disclosure tier, model state, data state, offline/error state, viewport, screenshots, and console status into one five-persona no-LLM/accountless bundle.
|
||||
- Component/unit state tests are not enough to prove shell routing, viewport fit, focus behavior, console health, or user-visible recovery copy.
|
||||
- Current high-confidence production native-dialog scan is clean after the focused fixes; T7/T12 now needs broader persona-state evidence, keyboard/screen-reader validation, and failure-state proof rather than more known browser-native dialog replacement.
|
||||
- Offline/API failure and sampled scale coverage is broader but still incomplete. Chat backend offline, Memory API unavailable, Memory slow-list handling, Memory large-list handling, Timeline/Event large-list handling, Launcher sidecar offline, Cockpit health degraded, Agents slow-list handling, Agents large-list handling, Marketplace unavailable, Marketplace large-catalog handling, Files upload failure, Files upload success, Files large-list handling, billing checkout unavailable, Team active billing state, Team settings unlocked state, billing checkout success return, billing checkout cancel return, backup creation failure, backup restore failure, approval grant revoke-all, local model runtime unavailable, and mobile chat backend offline now have bundled browser evidence; real deployed/provider checkout success/cancel, real authenticated Teams server, packaged desktop/launch, hook lifecycle, broader slow-data/performance states, and deeper performance evidence still need user-visible evidence or approved deferral.
|
||||
- Test output is noisy enough that a final judge lane could hide new regressions.
|
||||
- Mobile state evidence is now concrete for Home, Profile, Settings, Memory, mobile chat backend-offline recovery, Notification Inbox, and Command Center at 390 x 844, with Memory, workspace tabs, chat agent-strip, notification animation, and Command Center overflow/close evidence fixed; authenticated mobile Team paths still need targeted evidence.
|
||||
- Scale/performance state evidence now has five rendered large-list slices, Files, Marketplace, Memory, Agents, and Timeline/Events, plus two sampled slow-data slices for delayed Memory and Agents loading. Broader slow-data states and deeper app-payload/performance gates remain weak.
|
||||
|
||||
## Correction Candidates
|
||||
|
||||
| ID | Correction | Phase recommendation | Closure evidence |
|
||||
|---|---|---|---|
|
||||
| T12-A | Add a required state-bundle row to every judge scorecard and evidence folder. | Scorecard contract guarded; browser evidence captured for the default accountless/no-LLM lane. | Each scorecard now declares account, billing, disclosure, model, data, offline/error, viewport, and non-main gate decisions, guarded by `tests/five-persona-state-bundle-contract.test.ts`; `output/playwright/five-persona-state-bundles/*/state-bundle.md` now exists for all five personas. Remaining closure requires real authenticated Teams server/failure-state bundles or explicit deferrals. |
|
||||
| T12-B | Add a lightweight browser state-bundle smoke after Phase 1 for the five personas. | Codified and passing for default accountless/no-LLM lane plus sampled failure and overlay probes. | `tests/e2e/five-persona-state-bundles.spec.ts` captures route sequence, screenshot paths, console summary, visible element-bounds results, overlay evidence, failure probes, and state fields for all five personas. Remaining closure requires real authenticated Teams server, packaged desktop/launch, and broader destructive/failure evidence or approved deferrals. |
|
||||
| T12-C | Expand failure evidence beyond chat SSE to sidecar/offline, Stripe cancel/unavailable, marketplace unavailable, model unavailable, and backup/restore failure states. | Partially fixed for sampled persona-critical failures. | Current browser bundle proves chat backend offline, Memory API unavailable, Memory slow-list handling, Memory large-list handling, Wiki export failure, Timeline/Event large-list handling, Launcher sidecar offline, Cockpit health degraded, Agents slow-list handling, Agents large-list handling, Marketplace unavailable, Marketplace large-catalog handling, Files upload failure, Files upload success, Files large-list handling, billing checkout unavailable, Team active billing state, Team settings unlocked state, billing checkout success return, billing checkout cancel return, backup creation failure, backup restore failure, approval grant revoke-all, local model runtime unavailable, and mobile chat backend offline with branded recovery copy, successful workflow proof, sampled slow-data proof, sampled Teams-tier proof, sampled scale proof, or sampled destructive-action proof, 0 critical console/page/network failures, and 0 visible overflow. Remaining closure: real deployed/provider checkout success/cancel, real authenticated Teams server, packaged desktop/launch, hook lifecycle, broader slow-data/performance states, deeper performance evidence, and approved deferrals where applicable. |
|
||||
| T12-D | Replace or explicitly defer native dialogs on persona-critical destructive flows. | Phase 2, shared with T7 | Current high-confidence production native-dialog scan is clean; Approvals revoke-all now has component, rendered user-journey, and five-persona Team Admin bundle evidence; Artifact permanent delete, Memory Center delete/erase/re-import, Wiki export destinations, Settings telemetry/backup/restore, standalone BackupApp restore, Automation delete, compliance template delete, admin member removal, and Create Workspace template delete now have in-app confirmation evidence. Remaining closure needs focus/keyboard behavior evidence, authenticated/audit-history proof, and less common destructive paths. |
|
||||
| T12-E | Reduce expected warning noise in focused state tests. | Phase 2, shared with T10 | The state slice output is short enough that real failures are visible; known unavoidable warnings are isolated or documented. |
|
||||
| T12-F | Attach non-main gate decisions to persona bundles. | Phase 2/launch | T13/T14/T15/T16/T17/T18/T19 are evidenced or explicitly deferred per affected persona. |
|
||||
| T12-G | Add scale/performance state evidence for large lists and slow/failed data. | Partially fixed for sampled Files, Marketplace, Memory, Agents, Timeline/Event large-list states, plus slow Memory and Agents states. | `files-large-list` now renders 240 mocked files in the Engineer bundle, proves `bulk-file-000.md` plus `240 items`, and reports 0 critical console/page/network failures and 0 visible overflow. `marketplace-large-catalog` renders a 240-entry mocked Marketplace catalog, proves `Bulk Skill 000` plus grouped `Skills 120`, `Connectors 60`, and `MCPs 60`, and reports 0 critical console/page/network failures and 0 visible overflow. `memory-slow-list` delays Memory loading by 2 seconds, proves the aria-busy `Loading memories` live status before rendering `40 memories` plus `Bulk Memory 000`, and reports 0 critical console/page/network failures and 0 visible overflow. `memory-large-list` renders a 200-entry mocked Memory Center list in the Researcher bundle, proves `200 memories` plus `Bulk Memory 000`, and reports 0 critical console/page/network failures and 0 visible overflow. `agents-slow-list` delays Agents loading by 2 seconds, proves the aria-busy `Loading agents` live status before rendering `40 agents` plus `Bulk Agent 000`, and reports 0 critical console/page/network failures and 0 visible overflow. `agents-large-list` renders a 180-agent mocked roster in the Engineer bundle, proves `180 agents` plus `Bulk Agent 000`, and reports 0 critical console/page/network failures and 0 visible overflow. `timeline-large-events` renders 360 mocked Timeline events in the Researcher bundle, proves `360 events` plus `Used bulk_tool_000`, and reports 0 critical console/page/network failures and 0 visible overflow. Remaining closure needs broader slow-data/performance evidence or approved deferrals. |
|
||||
|
||||
## Approval Recommendation
|
||||
|
||||
Keep T12 out of Phase 1 implementation except for the state fields needed when Phase 1 adds T11 route evidence. After Phase 1, run a judge dry run only as advisory unless T12-A through the persona-critical portions of T12-F have evidence or approved deferrals.
|
||||
95
docs/audits/2026-07-08-ux-approval-brief.md
Normal file
95
docs/audits/2026-07-08-ux-approval-brief.md
Normal file
@@ -0,0 +1,95 @@
|
||||
# UX Phase 1 Approval Brief
|
||||
|
||||
Status: analysis complete enough to request Phase 1 implementation approval. No product code has been changed by this analysis packet.
|
||||
|
||||
Packet index: `docs/audits/2026-07-08-ux-packet-index.md`.
|
||||
|
||||
## Decision Requested
|
||||
|
||||
Approve Phase 1 implementation from `docs/superpowers/plans/2026-07-08-ux-phase-1-corrections.md`.
|
||||
|
||||
Recommended execution mode: Subagent-Driven, with one focused worker per task and review after each task.
|
||||
|
||||
Fallback execution mode: Inline Execution in this session, with checkpoints after each task.
|
||||
|
||||
## Why Phase 1 First
|
||||
|
||||
The current built app is broadly functional, but seven P0 issues prevent an honest five-persona 9/10 score:
|
||||
|
||||
1. Local auth/CSP/Clerk console errors.
|
||||
2. Mobile Settings layout failure.
|
||||
3. Mobile first-run onboarding hides the primary Continue action.
|
||||
4. Active user-facing Pro copy after the Solo/Team collapse.
|
||||
5. Visual snapshots failing every tracked view.
|
||||
6. `Ctrl+Shift+N` shortcut contract failure.
|
||||
7. Workspace Switcher overlay blocking route traversal.
|
||||
|
||||
Phase 1 also includes the minimum route-evidence work needed to stop thin routes from being skipped in the final judge gate.
|
||||
|
||||
## Approved Scope If User Says "Approve Phase 1"
|
||||
|
||||
- T1 local auth, Clerk, and CSP console health.
|
||||
- T2 mobile Settings and first-run onboarding responsive layout.
|
||||
- T3 Solo/Teams/Enterprise copy cleanup.
|
||||
- T4 `Ctrl+Shift+N` and Workspace Switcher route contract.
|
||||
- T5 visual snapshot triage.
|
||||
- T11 route evidence for thin judge paths.
|
||||
- Update the audit/register/scorecards with actual verification evidence.
|
||||
|
||||
## Explicitly Out Of Scope For Phase 1
|
||||
|
||||
- Broad redesigns or new product surfaces.
|
||||
- Full native-dialog replacement across the entire app.
|
||||
- Performance chunk-splitting beyond direct Phase 1 test blockers.
|
||||
- Local model pricing semantics unless a Phase 1 verification run forces a tiny supporting fix.
|
||||
- Marketplace live-sync architecture changes beyond documenting current flake and keeping standard audit deterministic.
|
||||
- Public launch site, download, checkout, legal, GitHub Pages deployment, installer, tray, update, signing, sidecar-startup, admin web, CLI launcher, marketplace CLI, memory/MCP utility gates, AI-tool hook lifecycle gates, Browser Companion extension gates, developer API/background/substrate verification gates, and ops/deployment/CI/benchmark/judging gates, except where T1 auth/CSP overlap directly blocks Phase 1 verification.
|
||||
|
||||
## Evidence Packet
|
||||
|
||||
- Analysis completion audit: `docs/audits/2026-07-08-analysis-completion-audit.md`
|
||||
- Full audit and findings: `docs/audits/2026-07-08-complete-ux-usage-audit.md`
|
||||
- Route/scenario manifest: `docs/audits/2026-07-08-ux-route-scenario-manifest.md`
|
||||
- Route evidence T11 analysis: `docs/audits/2026-07-08-route-evidence-t11-analysis.md`
|
||||
- State/failure scenario matrix: `docs/audits/2026-07-08-ux-state-failure-scenario-matrix.md`
|
||||
- Focused T12 state/failure analysis: `docs/audits/2026-07-08-state-failure-t12-analysis.md`
|
||||
- Focused Mobile Executive T2/T12 analysis: `docs/audits/2026-07-08-mobile-executive-t2-t12-analysis.md`
|
||||
- Focused visual T5 classification: `docs/audits/2026-07-08-visual-t5-classification.md`
|
||||
- Focused first-run onboarding T1/T2/T12 analysis: `docs/audits/2026-07-08-first-run-onboarding-t1-t2-t12-analysis.md`
|
||||
- Non-main surface scope: `docs/audits/2026-07-08-ux-non-main-surface-scope.md`
|
||||
- Five-persona scorecards: `docs/audits/2026-07-08-five-persona-judge-scorecards.md`
|
||||
- Correction register: `docs/audits/2026-07-08-ux-correction-register.md`
|
||||
- Desktop wrapper T14 analysis: `docs/audits/2026-07-08-desktop-wrapper-t14-analysis.md`
|
||||
- Admin/CLI utility T15 analysis: `docs/audits/2026-07-08-admin-cli-utility-t15-analysis.md`
|
||||
- AI-tool hook lifecycle T16 analysis: `docs/audits/2026-07-08-ai-tool-hook-t16-analysis.md`
|
||||
- Developer/substrate T17 analysis: `docs/audits/2026-07-08-developer-substrate-t17-analysis.md`
|
||||
- Ops/deployment/CI/judging T18 analysis: `docs/audits/2026-07-08-ops-deploy-ci-judging-t18-analysis.md`
|
||||
- Public launch funnel T13 analysis: `docs/audits/2026-07-08-launch-funnel-t13-analysis.md`
|
||||
- Browser Companion T19 analysis: `docs/audits/2026-07-08-browser-companion-t19-analysis.md`
|
||||
- Runtime accessibility T10 analysis: `docs/audits/2026-07-08-runtime-a11y-t10-analysis.md`
|
||||
- Shell overlay T10/T12 analysis: `docs/audits/2026-07-08-shell-overlays-t10-t12-analysis.md`
|
||||
- Phase 1 implementation plan: `docs/superpowers/plans/2026-07-08-ux-phase-1-corrections.md`
|
||||
|
||||
## Phase 1 Completion Gate
|
||||
|
||||
Phase 1 is not complete until:
|
||||
|
||||
- The Phase 1 verification commands in the plan pass, or visual baseline decisions are explicitly reviewed and recorded.
|
||||
- T5 uses the fresh visual classification as its starting point: current desktop actuals are coherent, baselines are stale, and `--update-snapshots` waits until Phase 1 UI changes are approved and landed.
|
||||
- Required desktop/mobile screenshots are captured or inspected.
|
||||
- Mobile checks prove critical visible controls stay within the viewport on `/settings`, `/settings?tab=models`, `/settings?tab=billing`, `/settings/profile`, `/memory`, and workspace chat; document-level scroll width alone is not sufficient.
|
||||
- First-run onboarding checks prove a clean accountless data dir can reach the wizard without Clerk/CSP console errors and the 390 x 844 Profile step keeps its primary Continue action visible or clearly reachable.
|
||||
- The selected Mobile Executive overlay path opens and closes without trapping focus or scroll. The fresh mobile smoke showed Workspace Switcher closing with Escape but Command Center still visible after Escape, so Command Center cannot be treated as passing until fixed or deferred.
|
||||
- The correction register rows for T1, T2, T3, T4, T5, and T11 are updated with current evidence.
|
||||
- The current all-route route-smoke evidence is codified into a repeatable route owner or explicitly deferred; `/benchmarks`, `/platform`, `/payment-cancelled`, Launcher watch mode, Usage & Cost, and the catch-all route have route-health assertions that distinguish expected states from console failures.
|
||||
- No P0 remains open.
|
||||
|
||||
## After Phase 1
|
||||
|
||||
Run a judge dry run. If no installed-app P0 remains and the scorecards are no longer capped by Phase 1 blockers, proceed to Phase 2/launch readiness: trust-critical dialogs, marketplace determinism, form accessibility, runtime axe/DOM accessibility findings, warning hygiene, T12 state/failure bundle evidence, mobile Memory/chat tab behavior, first-run import/model/auto-send polish, Command Center mobile close/label fit if selected for scoring, shell overlay semantics/close/hierarchy fixes, T13 launch-funnel repair/evidence, T14 desktop-wrapper evidence, T15 admin/CLI/MCP utility evidence, T16 AI-tool hook lifecycle evidence, T17 developer API/background/substrate verification evidence, T18 ops/deployment/CI/benchmark/judging evidence, T19 Browser Companion extension evidence, and other P1 items.
|
||||
|
||||
T12 now has focused supplements: a passing state-slice unit run is good signal; the mobile smoke proves screenshot and critical-element bounds are required because clean document scroll width can still hide clipped controls; the first-run smoke proves skip-onboarding harnesses do not cover accountless setup, high-volume import decisions, or mobile Profile primary-action reachability; and the shell-overlay smoke proves Notification Inbox and Create Workspace still fail basic close/semantics expectations. T10 now also has runtime evidence: axe/DOM smoke found critical unnamed controls/selects, a Files scroll-region keyboard issue, workspace semantics/image-alt issues, Command Center dialog/label-fit warnings, and shell-overlay unnamed icon/landmark gaps. Final scoring still needs state bundles that tie account mode, billing tier, UI disclosure, model state, data state, offline/error state, viewport, route screenshots, and console status to each persona.
|
||||
|
||||
T13 now includes a launch-scoped P0: www tests/typecheck/build pass and corrected-host public-site smoke renders core routes locally, but the canonical `waggle-os.ai` domain does not resolve from the audit environment, real signed installer artifacts are not published, deployed Vercel/DNS proof is still missing, and deployed Clerk/Stripe/legal sign-off evidence remains open. T14 records strong static/Rust/service evidence plus release/tray source hardening: Settings is bridged, Quit is native, and Pause/About are hidden, but packaged tray/close/shortcut evidence, update/service UX, installer/signing proof, package-local web Tauri command shape, and absent packaged interaction evidence keep T14 open. T15 now records focused marketplace built help/invalid-command plus package manifest/packed-file alignment and clean installed packed-CLI `npx` help/invalid recovery, launcher help/invalid-port/occupied-port recovery plus packed first-command, clean installed occupied-port startup recovery, and clean installed long-running `/health` startup, `@waggle/cli` built/packed/local package-closure installed `npx` help plus installed local REPL startup/slash-command/exit and streamed chat/provider plumbing, memory MCP read/write plus local package-closure installed read-only startup, hive-mind MCP read/write plus local package-closure installed read-only startup, hive-mind CLI local package-closure installed `npx` help, and admin-web unit plus rendered package coverage across all seven pages at desktop/mobile widths, including browser back/forward traversal, page-level keyboard traversal, desktop/mobile visual snapshots, malformed analytics response recovery, all-page initial API-failure recovery, rendered mutation/destructive-failure recovery for capability policy save, capability override create/remove, capability request decision, member invite, member role change, member removal, and team settings save, plus real local bearer-auth wrong-token/valid-token behavior. Registry-only proof after internal package publication still keeps the utility lane open.
|
||||
|
||||
T16 now records strong route/contract, hook package, hook/shim package-local scripts, typecheck, compiled-bin, packed-package `npx` install/verify/uninstall lifecycle, focused no-reconnect observed-output coverage, hook stdout/stderr/Backup-Recovery/Check-failed/uninstall-cleanup/More-output/structured-failure/empty-output/Claude Desktop launch-only detail coverage, focused registry-aware third-party adapter launch coverage, partial rendered Launcher Browser evidence, codified Playwright standard install changed-file/pointer/backup/recovery labels, all six hook-capable install/verify/uninstall rendered transitions, sidecar-offline Retry, long-stderr summarization, and non-built-in adapter launch-only/prompt proof, Browser smokes where real Verify renders `verify failed (exit 1)` with retry/uninstall/reinstall guidance instead of `HTTP 400`, mocked Codex Verify renders manual-approval rows without raw `[FAIL]`, and mocked Codex Uninstall renders restore/cleanup rows without `Install pointer`, plus a gated real-tool Playwright smoke where OpenClaw renders as a real detected CLI and launches through the sidecar with observed output, exit 0, and process cleanup. A broadened gated route smoke proves `/api/tools/hooks` can run real install, verify, and uninstall for `claude-code`, `codex`, `codex-desktop`, `cursor`, `hermes`, and `openclaw` against an isolated profile and clean up afterward; the earlier red run exposed Windows `execFile('npx')` shim resolution, now fixed by resolving the Node-installed `npx.cmd` and using the shared `.cmd` resolver. Windows npm shims now resolve to Node module targets when possible, with a quoted fallback for unknown `.cmd`/`.bat` files, Codex found only through the restricted WindowsApps alias is now installed-but-not-launchable with recovery copy, Tailwind motion-token ambiguity warnings are removed, and the `shape-selection.ts` dynamic/static import warning is removed. T16 still needs packaged desktop hook-status evidence and remaining warning hygiene. T17 records remaining non-hook package-local test command failures and a full server-suite perf flake. T18 records secret-safe ops logging, Render target, CI/infra, benchmark command-shape, and current judging-artifact gaps. T19 now records direct sidecar save success, content-script extraction from a normal page, legacy-trust extension save success, secure-default session-token bootstrap, MV3 service-worker no-Origin handling, background save auth, sticky setup errors, restricted-page recovery copy, disabled-state styling, honest memory-destination copy, loaded-extension save/frame confirmation, popup keyboard/focus/Enter save proof, direct popup Save page click evidence, rendered Memory UI confirmation after secure popup saves, context-menu handler coverage, stable packaged-ID pairing proof, Memory search provenance consistency for imported captures, and existing chat `auto_recall`/catch-up imported provenance. It still needs native toolbar-bubble evidence or manual release proof, native context-menu click evidence or explicit deferral, signed Web Store/installer-distributed extension proof if release packaging is scored, and separate proof for any future recall result shape if scored.
|
||||
148
docs/audits/2026-07-08-ux-correction-register.md
Normal file
148
docs/audits/2026-07-08-ux-correction-register.md
Normal file
@@ -0,0 +1,148 @@
|
||||
# Waggle OS UX Correction Register
|
||||
|
||||
Companion artifacts:
|
||||
|
||||
- `docs/audits/2026-07-08-ux-approval-brief.md`
|
||||
- `docs/audits/2026-07-08-complete-ux-usage-audit.md`
|
||||
- `docs/audits/2026-07-08-ux-route-scenario-manifest.md`
|
||||
- `docs/audits/2026-07-08-ux-state-failure-scenario-matrix.md`
|
||||
- `docs/audits/2026-07-08-ux-non-main-surface-scope.md`
|
||||
- `docs/audits/2026-07-08-five-persona-judge-scorecards.md`
|
||||
- `docs/audits/2026-07-08-five-persona-judge-runbook.md`
|
||||
- `docs/audits/2026-07-08-ux-post-phase-1-roadmap.md`
|
||||
- `docs/audits/2026-07-08-web-guidelines-line-findings.md`
|
||||
- `docs/audits/2026-07-08-runtime-a11y-t10-analysis.md`
|
||||
- `docs/audits/2026-07-08-visual-t5-classification.md`
|
||||
- `docs/audits/2026-07-08-source-inventory-consistency-audit.md`
|
||||
- `docs/audits/2026-07-08-browser-companion-t19-analysis.md`
|
||||
- `docs/audits/2026-07-08-desktop-wrapper-t14-analysis.md`
|
||||
- `docs/audits/2026-07-08-admin-cli-utility-t15-analysis.md`
|
||||
- `docs/audits/2026-07-08-ai-tool-hook-t16-analysis.md`
|
||||
- `docs/audits/2026-07-08-developer-substrate-t17-analysis.md`
|
||||
- `docs/audits/2026-07-08-ops-deploy-ci-judging-t18-analysis.md`
|
||||
- `docs/audits/2026-07-08-launch-funnel-t13-analysis.md`
|
||||
- `docs/audits/2026-07-08-route-evidence-t11-analysis.md`
|
||||
- `docs/audits/2026-07-08-state-failure-t12-analysis.md`
|
||||
- `docs/audits/2026-07-08-mobile-executive-t2-t12-analysis.md`
|
||||
- `docs/audits/2026-07-08-first-run-onboarding-t1-t2-t12-analysis.md`
|
||||
- `docs/audits/2026-07-08-shell-overlays-t10-t12-analysis.md`
|
||||
- `docs/superpowers/plans/2026-07-08-ux-phase-1-corrections.md`
|
||||
|
||||
Purpose: one tracking register for every UX finding currently blocking or reducing the five-persona 9/10 goal. This file does not replace the audit; it maps audit findings to tickets, phases, likely files, judge impact, and closure evidence.
|
||||
|
||||
Final reconciliation (2026-07-13): the row-level entries below preserve the chronology of the audit and may describe an earlier branch state. The current integrated result is authoritative in `docs/audits/2026-07-13-final-goal-verification.md`. In particular, Channels hardening is integrated, the final five-persona run passes 5/5 at 9.1-9.5, full current-head lint/typecheck/build/test/visual/accessibility gates pass, and Tauri now has generated MSI and NSIS packages plus a healthy launch from an extracted MSI payload. T13 public DNS/deployment, production signing/updater delivery, and credential-dependent external-provider smokes remain release gates; they are not represented as completed product work.
|
||||
|
||||
Status key:
|
||||
|
||||
- `Open`: analyzed, not fixed.
|
||||
- `Phase 1 Pending Approval`: included in the first implementation plan, awaiting user approval before coding.
|
||||
- `Partially Fixed`: implementation and verification landed for part of the finding, but the ticket still has named remaining scope.
|
||||
- `Phase 2 Pending`: important for 9/10, but intentionally after Phase 1.
|
||||
- `Phase 3 Pending`: polish/performance work after trust/flow blockers.
|
||||
- `Tooling`: not product UX, but affects repeatable audit confidence.
|
||||
|
||||
## Register
|
||||
|
||||
| Finding | Ticket | Phase | Status | Primary files/surfaces | Judge impact | Closure evidence |
|
||||
|---|---|---:|---|---|---|---|
|
||||
| P0-1 Accountless Clerk/CSP console health | T1 | 1 | Verified Fixed | `packages/server/src/local/security-middleware.ts`, `packages/server/tests/local/security-middleware.test.ts`, `apps/web/src/lib/clerk.ts`, `apps/web/src/providers/WaggleClerkProvider.tsx`, `tests/e2e/full-product-audit.spec.ts`, `tests/e2e/phase-ab-verification.spec.ts` | Solo founder, Team admin, Engineer | Accountless local lane has no Clerk/CSP errors in current focused checks: `clean first-run onboarding loads without Clerk, CSP, or page errors`, `no console errors on initial load`, and `no critical console errors on load` passed 3/3 on port `34196`. Auth still needs explicit-enabled coverage as a separate state bundle. |
|
||||
| P0-2 Mobile Settings structural usability | T2 | 1 | Verified Fixed | `apps/web/src/components/os/apps/SettingsApp.tsx`, `tests/e2e/user-journeys.spec.ts`, Settings visual/mobile screenshots | Mobile executive, Team admin | Current 390 x 844 Settings has no document-level overflow, no clipped critical controls, no squeezed two-pane rail, and no visible element overflow in general/models/billing/profile: `J-mobile: Settings is usable at 390px width` passed 1/1 on port `34195`. |
|
||||
| P0-7 Mobile first-run onboarding primary action reachability | T2/T12 | 1 | Verified Fixed | `apps/web/src/components/os/overlays/OnboardingWizard.tsx`, `apps/web/src/components/os/overlays/onboarding/WhoAreYouStep.tsx`, `tests/e2e/user-journeys.spec.ts` | Mobile executive, Solo founder | Current 390 x 844 clean-data onboarding Profile keeps the primary Continue action reachable with no visible overflow: `J-mobile: first-run onboarding keeps primary actions reachable at 390px width` passed 1/1 on port `34194`. |
|
||||
| P0-3 Cockpit pricing copy uses Solo/Team/Enterprise | T3 | 1 | Verified Fixed | `MarketplaceApp.tsx`, `AddCustomMcpForm.tsx`, `command-catalog.ts`, `LoginBriefing.tsx`, `SkillRow.tsx`, `SettingsApp.tsx`, `PaymentSuccessApp.tsx` | Solo founder, Team admin, Mobile executive | Active cockpit UI copy uses Solo, Team, and Enterprise. Current source search finds remaining `Pro` references only in explicit legacy billing compatibility (`Legacy Pro`, `Pro (legacy)`), tests/comments, or model names; public legal Pro copy is tracked under launch-funnel T13. |
|
||||
| P0-4 Visual regression suite for tracked views | T5 | 1 | Verified Fixed | `tests/visual/views.spec.ts`, `tests/visual/baselines/**`, affected rendered routes, `docs/audits/2026-07-08-visual-t5-classification.md` | All personas | Current visual suite passes 14/14 on port `34199`; canonical ASCII-hyphen baselines are active, volatile Home text is masked in the visual spec, and duplicate historical baseline families are documented. |
|
||||
| P0-5 `Ctrl+Shift+N` shortcut route contract | T4 | 1 | Verified Fixed | `apps/web/src/components/os/AppShell.tsx`, `apps/web/src/hooks/useKeyboardShortcuts.ts`, `tests/e2e/phase-ab-verification.spec.ts`, `tests/e2e/power-user-stress.spec.ts` | Engineer, Solo founder | Shortcut opens active or first available workspace chat; both shortcut E2E tests passed 2/2 on port `34197`. |
|
||||
| P0-6 Workspace Switcher route traversal | T4 | 1 | Verified Fixed | `AppShell.tsx`, `WorkspaceSwitcher.tsx`, `tests/e2e/full-wiring-audit.spec.ts` | Engineer, Mobile executive | Route-changing navigation closes or supersedes the switcher; `traverse all sidebar views — zero critical JS errors` passed 1/1 on port `34198`, proving traversal is not intercepted by backdrop. |
|
||||
| P1-1 Native browser dialogs interrupt branded workflows | T7 | 2 | Current Scan Fixed | `BackupApp.tsx`, `ApprovalsApp.tsx`, `AutomationCenterApp.tsx`, `ArtifactCenterApp.tsx`, `SettingsApp.tsx`, `MemoryCenterTab.tsx`, `WikiTab.tsx`, `CreateWorkspaceDialog.tsx`, `ComplianceTemplateModal.tsx`, `packages/admin-web/src/pages/Members.tsx` | Team admin, Researcher | Current high-confidence production native-dialog scan is clean. Create Workspace custom-template delete, Approvals revoke-all, Artifact permanent delete, Memory Center delete/erase/re-import, Wiki export destinations, Settings telemetry/backup/restore, standalone `BackupApp` restore, Automation delete, compliance template delete, and admin member removal now use in-app confirmations/forms/status with focused component/rendered coverage. Remaining T7 work is broader persona-state, failure-state, and accessibility evidence. |
|
||||
| P1-2 Test output is too noisy | T10 | 2 | Phase 2 Pending | Tests emitting repeated `act(...)` warnings; affected component tests | All personas indirectly | Standard verification output is short enough that real failures are visible; expected warning noise is eliminated or isolated. |
|
||||
| P1-3 Marketplace and local-first test determinism need tightening | T6 | 2 | Audit Lane Fixed | `packages/server/src/local/routes/marketplace.ts`, `marketplace-background-sync.ts`, E2E marketplace calls | Engineer, Solo founder | Manual `/api/marketplace/sync` now respects `WAGGLE_DISABLE_MARKETPLACE_SYNC=1` and returns a no-network skipped response; focused server tests pass 3/3, server typecheck passes, and the focused Playwright marketplace slice passes 4/4 on port `34203`. Live external-sync UX remains an explicit/non-default lane. |
|
||||
| P1-4 Initial app payload is too heavy | T8 | 3 | Focused Fixed; media residual | Route/app composition under `apps/web/src/components/os`, `shape-selection.ts`, persona/logo asset import sites | Mobile executive, Engineer | Startup JS is now below the Vite 500 kB warning threshold: routes, closed shell overlays, ChatHost, and PostHog are lazy-loaded and guarded by `build-warning-hygiene.test.ts`; current production build reports the startup chunk at 423.61 kB minified / 114.63 kB gzip with no large-chunk warning. Remaining media work is persona/logo asset optimization or explicit acceptance in the visual/CLS pass. |
|
||||
| P1-5 Form, focus, and icon-button accessibility need a focused pass | T10 | 2 | Partially Fixed | Settings, profile, workspace creation, workspace actions, agent builders/cards, Mission Control, compliance templates, onboarding, Launcher, Approvals, Files, workspace chat, Command Center | Mobile executive, Team admin, Engineer | Settings Prompt Shape/model/trust/team/KVARK fields, Profile identity/preferences/brand fields and Analyze Style action focus, Chat composer metadata/focus, Vault add-secret/actions, Launcher refresh/prompt and prompt focus ring, Approvals refresh/revoke, Cockpit/WaggleDance/ComplianceDashboard/AgentCard sampled actions, Agent/Skill/Automation Builder metadata and Skill Builder reorder focus rings, SkillEditorDrawer markdown-editor metadata, Automation Center template metadata/focus rings, AgentCard and GroupCard separate select/delete controls, full scoped production `transition-all` backlog closed with explicit transition properties, AgentCenterRow media dimensions, ReadyStep media dimensions, BootScreen/StatusBar media dimensions, LoginBriefing/Chat/SpawnAgentDialog media dimensions, Spawn Agent launch task/new-workspace metadata, Artifact Center search/create/detail-editor metadata and detail Kind focus ring, Agent Center search metadata, Agent Center templates search metadata, ExtensionCard inline connector-token metadata, InstallAuditPanel audit-filter metadata, ModelPilotCard budget-threshold slider metadata/focus ring, TelemetryApp daily-budget metadata, Skills Hub search metadata, Files toolbar action names/filter metadata/focus rings, Files new-folder/rename metadata, Files move/properties dialog close names/focus rings and row Properties context-menu behavior, Mission Control refresh/pause/resume/stop action names/focus rings, ConnectorCard setup metadata/action focus rings, Memory Center search/detail metadata and filter/action focus rings, MemoryCard selection checkbox labels/metadata, Memory Trust search/correction metadata and focus rings, TimelineTab search/filter metadata and filter-toggle name, EvolutionTab proposal review-note/New Run modal select/textarea metadata, close action name, and focus rings, Knowledge Graph search/scope metadata and toolbar/legend action names/focus rings, Harvest import/source action metadata and focus rings, Custom MCP form metadata/focus rings, MCP catalog search and scope-select metadata, ModelGate key/pull metadata, inline capability connector-token metadata, Telegram digest credential metadata, first-run onboarding profile/workspace/first-task metadata, EraseDataDialog destructive confirmation metadata/focus ring, Agent template creator/detail metadata/focus rings, All Workspaces search metadata/focus ring, Wiki search, WorkspaceActionsMenu rename/delete confirmations, warm AskBar metadata/focus ring, workspace TasksTab add-task metadata/focus ring, Timeline event-filter metadata/focus ring, Create Workspace visible setup/template/folder-picker metadata and focus rings, Compliance Dashboard report-option metadata and Compliance Template form metadata with token focus rings, Workspace Switcher focus-trap/return evidence, Files storage/browser scroll regions, workspace tab/image semantics, status-bar landmarks, sampled runtime axe, and Command Center description/mobile row fit plus search focus ring now have focused metadata/name/focus fixes and tests. Remaining T10 closeout is any broader unsampled metadata queue outside the covered surfaces, remaining unsampled icon-only controls, and broader modal focus-return evidence. |
|
||||
| P1-6 Local model pricing and no-LLM harness are inconsistent | T9 | 3 | Phase 3 Pending | `packages/agent/src/cost-tracker.ts`, `packages/agent/tests/cost-tracker.test.ts`, Usage & Cost UI | Engineer, Team admin | No unknown local-model cost warning in the audit lane; unpriced local models are labeled honestly. |
|
||||
| P1-7 Route coverage is uneven relative to the full shell | T11 | 1/2 | Verified Fixed | `tests/e2e/user-journeys.spec.ts`, `tests/visual/views.spec.ts`, `tests/vision/personas.spec.ts`, route/scenario manifest | All personas | Every registered route has an evidence owner in the manifest; prior zero/thin routes now have codified `J-route-coverage` owners, with the focused route coverage tests passing 2/2 on port `34200`. Deeper workflow/state evidence remains tracked in T10/T12/T16. |
|
||||
| P1-8 State and failure-mode coverage is not explicit enough for judge scoring | T12 | 2 | Default + Sampled Failure Bundles Captured | `docs/audits/2026-07-08-ux-state-failure-scenario-matrix.md`, `docs/audits/2026-07-08-five-persona-judge-scorecards.md`, `tests/five-persona-state-bundle-contract.test.ts`, `tests/e2e/five-persona-state-bundles.spec.ts`, `output/playwright/five-persona-state-bundles/`, `tests/e2e/failure-injection/network-drop.spec.ts`, `tests/e2e/user-journeys.spec.ts`, `tests/vision/personas.spec.ts` | All personas | Each judge scorecard now cites account mode, billing tier, disclosure tier, model state, data state, offline/error state, viewport, and non-main gate decisions, guarded by `tests/five-persona-state-bundle-contract.test.ts`; rendered accountless/no-LLM persona bundles now pass 5/5 with screenshots, sampled failure/workflow/scale/slow-data probes for chat backend offline, Memory API unavailable, Memory slow-list handling, Memory large-list handling, Timeline/Event large-list handling, Launcher sidecar offline, Cockpit health degraded, Agents slow-list handling, Agents large-list handling, Marketplace unavailable, Marketplace large-catalog handling, Files upload failure, Files upload success, Files large-list handling, billing checkout unavailable, Team active billing state, Team settings unlocked state, billing checkout success return, billing checkout cancel return, backup creation failure, backup restore failure, local model runtime unavailable, and mobile chat backend offline, plus 0 critical console/page/network failures and 0 visible overflow. Remaining closure requires real authenticated Teams server, real deployed/provider checkout success/cancel, packaged desktop/launch, hook lifecycle, broader slow-data/performance states, and approved deferrals where needed. |
|
||||
| P0-L1 Public launch funnel has release-blocking smoke and recovery gaps | T13 | 1/Launch | Partially Fixed Locally | `apps/www`, `apps/www/app/_components/Pricing.tsx`, `apps/www/app/_components/DownloadCTA.tsx`, `apps/www/app/download/page.tsx`, `apps/www/app/_lib/os-detection.ts`, `apps/www/app/api/stripe/checkout/route.ts`, `apps/www/app/(legal)/**`, `.github/workflows/deploy-www.yml`, DNS/deploy target | Solo founder, Team admin, Mobile executive | Local checkout/legal/download/deploy-workflow UX is improved: Pricing uses the canonical GET checkout route, cancelled checkout returns to `/?checkout=cancelled#pricing` with an inline retry notice, the public-site hydration issue badge is suppressed intentionally, legal placeholder/stale-tier copy is guarded, Download now routes to a controlled `/download` status page instead of an empty GitHub Releases target, mobile/tablet OS detection keeps the CTA generic instead of labeling iOS/Android as desktop installers, the public-site workflow now targets Vercel prebuilt deployment instead of GitHub Pages/static `apps/www/dist`, `apps/www` tests pass 19/19, typecheck passes, `npm run build:www` passes, rendered Browser recovery smoke has no warnings/errors, and built local smoke returns 200 for `/`, `/?checkout=cancelled`, and `/download`. Remaining launch blockers: canonical DNS/deployed Vercel proof, real signed installer/release artifact publication, real deployed Clerk/Stripe success/cancel evidence, and formal legal sign-off. |
|
||||
| P1-10 Desktop wrapper, installer, tray, update, and sidecar-startup UX lack installed-app evidence | T14 | 2/Launch | Partially Fixed; Packaged Startup Proven | `app/src-tauri/**`, `app/tests/**`, `app/scripts/**`, `.github/workflows/release.yml`, `packages/server/src/local/service.ts`, `scripts/build-sidecar.mjs`, `scripts/bundle-node.mjs`, `scripts/check-sidecar-resources.mjs`, `apps/web/src/lib/tauri-bindings.ts`, `apps/web/src/App.tsx` | All personas, especially Solo founder and Team admin | Release workflow now builds workspace packages before Windows/macOS sidecar packaging and is guarded by `tauri-config.test.ts` 25/25. Tray source now exposes only Open, Settings, and Quit; Settings is bridged to `/settings`, Quit uses native `app.exit(0)`, and unsupported Pause/About actions are hidden. Packaged debug startup now builds without installer, launches `target/debug/waggle.exe`, starts bundled `resources/service.js`, reaches `/health`, and has no CORS/module/ABI errors. Service/update events are consumed by the web bridge; native update emission remains disabled until signed updater artifacts exist. Remaining closure still requires MSI/installer packaging, real tray smoke, close-to-tray, global shortcut, forced watchdog-service UI, installer/signing trust, and signed-update proof or explicit deferral. |
|
||||
| P1-11 Admin web, CLI launcher, marketplace CLI, and memory MCP utility UX have broken built-entry and rendered/admin evidence blockers | T15 | 2/Launch | Phase 2 Pending; Utility CLI/MCP/Admin Rendered Fixes Landing | `packages/admin-web/**`, `packages/cli/**`, `packages/launcher/**`, `packages/marketplace/**`, `packages/memory-mcp/**`, `packages/hive-mind-mcp-server/**`, `packages/hive-mind-cli/**`, `vitest.config.ts` | Team admin, Engineer, Solo founder setup path | Marketplace CLI built help/invalid-command behavior, package manifest/packed-file alignment, and clean installed packed-CLI `npx` help/invalid recovery; `@waggle/cli` built/packed/local package-closure installed `npx` help plus installed local REPL startup/slash-command/exit and streamed chat/provider plumbing against a mock LiteLLM-compatible endpoint; launcher built help/invalid-port/occupied-port recovery plus packed first-command, clean installed occupied-port startup recovery, and clean installed long-running `/health` startup; sampled hive-mind CLI subcommand help/test discovery plus local package-closure installed `npx` help; legacy memory MCP read/write startup plus local package-closure installed read-only startup; hive-mind MCP write-scope roundtrip plus local package-closure installed read-only startup; and admin-web unit plus rendered package coverage are locally fixed and guarded. The rendered admin gate now includes browser back/forward traversal, page-level keyboard traversal, desktop/mobile visual snapshots, malformed analytics response recovery, all-page initial API-failure recovery, mutation/destructive-failure recovery for capability policy save, capability override create/remove, capability request decision, member invite, member role change, member removal, and team settings save, plus local bearer-auth wrong-token/valid-token behavior through protected Fastify routes. Remaining T15 closure still requires registry-only proof after internal package publication, or an explicit deferral. |
|
||||
| P1-12 AI-tool hook install/verify/uninstall UX lacks packaged desktop hook-status proof and fully quiet release output | T16 | 2/Launch | Phase 2 Pending; Rendered Lifecycle Fixed; Warning Hygiene Partially Fixed | `apps/web/src/components/os/apps/LauncherApp.tsx`, `apps/web/src/components/os/apps/launcher/ToolOutputPane.tsx`, `apps/web/src/lib/adapter.ts`, `packages/shared/src/tool-detection.ts`, `packages/agent/src/tool-detection.ts`, `packages/agent/src/tool-launcher.ts`, `packages/agent/src/tool-command.ts`, `packages/agent/src/tool-process-tracker.ts`, `packages/agent/src/tool-output-buffer.ts`, `packages/hive-mind-hooks-*/**`, `tests/e2e/launcher-rendered-states.spec.ts`, `tests/e2e/launcher-real-tool-lifecycle.spec.ts`, `tests/e2e/launcher-real-hook-lifecycle.spec.ts` | Engineer, Solo founder, Team admin | Packed-package `npx` command install/verify/uninstall now passes for the six hook-capable tools, hook/shim package-local scripts now pass, observed live output no longer reconnects/replays after exit in focused tests, Launcher component tests preserve hook stdout/stderr details, show Backup/Recovery labels for covered install output, summarize Verify `[FAIL]` output as Check-failed/manual-approval rows, label uninstall restore/cleanup rows without implying install state, cap long hook output behind `More output`, and prove launchable third-party adapters get a Launch action and prompt routing; adapter/route tests preserve structured hook failures and launch registered adapters through the runtime registry; live Verify now renders `verify failed (exit 1)` with retry/uninstall/reinstall guidance instead of `HTTP 400`; Browser mocks prove Codex Verify manual approval and Codex Uninstall cleanup panels; Playwright rendered coverage proves standard install changed-file/pointer/backup/recovery labels, sidecar-offline Retry, long stderr summarization, all six hook-capable rendered install/verify/uninstall transitions, non-built-in adapter launch-only/prompt behavior, and one real OpenClaw observed launch/output/exit/process-clear lifecycle; a broadened real route smoke proves `/api/tools/hooks` install/verify/uninstall for all six hook-capable tools in an isolated profile; Windows npm shims now resolve to Node module targets when possible and use a quoted fallback otherwise; Windows hook commands now resolve the Node-installed `npx.cmd`; Codex detected only through the restricted WindowsApps alias is now installed-but-not-launchable with recovery copy; Claude Desktop is explicitly launch-only with no hook actions; Tailwind motion-token ambiguity warnings are removed and guarded; and the `shape-selection.ts` dynamic/static import warning is removed and guarded. Remaining closure requires packaged desktop hook-status evidence and remaining warning hygiene; or T16 is explicitly deferred. |
|
||||
| P1-13 Developer API, background worker, and substrate package verification UX has command-shape and suite-stability gaps | T17 | 2/Tooling | Phase 2 Pending; Playwright Startup Fixed | `packages/sdk/**`, `packages/server/**`, `packages/worker/**`, `packages/waggle-dance/**`, `packages/hive-mind-core/**`, `packages/hive-mind-shim-core/**`, `packages/wiki-compiler/**`, `packages/*/package.json`, `vitest.config.ts`, `vitest.perf.config.ts`, Playwright `webServer` startup | Engineer, Team admin, release confidence | Root and package verification commands are documented and deterministic; package-local scripts either pass or point to the correct root/project-reference lane; Playwright `webServer` now starts with pinned `tsx@4.21.0` and matching `esbuild@0.27.7` evidence; the named server lane passes 185/2128 with one worker and the isolated perf lane passes 13/13; or T17 is explicitly deferred from the final score. |
|
||||
| P1-14 Ops, deployment, CI, benchmark, and judging evidence is incomplete for a complete-system claim | T18 | 2/Launch/Tooling | Phase 2 Pending | `.github/workflows/**`, `Dockerfile`, `docker-compose*.yml`, `render.yaml`, `litellm-config.yaml`, `ops/**`, `benchmarks/**`, `judging/**`, `vitest.infra*.ts` | Release confidence, Engineer, Team admin | YAML/Compose/benchmark evidence is attached; secret-safe validation commands are documented; Render/Docker/CI infra targets are coherent; current judge artifacts are generated from the current source; or T18 is explicitly deferred. |
|
||||
| P1-15 Browser Companion extension UX is a real capture surface with incomplete native-toolbar/context evidence | T19 | 2/Launch | Partially Fixed; Native Toolbar/Context Evidence Pending | `apps/browser-ext/**`, `packages/server/src/local/routes/browser-ext.ts`, `packages/server/src/local/cors-config.ts`, `packages/server/src/local/security-middleware.ts`, `packages/server/src/local/routes/memory.ts`, `apps/web/src/components/os/settings/CoverageCompassCard.tsx` | Researcher, Solo founder, Mobile executive | Secure token bootstrap, concrete extension-origin CORS matching, MV3 service-worker no-Origin handling, background save auth, sticky accessible popup recovery, restricted-page explanation, disabled-state styling, honest memory-destination copy, secure-default loaded-extension extraction/save/frame confirmation, popup keyboard/focus/Enter save proof, direct popup Save page click smoke, rendered Memory UI confirmation, context-menu handler coverage, stable packaged-ID pairing smoke, `/api/memory/search` imported provenance, and existing chat `auto_recall`/catch-up imported provenance are fixed and verified. Remaining closure requires native toolbar-bubble evidence or manual release proof, native context-menu click proof or deferral, signed Web Store/installer-distributed extension proof if release packaging is scored, and separate proof for any future recall result shape if scored; or T19 is explicitly deferred from the five-persona score. |
|
||||
| P1-16 Command Center mobile close and label fit needed proof | T10/T12 | 2 | Focused Fixed | `apps/web/src/components/os/overlays/CommandCenter.tsx`, `tests/e2e/user-journeys.spec.ts`, `tests/e2e/five-persona-state-bundles.spec.ts`, mobile overlay screenshots | Engineer, Mobile executive | Command Center now has a dialog description, mobile catalog subtitles truncate on their own line, `J-mobile: Command Center is described and fits at 390px` proves Control+K open/fit/Escape close, and the five-persona bundle now records Mobile Executive Command Center screenshot evidence with 0 visible overlay overflow. |
|
||||
| P1-17 High-volume first-run import is too easy to trigger | T7/T12 | 2 | Focused Fixed | `apps/web/src/components/os/overlays/onboarding/ImportStep.tsx`, `apps/web/src/test/onboarding-import-step.test.tsx`, Memory Harvest | Solo founder, Researcher, Team admin | Large detected histories now use deliberate review/secondary action copy: `Review after setup` is primary, immediate import is explicit as `Import 5,514 now`, and small histories keep the simple import CTA. Focused component coverage passes 2/2 and web typecheck passes; broader Harvest review/recovery states remain in T12. |
|
||||
| P1-18 First-run model/first-task handoff polish is inconsistent | T10/T12 | 2 | Focused Fixed | `ModelGateStep.tsx`, `ChatApp.tsx`, `ModelGateStep.test.tsx`, `lane-c-input-power.test.tsx` | Solo founder, Engineer | Model-ready onboarding now shows `Model ready` instead of the setup/checking gate while Continue is enabled, and first-task auto-send clears the untouched composer seed before the send promise resolves. Focused tests pass 5/5 and 12/12; the onboarding focused bundle passes 22/22; web typecheck passes. |
|
||||
| P1-19 Shell overlays do not share a consistent accessibility/close contract | T10/T12 | 2 | Partially Fixed | `NotificationInbox.tsx`, `CreateWorkspaceDialog.tsx`, `WorkspaceSwitcher.tsx`, `ContextRail.tsx`, `OnboardingTooltips.tsx`, `UpgradeModal.tsx`, `TrialExpiredModal.tsx`, focused overlay smoke | Mobile executive, Engineer, Solo founder | Notification Inbox, Create Workspace primary/subdialog/template-field contracts, Workspace Switcher focus-trap/return behavior, Context Rail, Onboarding Tooltips, and tier modal close labels now have focused contract coverage. Notification Inbox no longer uses horizontal entrance motion that can overflow mobile, and the five-persona bundle records Notification Inbox plus Command Center mobile overlay screenshots with 0 visible overflow; route-specific rendered evidence for less common overlay states remains. |
|
||||
| P1-20 Create Workspace is too dense and template-first on mobile | T10/T12/T7 | 2 | Partially Fixed; mobile disclosure verified | `CreateWorkspaceDialog.tsx`, workspace switcher/create flow, `shell-overlay-contracts.test.tsx`, `user-journeys.spec.ts` | Mobile executive, Solo founder, Team admin | Template delete now uses an in-app confirmation; optional templates and agent assignment use progressive disclosure; the agent section is collapsed by default at 390 x 844 and expands on demand. Focused shell-overlay contracts pass 15/15 and rendered `J-mobile` passes 1/1 on port `34380`. Broader screenshot/state coverage and modal focus-return evidence remain. |
|
||||
| P1-21 Provider model inventory was hardcoded and could hide newly released API models | T10/T12/T17/T18 | 2 | Hermetic routing fixed; external credential smoke pending | `packages/server/src/local/provider-model-catalog.ts`, `packages/server/src/local/litellm-runtime-config.ts`, `packages/server/src/local/provider-env.ts`, `packages/server/src/local/model-availability.ts`, `packages/server/src/local/routes/{providers,litellm,settings,agent,fleet}.ts`, `apps/web/src/{hooks/useProviders.ts,components/os/apps/ChatWindowInstance.tsx,components/os/model-gate/ModelGate.tsx}`, provider/router/completion tests | All personas, especially Engineer and Solo founder | Configured provider APIs supply every returned model id, including all Claude cursor pages and Gemini page tokens; Gemini discovery keeps the key in `x-goog-api-key`, not the URL. Startup migrates/hydrates credentials before the router snapshots env; a secret-free runtime config routes stable `provider/model` ids; key saves and manual retry rebuild/restart that catalog. If a provider releases a model while Waggle remains open, Settings/onboarding/Agent Builder and Chat re-pull catalogs on app focus; selecting or saving that model checks the running config, rebuilds and restarts the managed router only when the exact id is missing, and the shared resolver covers Chat, model switching, default-model saves, and fleet workers. Model-management requests now allow the router's real startup window instead of falsely timing out at 10 seconds. The latest focused hot-model lane passes 27/27 backend tests and 3/3 web tests; ModelGate/onboarding remains 28/28, both web typecheck and scoped diff checks pass, and rendered desktop plus 390 px recovery checks show one honest verdict with no horizontal overflow. A paid external-provider smoke remains a release-environment evidence item, not an implementation gap; the aggregate server typecheck is presently blocked only by unrelated parallel-session dependency/type changes recorded in current verification output. |
|
||||
| P1-22 Merged Channels had protected-route, loopback-auth, ordering, mobile, and WhatsApp secret-at-rest gaps | T10/T12/T14/T17/T18 | 2/Launch | Isolated hardening complete; integration and live-provider evidence pending | `apps/web/src/components/os/{apps/SettingsApp.tsx,settings/ChannelsSettings.tsx}`, `apps/web/src/lib/adapter.ts`, `packages/server/src/local/channels/**`, channel/chat/event routes, lockfile, focused tests, `docs/audits/2026-07-11-channels-ux-hardening.md` on `codex/channels-ux-hardening` | All personas, especially Engineer, Mobile executive, and Team admin | On the isolated branch, browser and channel loopback calls authenticate through the sidecar session token; configs validate atomically and audit without secret values; held tools enter durable app approval; duplicate deliveries are suppressed and same-chat turns serialize; workspace names resolve to stable IDs; errors/drafts/prerequisites are honest; Settings and channel controls fit at 390 px with the selected tab visible; and Baileys credentials/signal keys live in encrypted Vault state with one-time plaintext migration, fail-closed corruption behavior, logout wipe, and silent upstream logs. Normal clean install passes; backend Channels pass 89/89, focused web passes 12/12, server typecheck/package/web builds and diff checks pass, fresh sidecar/browser flows pass, and a real unpaired Baileys connection returns a QR without creating a plaintext auth directory. This is not yet merged into the dirty UX worktree. Remaining evidence: real Telegram/Discord/Slack credentials, WhatsApp secondary-device scan/restart/unlink, one real held-tool approval, and deliberate branch integration. |
|
||||
| P2-1 Stale comments and legacy terminology can mislead future work | Cleanup | 2/3 | Phase 2 Pending | Touched files only; especially tier/billing and route comments such as the stale `/workspaces` redirect note in `apps/web/src/routes/index.ts` | Indirect | Comments in touched files describe current Solo/Teams/Enterprise and route behavior; retired/embedded surfaces are classified before judge scoring. |
|
||||
| P2-2 Browser plugin DOM snapshot failed | Tooling | n/a | Tooling | Browser plugin path; Playwright CLI fallback | Audit process | Continue Playwright fallback until plugin mismatch is fixed; do not block product fixes on Browser DOM snapshot failure. |
|
||||
|
||||
## Current Non-Main Evidence Update
|
||||
|
||||
- Web Guidelines supplement added: `docs/audits/2026-07-08-web-guidelines-line-findings.md` records the native-dialog backlog (20 historical high-confidence hits, 0 current high-confidence hits after focused fixes), 0 current production `transition-all` hits after the focused transition-scoping sweep, 0 current high-confidence weak/missing focus replacements in the reviewed list after focused fixes, 0 remaining image-dimension findings after the SuggestedAgentCards, AgentCenterRow, ReadyStep, BootScreen, StatusBar, LoginBriefing, ChatApp, and SpawnAgentDialog fixes, and a broad form metadata queue. Phase 1 remains unchanged; the supplement strengthens T7/T8/T10/T12 closure evidence.
|
||||
- Current T10 follow-up evidence: Settings Prompt Shape and TimelineApp event-type selects now set `autocomplete="off"`, WikiTab search has an input-level token focus ring, and WikiTab Obsidian/Notion export targets expose target-specific `name`, `autocomplete="off"`, and token focus rings. Focused coverage: `settings-trust.test.tsx`, `timeline-app.test.tsx`, and `wiki-export-trust.test.tsx`.
|
||||
- Runtime accessibility T10 analysis updated: `docs/audits/2026-07-08-runtime-a11y-t10-analysis.md` records the original built-app axe/DOM smoke plus the codified gate, `tests/e2e/runtime-a11y.spec.ts`. The gate passes 2/2 on port `34193` for desktop and mobile Home, Settings, Profile, Vault, Mission Control, Memory, workspace chat, Agents, Waggle Dance, Launcher, MCP Hub, Files, and Approvals with zero axe violations. The first implementation slices fix named-control/form-metadata/focus issues for Settings, Profile identity/preferences/brand and Analyze Style action focus, Chat composer, Vault add-secret/actions, Launcher, Approvals, Cockpit/WaggleDance/ComplianceDashboard/AgentCard sampled actions, Agent/Skill/Automation Builder controls, Automation Center template controls, AgentCard and GroupCard select/delete controls, SuggestedAgentCards media/transition stability, AgentCenterRow media stability, ReadyStep media stability, BootScreen/StatusBar media stability, LoginBriefing/Chat/SpawnAgentDialog media stability, Spawn Agent launch controls, Agent template creator/detail controls, Artifact Center search/create/detail controls, Agent Center search, Skills Hub search, Files toolbar and inline rename/create-folder controls, Mission Control source action names/focus rings, ConnectorCard setup controls, Memory Center search/detail controls, MemoryCard selection checkbox labels/metadata, Memory Trust search/correction controls, TimelineTab search/filter controls, EvolutionTab proposal review-note and New Run modal controls, Knowledge Graph toolbar/search/scope controls, Harvest import/source controls, Custom MCP form controls, MCP catalog search and scope-select metadata, ModelGate key/pull controls, inline capability connector-token entry, Telegram digest credentials, first-run onboarding profile/workspace/first-task metadata, EraseDataDialog destructive confirmation metadata/focus ring, All Workspaces search, Wiki search, WorkspaceActionsMenu rename/delete confirmations, warm AskBar metadata/focus ring, workspace TasksTab add-task metadata/focus ring, Timeline event-filter metadata/focus ring, Create Workspace visible setup/template/folder-picker metadata and focus rings, Workspace Switcher focus-trap/return evidence, Files storage/browser scroll regions, workspace tab/image semantics, shell landmarks, and Command Center dialog/mobile row fit plus search focus ring with focused tests. T10 remains open for broader unsampled form metadata outside the covered surfaces, unsampled icon-only controls, and modal focus-return evidence.
|
||||
- 2026-07-10 T10 label follow-up: opacity-diluted uppercase labels in the principal Home, Agents, Marketplace, Artifact, Launcher, Login Briefing, Persona Switcher, WaggleDance, and MCP surfaces now use guaranteed semantic contrast tokens. `npm run ux:color-guard` reports 96 offenses, all frozen baseline entries, with no new violations.
|
||||
- 2026-07-10 Pillar 2 return-path follow-up: Connectors, MCP Hub, Artifact Center, Approvals, and Launcher preserve resolved session data during route remount/revalidation; Artifact keys include workspace/search/kind/status. The combined cache lane passes 8 files / 71 tests. Non-main packaged and deployed gates remain open.
|
||||
- 2026-07-10 onboarding affordance follow-up: inactive Who Are You chips now use a visible `--line-affordance` border, `--text-tertiary`, and hover/focus affordances instead of relying on color alone; the focused onboarding lane passes 14/14.
|
||||
- 2026-07-11 model setup recovery follow-up: the shared onboarding/Settings `ModelGate` now exposes provider-catalog failures with service-aware copy and a Retry action. Focused ModelGate + onboarding coverage passes 27/27 and web typecheck passes; the desktop service/update event bridge was re-audited and is already wired with binding coverage.
|
||||
- 2026-07-11 API-key setup follow-up: fresh-port Playwright coverage passes 2/2 for onboarding and Settings provider-key entry, including provider selection, key-field focus, validation, canonical `/api/settings` payload capture, saved feedback, and onboarding Continue enablement. Live provider acceptance still requires a real credential and network.
|
||||
- 2026-07-11 provider model catalog follow-up: built-in provider model inventories were removed from the server and web registries. Configured provider APIs now supply the full returned model list, normalized as stable `provider/model` ids and merged into Chat/Spawn; outage responses retain a visibly stale last-known catalog. Claude/Gemini pagination, hot-session focus refresh, exact-id router regeneration, default/model/fleet routing, and completion are now deterministic regression gates. A paid live credential request remains external release evidence only.
|
||||
- 2026-07-11 Create Workspace disclosure follow-up: optional agent assignment is collapsed by default on the 390 x 844 primary flow, expands through a named control, preserves selected template/persona context in its summary, and resets after a cancelled attempt. Focused shell-overlay contracts pass 15/15 and the rendered `J-mobile` journey passes 1/1 on fresh port `34380`; broader modal-state and focus-return evidence remains open.
|
||||
- T5 visual classification refreshed: `docs/audits/2026-07-08-visual-t5-classification.md` retains the original port 3463 failure classification, then records the current port `34199` verification where `tests/visual/views.spec.ts` passes 14/14. Canonical ASCII-hyphen baselines are active, volatile Home text is masked in the visual spec, and duplicate baseline families are documented for later test-readiness cleanup.
|
||||
- First-run onboarding T1/T2/T12 analysis added and refreshed: `docs/audits/2026-07-08-first-run-onboarding-t1-t2-t12-analysis.md` records a clean-data built-app smoke on port 3431 without skip flags. Desktop onboarding completed through first-task auto-send into workspace chat; the original lane reproduced T1 Clerk/CSP console errors, found the 390 x 844 Profile Continue action below the viewport, found high-volume Claude Code import too prominent for day-zero setup, and noted model/auto-send handoff polish issues. Current focused verification closes T1 console health for the sampled accountless paths (3/3 on port `34196`) and mobile Profile reachability (1/1 on port `34194`).
|
||||
- Source inventory supplement added: `docs/audits/2026-07-08-source-inventory-consistency-audit.md` confirms production shell routes are represented, identifies command-query evidence for `/launcher?watch=1` and `/settings?tab=billing`, records current `apps/` and 28-package inventory, and adds T19 for the Browser Companion extension.
|
||||
- Browser Companion T19 analysis added and refreshed: `docs/audits/2026-07-08-browser-companion-t19-analysis.md` records passing extension syntax/manifest/server checks, disconnected and connected-tab popup screenshots, direct sidecar save success, content-script extraction from a normal page, legacy-trust extracted save success, Memory UI confirmation, pre-fix unallowlisted-extension CORS and `401 MISSING_TOKEN` failures, plus a 2026-07-09 fix for secure token bootstrap, concrete extension-origin CORS matching, MV3 service-worker no-Origin handling, background save auth, sticky accessible popup recovery, restricted-page recovery copy, disabled-state styling, honest memory-destination copy, secure-default loaded-extension save/frame confirmation, popup keyboard/focus/Enter save proof, direct popup Save page click evidence, rendered Memory UI confirmation after the secure default path, context-menu handler coverage, stable packaged-ID pairing proof, Memory search provenance consistency for imported captures, and existing chat `auto_recall`/catch-up imported provenance. T19 remains open for native toolbar-bubble proof or manual release evidence, native context-menu click proof or deferral, signed Web Store/installer-distributed extension proof if release packaging is scored, and separate proof for any future recall result shape if scored.
|
||||
- Desktop wrapper T14 analysis added and refreshed: `docs/audits/2026-07-08-desktop-wrapper-t14-analysis.md` records passing app TypeScript, Rust `cargo check`, sidecar resource preflight, app service E2E, app helper/static tests, web-side Tauri binding tests, release workflow hardening, and tray source hardening. T14 remains open for packaged tray/close/shortcut evidence, packaged watchdog/update presentation, installer/signing proof, root command-shape gap for `apps/web` Tauri tests, and absent packaged interaction evidence.
|
||||
|
||||
- T11 focused analysis refreshed: `docs/audits/2026-07-08-route-evidence-t11-analysis.md` records the AppShell route registry, app-local route table tests 29/29, Benchmark/Platform/Payment component tests 17/17, all-route built-preview smoke on port 3457, and codified `J-route-coverage` tests passing 2/2 on port `34200`. T11 is now closed for route-existence ownership; remaining route-adjacent issues are tracked under T9/T10/T12/T16.
|
||||
- T6 marketplace determinism fix verified: `POST /api/marketplace/sync` now shares the marketplace-sync disable gate used by the background scheduler, so `WAGGLE_DISABLE_MARKETPLACE_SYNC=1` returns `skipped: true` without touching external sources. Focused Vitest passed 3/3 across manual-sync and background-sync contracts; `packages/server` typecheck passed; focused Playwright marketplace verification passed 4/4 on port `34203` after the T17 `tsx`/esbuild startup fix.
|
||||
- P1-17 high-volume import fix verified: first-run Claude Code detections above 1,000 items now default to `Review after setup`, format the count, and make immediate import explicit; `onboarding-import-step.test.tsx` passed 2/2 and `npm run typecheck:web` passed.
|
||||
- P1-18 first-run handoff fix verified: ready model state hides the stale setup/checking gate, and auto-sent first-task seeds clear immediately while pending; `ModelGateStep.test.tsx` passed 5/5, `lane-c-input-power.test.tsx` passed 12/12, the onboarding focused bundle passed 22/22, and `npm run typecheck:web` passed.
|
||||
- T12 focused analysis added and refreshed: `docs/audits/2026-07-08-state-failure-t12-analysis.md` records a passing focused state slice (9 files / 79 tests), existing chat SSE failure-injection source coverage, browser-native dialog line evidence, warning-noise risks, five-persona state-bundle fields, and T12-A through T12-G correction candidates. The current five-persona bundle passes 5/5 with retries disabled on port `34301`, with route, sampled failure/workflow/scale/slow-data, and overlay evidence plus Mobile Executive Notification Inbox, Command Center, and local model runtime-unavailable screenshots, Researcher Memory unavailable, Memory slow-list, Memory large-list, and Timeline/Event large-list evidence, Engineer Agents slow-list, Agents large-list, Marketplace unavailable, Marketplace large-catalog, Cockpit health-degraded, Files upload-failure, Files upload-success, and Files large-list evidence, and Team Admin billing checkout-unavailable, Team active billing state, Team settings unlocked state, billing checkout success return, billing checkout cancel return, backup creation failure, and backup restore-failure evidence. Failed uploads show branded recovery copy and do not render the failed filename as a successful row; successful uploads return `/successful-upload.md` and render the uploaded file in the Files UI; the files large-list probe renders 240 mocked files with `240 items`; the Marketplace large-catalog probe renders a 240-entry mocked catalog with grouped `Skills 120`, `Connectors 60`, and `MCPs 60`; the Memory slow-list probe delays `/api/memory` by 2 seconds, proves the aria-busy `Loading memories` status, and then renders 40 mocked memories with `40 memories` and `Bulk Memory 000`; the Memory large-list probe renders 200 mocked memories with `200 memories` and `Bulk Memory 000`; the Agents slow-list probe delays `/api/agents` by 2 seconds, proves the aria-busy `Loading agents` status, and then renders 40 mocked agents with `40 agents` and `Bulk Agent 000`; the Agents large-list probe renders 180 mocked agents with `180 agents` and `Bulk Agent 000`; the Team Admin active billing state mocks `/api/tier` to `TEAMS` and renders `Waggle Team`, `$49/mo per seat`, and `Manage subscription`; the Team settings unlocked state renders `Team Server URL`, `Auth Token`, and the team-server trust warning; the Team Admin checkout success return mocks `/api/stripe/sync` to `TEAMS` and renders the Team confirmation; the checkout cancel return redirects to Billing with `Checkout was cancelled` and `No charge was made`; the Timeline/Event large-list probe renders 360 mocked events with `360 events` and `Used bulk_tool_000`. The run reports 0 critical console/page/network failures and 0 visible overflow.
|
||||
- Mobile Executive T2/T12 analysis added and refreshed: `docs/audits/2026-07-08-mobile-executive-t2-t12-analysis.md` records the original 390 x 844 built-app smoke on port 3419 with screenshots for Home, Settings general/models/billing/profile, Memory, workspace chat, Command Center, and Workspace Switcher. The original pass kept T2 open because Settings controls visibly clipped despite clean document scroll width. Current focused verification closes P0-2: `J-mobile: Settings is usable at 390px width` passed 1/1 on port `34195`.
|
||||
- T13 focused analysis added and refreshed: `docs/audits/2026-07-08-launch-funnel-t13-analysis.md` records passing www test/typecheck/build evidence, localhost route/API smoke, rendered Browser evidence under `output/playwright/www-t13-3491/`, a current external refresh, and local checkout/legal/download/deploy-workflow fixes. T13 remains a launch P0 because the canonical `waggle-os.ai` and `www.waggle-os.ai` domains do not resolve from the audit environment, real signed installers are not published, Vercel production deploy/DNS smoke is not proved, real deployed Clerk/Stripe checkout success/cancel evidence is still missing, and legal sign-off is not proved. Locally fixed: signed-out Team CTA now uses GET auth continuation, cancelled checkout returns to `/?checkout=cancelled#pricing` with a retry notice, the intentional `html.js` hydration mismatch no longer raises the Next dev issue badge, legal Day-0/stale Pro placeholder copy is guarded, public Download uses `/download` instead of an empty release target, mobile/tablet visitors keep a generic Download label, and `deploy-www.yml` targets Vercel prebuilt deployment instead of GitHub Pages/static `apps/www/dist`.
|
||||
- Shell Overlay T10/T12 analysis added and partially updated: `docs/audits/2026-07-08-shell-overlays-t10-t12-analysis.md` records the original built-app overlay smoke on port 3437 plus the Phase 2 partial fixes. Notification Inbox and Create Workspace primary/subdialog contracts now have codified dialog/close evidence, custom-template delete no longer uses native `confirm()`, Context Rail has a labelled complementary contract, Onboarding Tooltips has an explicit non-modal dialog/Escape contract, tier modal close labels are named, and the sampled 390 x 844 Create Workspace hierarchy passes on port `34157`; the expanded user-journey suite passed 19/19 on port `34158`. The five-persona bundle now refreshes Mobile Executive Notification Inbox and Command Center overlay screenshots, and `NotificationInbox` avoids horizontal opening motion that caused transient mobile overflow. Route-specific overlay state evidence for less common overlay states remains open.
|
||||
- T7 trust-dialog update: Approvals revoke-all now uses the shared in-app `ApprovalModal`; `p7-b1-approvals-error.test.tsx` covers the no-native-confirm contract, rendered `J3d` passes on port `34159`, and the expanded user-journey suite passed 20/20 on port `34160` before the Artifact `J3e` addition.
|
||||
- T7 trust-dialog update: Artifact permanent delete now uses the shared in-app `ApprovalModal`; `artifact-center-trust.test.tsx` covers the no-native-confirm contract, rendered `J3e` passes on port `34161`, and the expanded user-journey suite passes 21/21 on port `34162`.
|
||||
- T7 trust-dialog update: Memory Center permanent delete, GDPR erase, and allow re-import now use the shared in-app `ApprovalModal`; `memory-center-trust.test.tsx` covers all three no-native-confirm contracts, rendered `J3f` passes on port `34164`, and the expanded user-journey suite passes 22/22 on port `34165`.
|
||||
- T7/T10 trust-dialog and accessibility update: Wiki Obsidian and Notion exports now use in-app form dialogs; `wiki-export-trust.test.tsx` covers both no-native-prompt contracts; rendered `J3g` passes on port `34167`; and the path remains included in the latest 24/24 user-journey suite on port `34177`.
|
||||
- T14 is narrowed, not closed: `app` TypeScript, static updater/installer/signing/runtime tests, static Tauri config/update/tray tests, sidecar resource preflight, web typecheck, Rust `cargo check`, service-level startup/chat/workspace E2E, the release package-before-sidecar guard, focused web desktop-navigation/shell-event binding tests, updater-disabled static contracts, Rust sidecar path unit tests, and Tauri webview-origin CORS tests pass. Native source directly handles tray Open/focus, close-to-tray, `Ctrl+Shift+W`, and Quit; Settings routes through a tested `/settings` desktop bridge; Pause and About are hidden. Packaged debug Tauri startup now launches, starts bundled `resources/service.js`, and reaches `/health` without CORS/module/ABI errors. MSI/installer packaging, packaged tray interactions, close-to-tray, global shortcut, forced watchdog UI, installer trust, and signed-update proof still need installed-app evidence.
|
||||
- T15 deep-dive updated: `docs/audits/2026-07-08-admin-cli-utility-t15-analysis.md` records passing admin-web, launcher package-local tests, `@waggle/cli` package-local tests, memory MCP package-local tests, hive-mind MCP package-local tests, marketplace targeted tests, hive-mind CLI package-local tests, no-emit TypeScript, package build scripts, and admin-web rendered package evidence. It also records focused marketplace, launcher, `@waggle/cli`, memory MCP, hive-mind MCP, hive-mind CLI, and admin-web fixes: marketplace NodeNext emitted imports, built help, source/built/installed invalid-command smokes pass without clean-home DB side effects; marketplace publish manifest/packed-file alignment and installed-bin help are guarded; launcher source/built help and invalid-port paths pass without service banners or `.waggle` side effects; built and installed launcher occupied-port recovery prints a concrete `npx waggle --port <next-port>` command; launcher packed tarball first-command help runs without service setup; clean installed launcher startup serves `/health`, prints manual-open copy, and creates the configured data dir; `@waggle/cli` built, bin-wrapper, packed-bin, local package-closure installed `npx` help, local package-closure installed REPL startup/slash-command/exit, and local package-closure installed streamed chat/provider plumbing against a mock LiteLLM-compatible endpoint pass, with `@waggle/agent`/`@waggle/weaver` package metadata corrected to built `dist` entries and `@waggle/agent` runtime dependency declarations fixed; legacy memory MCP and hive-mind MCP built write-scope roundtrips save and recall unique memories; legacy memory MCP and hive-mind MCP local package-closure installed read-only startup lists tools; hive-mind CLI source/built/local package-closure installed sampled subcommand help prints focused help without creating `personal.mind`; the installed hive-mind CLI proof found and fixed the missing `@waggle/shared` dependency in `@waggle/hive-mind-core`; the installed memory MCP proof found and fixed the missing `glob` dependency in `@waggle/core`; admin-web package-local tests pass without React `act(...)` warnings; and built-preview Playwright evidence proves all seven admin pages at desktop and mobile widths, hash deep links, browser back/forward traversal, `aria-current`, labelled table scroll regions, labelled rendered controls, clean app console/pageerror collection, mobile shell keyboard reachability, page-level keyboard traversal from connection fields into the covered admin pages, desktop/mobile visual snapshots, capability governance forms, malformed analytics response recovery without blanking the shell, all-page initial API-failure recovery with accessible alerts, mutation/destructive-failure recovery for capability policy save, capability override create/remove, capability request decision, member invite, member role change, member removal, and team settings save, plus local bearer-auth wrong-token/valid-token behavior through protected Fastify routes. T15 remains open because registry-only proof after internal package publication remains open. The Members native-confirm blocker is fixed in focused component coverage.
|
||||
- T16 deep-dive updated: `docs/audits/2026-07-08-ai-tool-hook-t16-analysis.md` records passing shared/agent/server route tests, Launcher/prompt/adapter tests, root-run hook package tests, hook/shim package-local scripts, official package typechecks, Claude Desktop stub build, compiled hook-bin help smokes, packed-package `npx` install/verify/uninstall lifecycle for all six hook-capable packages, focused no-reconnect observed-output regression coverage, hook stdout/stderr/Backup-Recovery/Check-failed/uninstall-cleanup/More-output/structured-failure/empty-output recovery/Claude Desktop launch-only coverage, focused registry-aware third-party adapter launch coverage, partial rendered Launcher Browser evidence under `output/playwright/launcher-t16-54147/`, a fresh real Browser `/launcher` smoke where Verify now renders `verify failed (exit 1)` with retry/uninstall/reinstall guidance instead of `HTTP 400`, Browser-rendered mocked Claude Desktop, Codex Verify manual-approval, and Codex Uninstall cleanup states, codified Playwright rendered standard install changed-file/pointer/backup/recovery labels, sidecar-offline Retry, long-stderr summarization, all six hook-capable install/verify/uninstall transitions, and non-built-in adapter launch-only/prompt behavior, gated real-tool Playwright evidence where OpenClaw renders as a real detected CLI and launches through the sidecar with observed output, exit 0, and process cleanup, gated route-level evidence where all six hook-capable tools install/verify/uninstall through `/api/tools/hooks` against an isolated profile and clean up, Codex WindowsApps recovery evidence where the restricted app alias is installed-but-not-launchable with no Launch button, Tailwind motion-token warning cleanup evidence, and `shape-selection.ts` dynamic/static import warning cleanup evidence. It keeps T16 open for packaged desktop hook-status evidence and remaining noisy test output.
|
||||
- T17 deep-dive updated: `docs/audits/2026-07-08-developer-substrate-t17-analysis.md` records 13/13 direct no-emit typechecks, passing agent/core/optimizer/weaver package tests, passing root-run substrate/SDK/shared/WaggleDance/worker/compiler tests, a dedicated server performance lane passing 13/13, a named server release lane passing 185 files / 2128 tests with one worker, hermetic marketplace-sync tests, and package-local test commands now passing for hive-mind-core (59/745), wiki-compiler (2/25), sdk (5/89), worker (4/46), WaggleDance (3/42), shared (5/40), and hive-mind-wiki-compiler (3/26). It keeps T17 open for warning hygiene and incomplete developer recovery-journey evidence.
|
||||
- T18 deep-dive added and partially refreshed: `docs/audits/2026-07-08-ops-deploy-ci-judging-t18-analysis.md` records passing YAML parse, safe Compose scans, ignored env-file tracking checks, benchmark harness TypeScript and package-local tests (29/325), production Compose fail-closed credential guards, explicit Render sidecar-mode guards, root-run benchmark tests, and a focused public-site workflow fix from GitHub Pages/static artifact upload to Vercel prebuilt deployment. It keeps T18 open for missing deployed Vercel/DNS smoke, non-blocking CI E2E, no CI infra lane, unavailable local Docker engine, LiteLLM live-routing gap, and stale historical judging artifacts.
|
||||
|
||||
## Phase Approval Boundaries
|
||||
|
||||
Phase 1 approval includes only:
|
||||
|
||||
- T1 local auth, Clerk, and CSP console health.
|
||||
- T2 follow-through for mobile Settings and first-run onboarding is now verified fixed in focused 390px journeys; keep broader mobile polish evidence for Memory/chat tabs and selected overlays.
|
||||
- T3 Solo/Teams/Enterprise copy cleanup.
|
||||
- T4 `Ctrl+Shift+N` and Workspace Switcher route contract are verified fixed in focused runs; keep them in regression.
|
||||
- T5 visual snapshot lane is verified fixed for the seven tracked desktop views; keep broader visual scope in the route/scenario manifest and T12 evidence.
|
||||
- T11 route evidence for thin judge paths is verified fixed; keep deeper state/action coverage in T12 and launcher/runtime proof in T16.
|
||||
|
||||
T12 state/failure bundle evidence is not part of Phase 1 implementation unless Phase 1 verification directly needs it. It becomes a Phase 2 judge-readiness gate.
|
||||
|
||||
T13 launch-funnel evidence, T14 desktop-wrapper evidence, T15 utility evidence, T16 hook lifecycle evidence, T17 developer API/background/substrate verification evidence, T18 ops/deployment/CI/benchmark/judging evidence, and T19 Browser Companion evidence are not part of Phase 1 implementation unless Phase 1 verification directly needs a minimal supporting fix. They become final-product gates after the in-app P0 blockers are cleared.
|
||||
|
||||
Phase 1 approval does not include broad redesign, new surfaces, native dialog replacement across the whole app, performance chunk splitting, launch-site deployment/checkout/legal work, desktop wrapper/install/update work, admin/CLI/MCP utility work, AI-tool hook lifecycle work, developer API/background/substrate tooling work, ops/deployment/CI/benchmark/judging work, Browser Companion extension work, or local model pricing semantics unless a Phase 1 verification command directly requires a minimal supporting fix.
|
||||
|
||||
Post-Phase-1 sequencing lives in `docs/audits/2026-07-08-ux-post-phase-1-roadmap.md`.
|
||||
|
||||
## Judge Blocking Rules
|
||||
|
||||
- Any open P0, including launch-scoped P0s unless explicitly deferred from the final score, blocks the final five-persona 9/10 claim.
|
||||
- Any open P1 that touches a persona's primary journey must be fixed or explicitly deferred before scoring that persona.
|
||||
- Any missing T12 state bundle blocks the scorecard for the persona whose primary journey depends on that state.
|
||||
- Missing T13/T14/T15/T16/T17/T18/T19 evidence blocks a full "complete UX" claim unless the user explicitly scopes those gates out of the five-persona score.
|
||||
- P2 items do not block scoring by themselves, but must not contradict the implementation or reintroduce wrong copy in touched files.
|
||||
- A green test is not enough unless it covers the route, viewport, and user state named in the finding.
|
||||
|
||||
## Current Recommendation
|
||||
|
||||
Phase 1 is implemented and verified. Continue Phase 2 in focused slices; the overlay-contract slices now include Create Workspace mobile hierarchy/subdialogs, while screenshot refresh plus the broader trust/accessibility/native-dialog backlog stay open.
|
||||
|
||||
Decision brief: `docs/audits/2026-07-08-ux-approval-brief.md`.
|
||||
438
docs/audits/2026-07-08-ux-non-main-surface-scope.md
Normal file
438
docs/audits/2026-07-08-ux-non-main-surface-scope.md
Normal file
@@ -0,0 +1,438 @@
|
||||
# Non-Main UX Surface Scope
|
||||
|
||||
Status: analysis only. No product code was changed.
|
||||
|
||||
Purpose: the main audit already covers the installed cockpit experience in `apps/web`. This note prevents "complete UX" from silently excluding the public launch funnel, Browser Companion extension, Tauri desktop wrapper, installer/update path, active local service startup path, admin web surface, CLI launchers, marketplace CLI, memory MCP utilities, AI-tool hooks, developer APIs, background jobs, substrate package verification, deployment/ops configuration, CI, benchmarks, and judging artifacts.
|
||||
|
||||
## Scope Decision
|
||||
|
||||
The five-persona 9/10 judge gate should primarily score the installed product experience: `apps/web` rendered through the local service, because the requested personas are operating the product.
|
||||
|
||||
The final "complete UX" claim still needs one of these explicit decisions:
|
||||
|
||||
1. Fix and verify the non-main launch/desktop/utility/hook/developer/ops gates before final scoring.
|
||||
2. Approve a written deferral that scopes the five-persona gate to the installed app only.
|
||||
|
||||
Without one of those decisions, public download, checkout, legal trust, browser capture, installer, tray, update, sidecar-startup, admin, CLI, MCP utility, hook lifecycle, SDK, server API, worker, substrate verification, deployment, CI, benchmark, and judging failures could remain outside the score while still affecting real users.
|
||||
|
||||
## Public Launch Funnel: `apps/www`
|
||||
|
||||
User jobs:
|
||||
|
||||
- Understand what Waggle is.
|
||||
- Download the desktop app.
|
||||
- Sign in or sign up.
|
||||
- Start Team checkout.
|
||||
- Recover from cancelled checkout.
|
||||
- Read methodology, legal, privacy, cookies, and EU AI Act trust pages.
|
||||
- Reach KVARK for Enterprise.
|
||||
|
||||
Current strengths:
|
||||
|
||||
- Homepage is structured as a coherent launch narrative: hero, problem, how it works, memory, proof, features, trust, personas, open source, pricing, final CTA.
|
||||
- Landing page has a skip link, semantic sections, absolute nav anchors, mobile menu semantics, and OS-aware download CTA.
|
||||
- Pricing is mostly aligned to Solo, Team, Enterprise.
|
||||
- Historical Lighthouse note reports 96 performance, 96 accessibility, and 100 SEO on a local prod build.
|
||||
- `BrandPersonasCard` has useful component coverage, including keyboard activation and image-failure fallback.
|
||||
|
||||
Current command evidence, 2026-07-08 continuation:
|
||||
|
||||
| Check | Result | Interpretation |
|
||||
|---|---:|---|
|
||||
| `npm run test -w apps/www` | Pass, 1 file / 10 tests | The existing component coverage is healthy, but still narrow. |
|
||||
| `npm run build:www` | Pass | Next build succeeds and reports dynamic routes for account, Clerk auth, Stripe checkout, and Stripe webhook. |
|
||||
| `npx tsc --noEmit --project apps/www/tsconfig.json` | Pass | Direct TypeScript check passes for the public site. |
|
||||
| `Test-Path apps/www/dist`, `Test-Path apps/www/.next`, `Test-Path apps/www/out` | `False`, `True`, `False` | The build output is `.next`, not the `apps/www/dist` artifact uploaded by the current GitHub Pages workflow. |
|
||||
| Env-shape check for Clerk/Stripe keys | Mixed | `apps/www/.env.local` has Clerk test keys that differ from the root Clerk pair, and a live-shaped Stripe secret. Values were not printed. |
|
||||
| `next start` bound to `127.0.0.1` + bounded route/API smoke | Invalid harness / fail | Next middleware attempted to proxy to `localhost:<port>` and returned timeouts/500s. Re-running against `--hostname localhost` showed the core public routes can render, so the `127.0.0.1` result is a Windows host-binding trap, not by itself product behavior. |
|
||||
| `next start` bound to `localhost` + status/browser smoke | Mixed | `/`, `/#pricing`, `/privacy`, `/terms`, `/cookies`, `/eu-ai-act`, `/sign-in`, `/sign-up`, `/docs/methodology`, and unauthenticated `/account` redirect render; `/pricing?checkout=cancelled` is 404; signed-out `Get Team` leaves a small inline `Sign in required` error; browser console/server log still show Clerk development-key and session-loop warnings. |
|
||||
| Focused T13 continuation smoke | Mixed | Fresh localhost route/API smoke confirms core pages and signed-out GET checkout redirect work; `/pricing?checkout=cancelled` and `/methodology` return 404; signed-out POST checkout returns `401` with a `signInUrl` the pricing UI does not use. |
|
||||
| Live canonical-domain smoke | Fail | Current external refresh on 2026-07-08 could not resolve `waggle-os.ai`; `nslookup waggle-os.ai` returned non-existent domain from the audit environment. |
|
||||
| GitHub Releases latest target | Fail | The public download CTA points to GitHub Releases latest, but GitHub currently reports no releases for the repo; API refresh returned an empty releases list. |
|
||||
|
||||
Open UX risks:
|
||||
|
||||
| ID | Risk | Evidence | Why it matters |
|
||||
|---|---|---|---|
|
||||
| WWW-0 | The standard launch-funnel smoke is host-sensitive and Clerk-noisy. | Binding `next start` to `127.0.0.1` caused Next middleware proxy failures to `localhost:<port>` on Windows; binding to `localhost` rendered core routes, but the browser/server still emitted Clerk development-key and session-loop warnings. | A release gate needs a stable, documented smoke command and zero auth-loop noise before buyers or reviewers use the public site. |
|
||||
| WWW-0A | The canonical public domain does not currently resolve. | Current external refresh could not resolve `waggle-os.ai`; the app metadata, sitemap, robots, and methodology page declare `https://waggle-os.ai` as canonical. | The acquisition, pricing, legal, auth, account, and download funnel is unreachable at the URL the app declares canonical. |
|
||||
| WWW-1 | Signed-out Team checkout can dead-end as a JSON error. | `Pricing.tsx` calls `POST /api/stripe/checkout`; the route returns `401 { message: "Sign in required", signInUrl }`; the component only shows the message and ignores `signInUrl`. | A buyer clicking "Get Team" before signing in should be guided into sign-in/sign-up and then checkout, not shown an implementation-shaped error. |
|
||||
| WWW-2 | Stripe cancel recovery points to a likely dead route. | Checkout route uses `cancel_url: ${origin}/pricing?checkout=cancelled`; the site has a pricing section on `/`, not an `app/pricing/page.tsx` route. | Cancelled checkout should return to pricing with clear recovery, not a 404 or unrelated page. |
|
||||
| WWW-3 | GitHub Pages deployment workflow likely cannot publish this Next app as written. | `deploy-www.yml` uploads `apps/www/dist`; `npm run build:www` produced `.next`, not `dist` or `out`, and the app includes dynamic Clerk and Stripe API routes. | A strong local landing page still fails the launch funnel if the deploy target cannot serve it. |
|
||||
| WWW-4 | Legal trust pages are visibly unfinished and contain stale tier copy. | Terms/privacy/cookies/EU AI Act pages say "Day-0 placeholder"; privacy says "upgrade to Pro or Teams"; launch-date/address placeholders remain. | Legal/trust copy is part of UX for founders, admins, and enterprise reviewers. |
|
||||
| WWW-5 | Coverage is thin for funnel flows. | Only `BrandPersonasCard.test.tsx` was found under `apps/www/__tests__`; no route E2E, checkout recovery, account, mobile nav, or fresh Lighthouse evidence was collected in this packet. | One component test cannot justify a complete launch UX claim. |
|
||||
| WWW-6 | Public download currently leads to no installable artifact. | `DownloadCTA` targets GitHub Releases latest, but the repository currently has no releases; OS detection can also label iPhone/iPad as macOS and unsupported desktops as Linux while public copy says Windows and macOS. | Download is the first conversion path for Solo founders and mobile evaluators; an empty target makes the product look unavailable. |
|
||||
|
||||
Recommended ticket: T13, Launch Funnel UX Gate. Focused supplement: `docs/audits/2026-07-08-launch-funnel-t13-analysis.md`.
|
||||
|
||||
Suggested acceptance:
|
||||
|
||||
- `apps/www` build/deploy target is valid for the actual app shape, or deployment is moved to an appropriate Next host.
|
||||
- Canonical `https://waggle-os.ai/` resolves and serves the selected public-site deployment.
|
||||
- Download CTA leads to valid Windows/macOS artifacts or a controlled download/status page; mobile and unsupported OS labels are honest.
|
||||
- Local production route smoke uses the correct host binding and serves `/`, `/#pricing`, `/docs/methodology`, `/privacy`, `/terms`, `/cookies`, `/eu-ai-act`, `/sign-in`, `/sign-up`, unauthenticated `/account` redirect, and checkout recovery without 500s, timeouts, 404s, or Clerk redirect-loop spam.
|
||||
- Homepage route smoke passes on desktop and mobile with no critical console errors.
|
||||
- Mobile nav, download CTA, billing toggle, signed-out Team checkout, signed-in Team checkout stub/live lane, checkout cancel recovery, account redirect, sign-in, and sign-up are exercised.
|
||||
- Legal pages have no placeholder launch dates/addresses and no active Pro copy except explicit legacy billing context.
|
||||
- Fresh Lighthouse or equivalent accessibility/performance/SEO evidence is attached.
|
||||
|
||||
## Browser Companion Extension: `apps/browser-ext`
|
||||
|
||||
User jobs:
|
||||
|
||||
- Save selected text from any web page into Waggle memory.
|
||||
- Save the current page into Waggle memory.
|
||||
- See whether Waggle desktop is reachable.
|
||||
- Understand which workspace or personal memory receives the save.
|
||||
- Recover from CORS/sidecar disconnected states.
|
||||
|
||||
Current strengths:
|
||||
|
||||
- The folder is intentionally documented as "Waggle Companion", a Chrome MV3 extension.
|
||||
- Manifest is simple and local-first: popup, background service worker, content script, context menu, active tab, storage.
|
||||
- Popup has connected/disconnected status, workspace label, save-selection/save-page actions, and an Open Waggle button.
|
||||
- Background script talks to the local sidecar; popup code uses `textContent` for workspace names rather than `innerHTML`.
|
||||
- Server has dedicated `GET /api/browser-ext/session-token` and `GET /api/browser-ext/health` endpoints, plus CORS/config checks for extension IDs.
|
||||
|
||||
Current source evidence, 2026-07-08 continuation:
|
||||
|
||||
| Check | Result | Interpretation |
|
||||
|---|---:|---|
|
||||
| `Get-ChildItem apps/browser-ext -Recurse -File` | Pass | Extension files are present: `manifest.json`, `popup.html`, `popup.js`, `background.js`, `content.js`, `README.md`. |
|
||||
| Source search for `browser-ext` and extension env vars | Pass | README, server route, CORS config, backend docs, and settings coverage card all reference the extension. |
|
||||
| `packages/server/src/local/routes/browser-ext.ts` inspection | Pass | Token endpoint bootstraps allowlisted Browser Companion requests; health endpoint returns `{ ok, version, activeWorkspaceId, activeWorkspace }` with `activeWorkspace` kept for legacy extension builds; save flows reuse `/api/memory/frames`. |
|
||||
| Extension JS syntax, manifest parse, and server typecheck | Pass | Root extension files parse and the server side of the contract typechecks. |
|
||||
| Playwright disconnected-state smoke | Partial | Unpacked extension loads and disconnected popup renders; screenshot at `output/playwright/browser-companion-disconnected-state.png`. This does not prove toolbar-popup-over-page behavior or save success. |
|
||||
| Secure-default loaded-extension save smoke | Pass with known gaps | `output/playwright/browser-companion-toolbar-3333/run-secure-default-live-smoke.mjs` proves content-script extraction, MV3 service-worker no-Origin handling, token bootstrap during save, background save through `chrome.runtime.sendMessage`, and `/api/memory/frames` imported-frame confirmation. `run-popup-button-click-live-smoke.mjs` proves popup Tab order, visible Save selection focus, Enter-to-save selection, Save page click, frame creation through the secure sidecar path with an active-tab shim, `/api/memory/search` `source: import`, rendered `/memory` visibility with the `imported` provenance chip when `WAGGLE_T19_WEB_URL` is supplied, and a restricted-page popup state with non-primary disabled styling plus normal-webpage recovery copy. `run-packaged-id-pairing-smoke.mjs` proves stable extension-ID sidecar pairing. `orchestrator-recall-hardening.test.ts` proves existing chat `auto_recall`/catch-up imported provenance. Native toolbar-bubble exposure, native context-menu click proof, signed release-package proof if scored, and any future scored recall result shape remain open. |
|
||||
|
||||
Open UX risks:
|
||||
|
||||
| ID | Risk | Evidence | Why it matters |
|
||||
|---|---|---|---|
|
||||
| EXT-1 | Browser Companion is user-facing and must stay in the judge/evidence packet. | `apps/browser-ext/README.md` presents it as a workflow surface for researcher/journalist/marketer/writer users. T19 now records loaded-extension and popup save evidence, but native toolbar/context proof remains open. | A complete UX claim should not omit a capture surface that feeds the memory moat. |
|
||||
| EXT-2 | Connected/disconnected/CORS-denied states are only partly visually proved. | Disconnected and connected-tab screenshots exist; focused code maps setup denial to recovery copy. Live CORS-denied screenshot is still optional evidence. | The first user experience is likely "why is this not connected?" if the sidecar or extension ID is not configured. |
|
||||
| EXT-3 | Save-to-memory success is end-to-end verified for the background path and direct popup keyboard/click path, but not the native toolbar bubble. | Secure-default smoke saves extracted selected text and confirms `/api/memory/frames`; popup smoke tabs through the action order, saves selection via Enter, clicks Save page, and confirms both frames. Playwright still cannot observe the native toolbar popup as a page. | The extension's core promise is a memory ingestion path, but final release evidence still needs native toolbar-bubble proof or a deliberate deferral. |
|
||||
| EXT-4 | Popup accessibility and polish are partly fixed but not fully reviewed. | Toast now has `role="status"`/`aria-live`; restricted-page and setup errors are sticky; keyboard Tab order, focus ring, Enter-to-save behavior, and restricted-page disabled styling have live evidence. | A small popup can still fail status announcement if that behavior is scored without a screen-reader proof pass. |
|
||||
| EXT-5 | Coverage Compass claimed browser extensions were covered before native toolbar/context behavior was proved. | `CoverageCompassCard.tsx` now marks Browser AI extensions as `partial` and names the popup/capture coverage plus the pending native entry points; secure save-flow smoke exists, but native toolbar-bubble and native context-menu evidence remain open. | Fixed 2026-07-10; keep the partial state until native entry-point proof or an explicit score deferral exists. |
|
||||
| EXT-6 | Disconnected/restricted-page states needed stronger copy and accessibility. | T19 now has sticky setup/restricted recovery, explicit normal-webpage copy, non-primary disabled styling, and honest memory-destination copy. Live CORS-denied screenshot remains optional if scored. | First-run extension failure should feel recoverable and accessible, not like a dead popup. |
|
||||
|
||||
Recommended ticket: T19, Browser Companion Extension UX Gate.
|
||||
|
||||
Suggested acceptance:
|
||||
|
||||
- Load the unpacked extension in Chromium/Chrome with a fresh sidecar and extension origin allowlist.
|
||||
- Verify popup connected and disconnected states.
|
||||
- Verify save selected text and save whole page popup interactions show visible extension feedback and create Memory frames in Waggle; native toolbar-bubble proof can be manual if Playwright cannot expose it.
|
||||
- Verify native context-menu save, or explicitly defer it. The handler itself is regression-covered.
|
||||
- Verify CORS-denied recovery copy tells the user how to configure `WAGGLE_BROWSER_EXT_IDS` or the dev escape hatch.
|
||||
- Verify status copy and screen-reader announcement behavior if scored separately; popup keyboard/focus basics now have live evidence.
|
||||
- Verify `CoverageCompassCard` does not claim browser-extension coverage beyond current evidence.
|
||||
- Add automated smoke evidence or explicitly defer the extension from the five-persona score.
|
||||
|
||||
## Desktop Wrapper, Installer, Tray, Update: `app/`
|
||||
|
||||
User jobs:
|
||||
|
||||
- Install the app.
|
||||
- Launch the app and see the web cockpit with the local service ready.
|
||||
- Close to tray and re-open.
|
||||
- Use tray actions for open, pause agents, settings, about, and quit.
|
||||
- Use global shortcut behavior.
|
||||
- Recover if the sidecar is unhealthy or port 3333 is occupied.
|
||||
- Receive updates when the update channel is intentionally supported.
|
||||
|
||||
Current strengths:
|
||||
|
||||
- Tauri config points the desktop binary at `apps/web/dist` and includes bundle resources, NSIS configuration, window sizing, CSP, and tray metadata.
|
||||
- Release and PR workflows stage sidecar dependencies, native dependencies, and Node runtime before Tauri build.
|
||||
- Static tests cover updater config, signing helpers, installer config helpers, runtime bundling helpers, and API-level startup/chat/workspace flows.
|
||||
- `scripts/check-sidecar-resources.mjs` fails loudly if packaged sidecar runtime artifacts are missing.
|
||||
|
||||
Current command evidence, 2026-07-08 continuation:
|
||||
|
||||
| Check | Result | Interpretation |
|
||||
|---|---:|---|
|
||||
| `npx tsc --noEmit --project app/tsconfig.json` | Pass | Desktop wrapper TypeScript scripts/config compile. |
|
||||
| `npx vitest run app/tests/auto-update.test.ts app/scripts/installer-config.test.ts app/scripts/signing-config.test.ts app/scripts/bundle-runtimes.test.ts` | Pass, 4 files / 85 tests | Static updater, installer, signing, and runtime-bundling helper checks are healthy. |
|
||||
| `npx vitest run packages/server/tests/tauri-config.test.ts app/tests/auto-update.test.ts` | Pass, 2 files / 30 tests | Static Tauri config and update-emitter expectations are healthy. |
|
||||
| `node scripts/check-sidecar-resources.mjs` | Pass | Staged Node runtime, native deps, and sidecar `node_modules` are present locally. |
|
||||
| `cargo check --manifest-path app/src-tauri/Cargo.toml` | Pass | Tauri Rust shell compiles in dev profile. |
|
||||
| `npm run test -- tauri-bindings.test.ts adapter.tauri-branch.test.ts --reporter=dot` from `apps/web` | Pass, 2 files / 19 tests | Web-side Tauri command bindings and adapter Tauri branch tests pass through the correct package-local command. |
|
||||
| `npm run test -w apps/web -- src/lib/tauri-bindings.test.ts --reporter=dot` | Pass, 1 file / 15 tests | Focused desktop navigation binding test covers `/settings` allowlist and unsupported payload filtering. |
|
||||
| `npx vitest run app/tests/e2e/startup.test.ts app/tests/e2e/chat.test.ts app/tests/e2e/workspaces.test.ts` | Pass, 3 files / 11 tests | Local service startup, health, settings persistence, chat SSE, workspace/session, and memory-scope API flows work in the Vitest harness. |
|
||||
|
||||
Native event consumer matrix:
|
||||
|
||||
| Native event / behavior | Source | Current React consumer evidence | UX reading |
|
||||
|---|---|---|---|
|
||||
| Tray icon click / Open Waggle | `tray.rs` handles show/focus directly | Native-handled; no React consumer needed | Source-wired, but still needs packaged smoke evidence. |
|
||||
| Close window to tray | `lib.rs` prevents close and hides the window | Native-handled; no React consumer needed | Source-wired, but still needs target-OS smoke evidence. |
|
||||
| `Ctrl+Shift+W` global shortcut | `lib.rs` toggles window visibility directly | Native-handled; no React consumer needed | Source-wired, but still needs target-OS smoke evidence. |
|
||||
| Tray Settings | `tray.rs` emits `waggle://navigate` to `/settings` after show/focus | `App.tsx` mounts `TauriDesktopEventBridge`; `tauri-bindings.ts` accepts only `/settings` | Source-wired, but still needs packaged smoke evidence. |
|
||||
| Pause Agents tray action | `tray.rs` | Menu item and emitter removed | Hidden until pause/resume has real product behavior. |
|
||||
| About Waggle tray action | `tray.rs` | Menu item and `/about` emitter removed | Hidden until there is a real About destination. |
|
||||
| Tray Quit | `tray.rs` calls `app.exit(0)` | Native-handled; no React consumer needed | Source-wired to Tauri exit cleanup, but still needs packaged smoke evidence. |
|
||||
| `waggle://update-available` | `lib.rs` emit | 0 matching `apps/web/src` listeners; covered only as future-emitter static test | Update availability is not user-visible. |
|
||||
| `waggle://service-status` | `service.rs` emit | 0 matching `apps/web/src` listeners | Failed/restarting service state may not be visible. |
|
||||
| `waggle://service-restart-needed` | `service.rs` emit | 0 matching `apps/web/src` listeners | Restart-needed state may not guide the user. |
|
||||
|
||||
Open UX risks:
|
||||
|
||||
| ID | Risk | Evidence | Why it matters |
|
||||
|---|---|---|---|
|
||||
| DESK-1 | Update/service events still have emitters but no current frontend consumers. | Tray route false affordances are narrowed: Settings has a `/settings` bridge, Quit is native, and Pause/About are hidden. Remaining unconsumed events are `waggle://update-available`, `waggle://service-status`, and `waggle://service-restart-needed`. | Update availability and service recovery may not be visible enough in the installed app. |
|
||||
| DESK-2 | Auto-update UX is intentionally disabled/unfinished for v1. | `auto-update.test.ts` says updater plugin config is intentionally disabled; `release.yml` says former update manifest published empty signatures; Rust still checks and emits `waggle://update-available` for future consumers. | Users need either a working update flow or no misleading update UX surface. |
|
||||
| DESK-3 | Installed-app interaction evidence is missing. | Current app tests start the Fastify service through Vitest, inspect static config, and now pass sidecar resource preflight. Open/focus, close-to-tray, and `Ctrl+Shift+W` are source-wired natively, but this packet did not run a packaged Tauri app, tray menu, global hotkey, installer, or close-to-tray smoke. | API tests and source inspection do not prove install, launch, tray, window, or WebView recovery UX. |
|
||||
| DESK-4 | Sidecar failure and port-conflict recovery are not proven visually. | `startService()` has port-availability errors and watchdog events, but no current packet evidence shows the user-facing UI for service failure/restart-needed states. | A failed local service is a first-run blocker; the app should tell the user what happened and how to recover. |
|
||||
| DESK-5 | Signing/notarization and update-channel trust are release gates, not product-screen tests. | Code-signing docs still describe pilot self-sign vs public cert procurement; release workflow builds draft assets but does not prove trusted install/update UX in this audit packet. | Installer warnings and untrusted binaries are UX for real first-time users. |
|
||||
| DESK-6 | Web-side Tauri binding tests are package-local, not root-discoverable. | Root Vitest excludes `apps/**`; the correct command from `apps/web` passes. | Final verification docs must use the right command, or root verification will look falsely missing. |
|
||||
|
||||
Recommended ticket: T14, Desktop Wrapper and Release UX Gate.
|
||||
|
||||
Suggested acceptance:
|
||||
|
||||
- Packaged app launches on a clean data dir and reaches a healthy Home or clear service-recovery screen.
|
||||
- Tray open/settings/about/quit/pause actions are verified or disabled until wired.
|
||||
- Close-to-tray and global shortcut behavior are verified on the target OS lane.
|
||||
- Sidecar port-conflict and sidecar-crash recovery have user-facing evidence.
|
||||
- Update UX is either wired end to end with signed updater artifacts or explicitly hidden/deferred.
|
||||
- Installer/signing warnings are documented for the target release channel.
|
||||
|
||||
## Active Local Service Path
|
||||
|
||||
The active installed-app sidecar path is not `sidecar/src/main.ts`. The Tauri wrapper builds `packages/server/src/local/service.ts` into `app/src-tauri/resources/service.js` through `scripts/build-sidecar.mjs`, stages external dependencies with `scripts/stage-sidecar-deps.mjs`, and launches that service from `app/src-tauri/src/service.rs`.
|
||||
|
||||
Implication:
|
||||
|
||||
- Judge the `packages/server/src/local/service.ts` startup path for installed desktop UX.
|
||||
- Treat the top-level `sidecar/` JSON-RPC code as legacy or separate tooling unless a current launch path imports it.
|
||||
|
||||
Open service UX requirements:
|
||||
|
||||
- Startup progress should be visible or the first rendered app should clearly recover while the service comes up.
|
||||
- Port conflict should produce a user-facing recovery path.
|
||||
- Pending data erasure should surface a receipt/instruction after restart.
|
||||
- Health should distinguish `ok`, `degraded`, and `unavailable` in user language.
|
||||
|
||||
## Admin Web, CLI, Marketplace CLI, and MCP Utilities
|
||||
|
||||
User jobs:
|
||||
|
||||
- Use `packages/admin-web` to review team dashboard, analytics, members, capability policies, jobs, audit, and team settings.
|
||||
- Use `packages/launcher` as the `npx waggle` entry point.
|
||||
- Use `packages/cli` for command-line Waggle workflows.
|
||||
- Use `packages/marketplace` CLI for package search, install, sync, scan, and audit jobs.
|
||||
- Use `packages/memory-mcp`, `packages/hive-mind-mcp-server`, and `packages/hive-mind-cli` for memory/MCP setup, recall, harvest, maintenance, and machine-readable JSON output.
|
||||
|
||||
Dedicated supplement: `docs/audits/2026-07-08-admin-cli-utility-t15-analysis.md`.
|
||||
|
||||
Current command evidence, 2026-07-08 continuation:
|
||||
|
||||
| Check | Result | Interpretation |
|
||||
|---|---:|---|
|
||||
| Targeted root/package Vitest slices | Mixed | Admin-web, launcher package-local tests, `@waggle/cli` package-local tests, memory MCP package-local tests, hive-mind MCP, marketplace targeted tests, and hive-mind CLI package-local tests pass. Remaining mixed status is from rendered/admin and deeper state evidence, not this test command lane. |
|
||||
| No-emit TypeScript | Pass | Admin web, Waggle CLI, marketplace, memory MCP, hive-mind MCP, and hive-mind CLI compile. `packages/launcher` has no local `tsconfig.json` and relies on `tsup`. |
|
||||
| Package builds | Pass | Admin web, launcher, Waggle CLI, marketplace, memory MCP, hive-mind MCP, and hive-mind CLI build scripts completed. |
|
||||
| Source help smokes | Pass | Launcher, Waggle CLI, hive-mind CLI, and marketplace help paths work through `tsx`. |
|
||||
| Built help/runtime smokes | Mixed | Built marketplace help works with no DB side effect; built `@waggle/cli` help and bin-wrapper help work with no `.waggle` side effect; built launcher help works without a service banner or `.waggle` creation. |
|
||||
| MCP protocol smoke | Improved mixed | Built hive-mind MCP server and built legacy `waggle-memory-mcp` both complete official MCP client handshakes and write-scope save/recall roundtrips from temp data dirs. Package-publish proof remains open. |
|
||||
| Negative-path smokes | Improved mixed | Marketplace unknown commands now print help, exit 1, and avoid DB creation; hive-mind CLI sampled subcommand help now prints focused help and avoids `personal.mind` creation. Broader package/publish and every-subcommand sampling remain open. |
|
||||
|
||||
Open utility UX risks:
|
||||
|
||||
| ID | Risk | Evidence | Why it matters |
|
||||
|---|---|---|---|
|
||||
| UTIL-1 | Admin web is a separate Vite UI with shallow rendered UX evidence. | `packages/admin-web/src/App.tsx` uses its own inline-styled sidebar, token/slug setup, and pages. Component tests pass but emit repeated `act(...)` warnings; no current screenshot, mobile, keyboard, or visual-guideline evidence is attached. | Team admins can hit this outside the cockpit; it needs either a judge pass or explicit deferral. |
|
||||
| UTIL-2 | Built/published-style CLI/MCP entries are locally healthier but still not publish-proved. | Built marketplace help/invalid-command paths, built launcher help, built `@waggle/cli` help, built legacy `waggle-memory-mcp` read/write handshakes, and built hive-mind MCP write roundtrip are locally fixed; clean npm tarball/`npx` invocation remains unproved for the locally fixed CLIs/MCP packages. | A source-mode dev smoke can pass while the command a user installs or runs via `npx` is unusable. |
|
||||
| UTIL-3 | Hive-mind CLI tests are present outside the root Vitest include pattern, with a package-owned test lane now added. | Root-focused commands against `packages/hive-mind-cli/src/*.test.ts` returned "No test files found"; `npm run test --workspace @waggle/hive-mind-cli -- --reporter=dot` now runs 5 files / 45 tests through the package-local config. | Root verification can still miss this package unless the documented package command is included. |
|
||||
| UTIL-4 | Launcher CLI has clean help smoke and argument tests, but no no-emit TypeScript path and no full startup/failure UX smoke in this packet. | `packages/launcher` has `tsup.config.ts` but no `tsconfig.json`; launcher tests copy argument parsing shape and do not start the service; built help no longer logs service data-dir information before help. | First-run `npx waggle` is a real acquisition path; port conflict, service failure, and browser-open behavior need evidence or deferral. |
|
||||
| UTIL-5 | Utility negative paths need broader sampling, though marketplace and sampled hive-mind CLI paths are now guarded. | Marketplace unknown commands now exit 1 without DB construction; sampled hive-mind CLI source/built subcommand help exits 0 without `personal.mind` creation. Other utility invalid-input, JSON, missing-env, and publish-style paths are not fully sampled. | Help and invalid-input paths should be safe, obvious, and scriptable. |
|
||||
| UTIL-6 | CLI/MCP utility UX remains incomplete beyond the successful MCP protocol smokes. | Memory MCP and hive-mind MCP read/write protocol smokes pass, but JSON output, invalid args, missing env, bad data dir, auth failure states, and clean npm tarball behavior are not fully sampled. | Engineer and admin personas may rely on these tools when desktop UI is unavailable or during setup. |
|
||||
|
||||
Recommended ticket: T15, Admin and CLI Utility UX Gate.
|
||||
|
||||
Suggested acceptance:
|
||||
|
||||
- Admin web has screenshot/keyboard/mobile evidence or is explicitly deferred from the product UX score.
|
||||
- Root test discovery includes or deliberately excludes hive-mind CLI colocated tests, with the passing package-local command kept in the verification lane.
|
||||
- Launcher CLI has startup, invalid port, port conflict, `--no-open`, and failure copy evidence, or is deferred.
|
||||
- Marketplace/memory MCP/hive-mind CLI commands have help, invalid input, JSON output, and missing-env/error-state evidence for the supported user paths.
|
||||
|
||||
## AI-Tool Hook Lifecycle: `packages/hive-mind-hooks-*` and Launcher
|
||||
|
||||
User jobs:
|
||||
|
||||
- Detect a supported AI tool.
|
||||
- Install memory hooks without corrupting the tool's existing configuration.
|
||||
- Verify installed hooks and understand failures.
|
||||
- Uninstall hooks byte-identically or remove only Waggle-managed config.
|
||||
- Understand that Claude Desktop is launchable but not hook-capable.
|
||||
- Trust hook events to fail open when hive-mind CLI or the sidecar is unavailable.
|
||||
|
||||
Current command evidence, 2026-07-08 continuation:
|
||||
|
||||
| Check | Result | Interpretation |
|
||||
|---|---:|---|
|
||||
| Shared/agent/server T16 route-contract plus package-runtime slice | Pass, original 7 files / 96 tests plus focused registry slice 5 files / 108 tests | Manifest, third-party loader, hook-capable cohort, backend hook route, launch, process, detection route contracts, packed local hook-package lifecycle, and registry-aware third-party launch agree. |
|
||||
| Launcher/prompt/adapter slice | Pass, 3 files / 28 tests | Launcher A/B toggle, one non-Claude hook action exposure, live output pane wiring, prompt args, and adapter methods pass. Node emits `punycode` warnings. |
|
||||
| Root-run hook and shim package tests | Pass, 60 files / 573 tests, 1 skipped | Hook install/verify/uninstall, config merge/unmerge, adapters, lifecycle handlers, fail-open behavior, Codex Desktop parity, hook core, and shim core are well-covered at unit/package level. Output includes expected fail-open warnings and sidecar-unreachable signal drops. |
|
||||
| Official hook/shim package typechecks | Pass, 8/8 | `hive-mind-shim-core`, `hive-mind-hooks-core`, and the six real hook packages pass their `tsc --build && tsc --noEmit -p tsconfig.test.json` lanes. |
|
||||
| `npm run build --workspace @waggle/hive-mind-hooks-claude-desktop` | Pass | The intentional Claude Desktop no-bin stub still builds. |
|
||||
| Compiled hook-bin help smokes | Pass, 6/6 | `claude-code-hooks`, `codex-hooks`, `codex-desktop-hooks`, `cursor-hooks`, `hermes-hooks`, and `openclaw-hooks` boot locally and show usage. This does not prove published `npx` resolution. |
|
||||
| Rendered Launcher Browser smoke | Pass, partial | Fresh `npm run build` plus in-app Browser evidence under `output/playwright/launcher-t16-54147/` covers mixed installed/not-installed/hooks-active/running/Phase 4 states, prompt summary, install success, verify failure, and running output against a mock local API. |
|
||||
| Rendered Launcher Playwright states | Pass, 1 file / 4 tests | Production-build Playwright coverage proves sidecar-offline Retry recovery, long hook stderr summarization with `More output`, hidden-line count, and recovery guidance, standard install output with `Changed file`, `Install pointer`, `Backup`, and `Recovery` labels, and non-built-in adapter launch-only/prompt behavior. |
|
||||
| Package-local hook/shim `npm run test --workspace ...` scripts | Pass, 60 files / 574 tests | Hook core, all six hook-capable packages, and shim core now run package-local scripts against the intended root Vitest config/package paths. The shim lane also verifies the CLI ESM resolver fix for the MCP server entry. |
|
||||
| Focused Launcher/web regressions | Pass, 5 files / 72 tests | Observed output closes after exit, covered install output shows Backup/Recovery labels without raw `stdout:`, Verify `[FAIL]` output becomes Check-failed/manual-approval rows without raw `[FAIL]`, uninstall output labels restore/cleanup rows without implying install state, long hook output is capped behind a `More output` summary, hook stdout/stderr details are preserved, structured hook failures are not replaced with `HTTP 400`, empty-output Verify failures show retry/uninstall/reinstall recovery copy, installed Claude Desktop is explicitly launch-only with no hook actions, and a detected launchable third-party adapter gets Launch plus prompt routing. |
|
||||
|
||||
Open hook UX risks:
|
||||
|
||||
| ID | Risk | Evidence | Why it matters |
|
||||
|---|---|---|---|
|
||||
| HOOK-1 | Real hook-management lifecycle is now proved for all six hook-capable route paths and rendered UI transitions; packaged-desktop evidence remains open. | The gated real-tool smoke now proves a detected OpenClaw CLI can render in Launcher and launch safely through `/api/tools/launch` with observed output, exit 0, and process cleanup. A broadened gated route smoke drives `/api/tools/hooks` through real install, verify, and uninstall for `claude-code`, `codex`, `codex-desktop`, `cursor`, `hermes`, and `openclaw` against an isolated `USERPROFILE/HOME`, proving config/pointer creation and cleanup without touching the user's real profile. A codified rendered Playwright spec covers all six hook-capable cards through install, verify, uninstall, and refreshed hooks-active UI transitions. No current evidence proves packaged desktop hook-status transitions. | Hook install modifies user tool config, so reversibility and status copy are trust-critical. |
|
||||
| HOOK-2 | Rendered Launcher hook-management states are codified across the all-tool matrix; screenshot breadth remains partial. | The Browser smokes cover detected, not detected, hooks active, install success, verify failure, mocked uninstall cleanup, running output, and Phase 4/unsupported states. Codified Playwright coverage now covers sidecar offline Retry, long-stderr summarization, standard install changed-file/pointer/backup/recovery labels, all six hook-capable tool cards rendering install/verify/uninstall transitions, non-built-in adapter launch-only/prompt behavior, one real detected CLI observed-launch lifecycle, and real route lifecycle for all six hook-capable tools. | Engineer and Solo founder judges need to understand what happened without reading logs. |
|
||||
| HOOK-3 | Focused fixed: Claude Desktop is a deliberate unsupported hook case. | `BUILTIN_TOOL_MANIFESTS` marks `claude-desktop` as non-hook-capable; `packages/hive-mind-hooks-claude-desktop` is a stub/no-bin package. The Launcher now shows installed Claude Desktop as `Launch only`, explains hooks are not supported yet, and hides hook actions; component coverage owns the regression and a Browser-rendered mocked state confirmed the visible layout. | Keep the copy in the rendered state matrix; no hook package/bin should be exposed for Claude Desktop until the MCP-bridge category ships. |
|
||||
| HOOK-4 | Local fixed: production-shaped `npx` hook package resolution is proved for the local package closure. | `hook-packages-runtime.test.ts` packs and installs the local hook package closure into a clean temp project, then runs `npx --yes @waggle/hive-mind-hooks-<id> install/verify/uninstall` for all six hook-capable tools. Registry-only proof remains a publication/release check. | A user click in Launcher depends on `npx --yes @waggle/hive-mind-hooks-<id>`. |
|
||||
| HOOK-5 | Rendered fixed; release residual: hook success/failure copy preserves stdout/stderr/structured-failure details, shows Backup/Recovery labels for covered install output, turns Verify `[FAIL]` output into Check-failed/manual-approval rows, labels uninstall restore/cleanup rows, caps long output behind `More output`, gives empty-output recovery copy, exposes offline retry, and has real route plus rendered lifecycle proof for all six hook-capable tools. | Component tests now prove install success shows a Backup label and backup path without raw `stdout:`, verify failure shows stderr even when the route returns a generic error, Verify check failure shows manual-approval detail without raw `[FAIL]`, uninstall output shows `Changed file`, `Restored from`, `Created file removed`, `Backup removed`, and `Pointer removed` without `Install pointer`, long stderr hides later noisy lines behind a count while preserving `Recovery`, and empty-output Verify failure shows retry/uninstall/reinstall guidance. Adapter tests now prove a structured hook failure with no stderr/error is not replaced with `HTTP 400`. Browser evidence shows mocked Codex Verify manual-approval and Uninstall cleanup results plus a real `/launcher` smoke renders `verify failed (exit 1)` with recovery guidance instead of `HTTP 400`. Playwright rendered evidence covers standard install changed-file/pointer/backup/recovery labels, sidecar-offline Retry, long-output summarization, all six hook-capable rendered install/verify/uninstall transitions, real OpenClaw observed launch/output/exit/process cleanup, and real route install/verify/uninstall for all six hook-capable tools against an isolated profile. | Users still need packaged desktop hook-status proof and quieter release output for config-editing flows. |
|
||||
| HOOK-11 | Focused fixed: Windows hook route commands now resolve `npx` correctly. | The first gated route smoke reproduced `/api/tools/hooks` returning HTTP 400 with empty stdout/stderr because Windows `execFile('npx')` could not resolve the npm shim. `runHookCommand()` now prefers the `npx.cmd` beside `process.execPath`, and default exec capture uses the shared `.cmd` resolver. The broadened isolated route lifecycle now passes for all six hook-capable tools. | Keep the gated route smoke in the release lane. |
|
||||
| HOOK-10 | Focused fixed: Windows npm shims can launch, and restricted Codex WindowsApps aliases show recovery instead of a failing launch path. | Detection now prefers spawnable Windows PATH hits such as `.cmd` over extensionless POSIX npm shims. Standard npm `.cmd` shims resolve to their Node module target so prompt args stay literal; unknown `.cmd`/`.bat` files use a quoted fallback. The real smoke passed against OpenClaw. Built detection now reports this host's Codex WindowsApps `codex.exe` as installed but `launchable: false` with recovery copy, and Launcher hides the Launch button for that state. | Windows users should not see a launchable tool that crashes the sidecar. Keep the regression and require a real PATH CLI or proven desktop bridge for future direct Codex launch. |
|
||||
| HOOK-6 | Focused/rendered fixed for launchable third-party adapters. | Detection merges `~/.waggle/adapters/*.json`; `/api/tools/launch` now validates against the runtime registry and applies adapter prompt templates, while `/api/tools/hooks` stays built-in-only until a safe hook policy exists. Rendered Playwright coverage proves a non-built-in adapter shows launch-only/no-hook copy, routes the prompt, and sends the expected launch payload. | Advanced adapter users can launch a detected launchable adapter; third-party hook management remains intentionally disabled until a safe hook policy exists. |
|
||||
| HOOK-7 | Standard output is noisy during successful hook tests. | Passing test output includes fail-open warnings, sidecar-unreachable signal drops, package logs, and server embedding warnings. | Noise can hide real failures and reduce confidence in release checks. |
|
||||
| HOOK-8 | Local fixed: package-local hook test commands now run their intended lanes. | Package-local hook/shim `npm run test --workspace ...` scripts pass for hook core, all six hook-capable packages, and shim core. | Keep these scripts in the release lane; remaining package-local command-shape gaps are tracked under DEV-1. |
|
||||
| HOOK-9 | Focused fixed: observed live output does not reconnect and replay after exit. | `streamToolOutput()` now closes its EventSource on a valid `exit` event. The focused web Launcher/adapter suite proves no reconnect/replay after exit. | Keep the regression; refresh rendered Browser evidence when the full Launcher state matrix is rerun. |
|
||||
|
||||
Recommended ticket: T16, AI-Tool Hook Lifecycle UX Gate.
|
||||
|
||||
Suggested acceptance:
|
||||
|
||||
- Rendered Launcher screenshots or Browser/Playwright evidence cover hook install/verify/uninstall states across all six hook-capable tools, duplicate-free running output, and unsupported Claude Desktop copy.
|
||||
- Command-level lifecycle evidence covers all six hook-capable tools.
|
||||
- Real detected CLI launch has at least one observed safe smoke; hook install/verify/uninstall has isolated route lifecycle evidence for all six hook-capable tools.
|
||||
- Packaged desktop hook-status transitions are proved or explicitly deferred.
|
||||
- Production-like `npx` or package-pack resolution is verified for every hook package Launcher can invoke.
|
||||
- Third-party adapter launch has focused route/UI proof and rendered non-built-in adapter proof; third-party hook management stays out of scope until a safe hook policy exists.
|
||||
- Hook test/log output is quiet enough that real failures stand out, or expected warning noise is explicitly filtered/documented.
|
||||
- Typecheck/build commands for project-reference hook packages are documented and pass in the approved verification lane.
|
||||
|
||||
Focused supplement: `docs/audits/2026-07-08-ai-tool-hook-t16-analysis.md`.
|
||||
|
||||
## Developer APIs, Background Jobs, and Substrate Packages
|
||||
|
||||
Dedicated supplement: `docs/audits/2026-07-08-developer-substrate-t17-analysis.md`.
|
||||
|
||||
User jobs:
|
||||
|
||||
- Use `@waggle/sdk` to validate, install, and run skills/plugins.
|
||||
- Rely on local server APIs for chat, workspaces, marketplace, billing, backup, compliance, hooks, and startup recovery.
|
||||
- Trust background worker jobs for dispatch, chat handling, and job processing.
|
||||
- Trust WaggleDance protocol behavior and signal handling.
|
||||
- Trust memory substrate, wiki compiler, optimizer, and shared package behavior that powers the UI.
|
||||
- Run documented package-local and root verification commands without false failures or hidden skipped tests.
|
||||
|
||||
Current command evidence, 2026-07-08 continuation:
|
||||
|
||||
| Check | Result | Interpretation |
|
||||
|---|---:|---|
|
||||
| `npx tsc --noEmit --project packages/<target>/tsconfig.json` for `agent`, `core`, `hive-mind-core`, `hive-mind-shim-core`, `hive-mind-wiki-compiler`, `optimizer`, `sdk`, `server`, `shared`, `waggle-dance`, `weaver`, `wiki-compiler`, and `worker` | Pass, 13/13 workspaces | Remaining developer/backend/substrate workspaces compile under direct no-emit TypeScript. |
|
||||
| `npm run test -w @waggle/agent` | Pass, 194 files / 3079 tests | Agent runtime, connectors, tool launcher, orchestration, memory recall, personas, security, and workflow behavior are broadly covered. |
|
||||
| `npm run test -w @waggle/core -- --reporter=dot` | Pass, 19 files / 295 tests | Core config, cron/file stores, compliance, vault-adjacent helpers, and quota behavior pass; output includes embedding provider probing noise. |
|
||||
| `npm run test -w @waggle/optimizer -- --reporter=dot` and `npm run test -w @waggle/weaver -- --reporter=dot` | Pass, 4 files / 52 tests | Optimizer and Weaver unit behavior pass through package scripts. |
|
||||
| `npx vitest run packages/hive-mind-core/tests packages/hive-mind-shim-core/tests packages/wiki-compiler/tests --config vitest.config.ts --reporter=dot` | Pass, 71 files / 875 tests | Memory substrate, shim core, and wiki compiler tests pass through the root runner. |
|
||||
| `npx vitest run packages/sdk/tests --config vitest.config.ts --reporter=dot` | Pass, 5 files / 88 tests | SDK behavior tests pass through the root runner. |
|
||||
| `npx vitest run packages/shared/tests packages/waggle-dance/tests packages/worker/tests packages/hive-mind-wiki-compiler/src --config vitest.config.ts --reporter=dot` | Pass, 12 files / 128 tests | Shared contracts, WaggleDance protocol, worker jobs, and hive-mind wiki compiler tests pass through the root runner. |
|
||||
| `npx vitest run packages/server/tests --config vitest.config.ts --reporter=dot` | Pass, 184 files passed / 1 skipped; 2117 tests passed / 1 skipped | Current full server run passed, but took 98.34s and emitted very large marketplace/embedding/startup logs. Earlier packet evidence saw one performance-budget failure under full-suite load. |
|
||||
| `npx vitest run packages/server/tests/performance/benchmarks.test.ts --config vitest.config.ts --reporter=default` | Pass, 13 tests | The same benchmark passes in isolation, so the risk is full-suite stability/contention and audit determinism, not a consistently broken endpoint. |
|
||||
| `npm run test -w @waggle/hive-mind-core` and `@waggle/wiki-compiler` | Fail | Package-local Vitest tries to load `vitest.setup.ts` relative to the package cwd; root-run tests pass. `@waggle/hive-mind-shim-core` now passes package-local tests. |
|
||||
| `npm run test -w @waggle/sdk` | Fail | Workspace-local Vitest resolves into `packages/sdk/node_modules` and cannot find `convert-source-map`; root-run SDK tests pass. |
|
||||
|
||||
Open developer/substrate UX risks:
|
||||
|
||||
| ID | Risk | Evidence | Why it matters |
|
||||
|---|---|---|---|
|
||||
| DEV-1 | Several package-local `npm test` scripts fail even when the same tests pass from the root runner. | `hive-mind-core` and `wiki-compiler` fail on missing package-local `vitest.setup.ts`; SDK fails on missing local `convert-source-map`. `hive-mind-shim-core` is now fixed. | Engineers following package scripts hit false failures, slowing fixes and reducing trust in the verification lane. |
|
||||
| DEV-2 | The full server route suite is green in the current run but not release-clean. | Current full server run passed, but took 98.34s and emitted very large logs; earlier packet evidence saw one perf-budget failure at 841ms vs 500ms while isolated benchmark passed. | A release-quality UX gate needs deterministic, quiet command evidence with perf checks in a stable lane. |
|
||||
| DEV-2a | Marketplace sync behavior leaks into the normal server lane. | `marketplace-sync.test.ts` logs many source sync attempts, add counts, and error counts during the full server run. | The default release lane should be hermetic and quiet; live/external catalog behavior belongs in a named integration lane. |
|
||||
| DEV-3 | Server and substrate verification output is very noisy. | Passing runs emit repeated embedding fallback warnings, marketplace sync logs, sidecar-unreachable-style warnings, and route startup logs. | Noise makes real failures harder to spot and weakens confidence in "all tested." |
|
||||
| DEV-4 | Developer-facing SDK/server/worker happy paths are tested, but user-facing recovery/error copy is not fully sampled. | Unit/API tests pass; no current packet evidence covers SDK docs/examples, server API consumer ergonomics, worker job failure UI copy, or bad config paths as a coherent user journey. | Engineer and admin personas depend on these APIs when integrating, debugging, or recovering from failed jobs. |
|
||||
|
||||
Recommended ticket: T17, Developer API, Background Worker, and Substrate Verification UX Gate.
|
||||
|
||||
Suggested acceptance:
|
||||
|
||||
- Package-local scripts either pass or clearly delegate to the correct root/project-reference verification command.
|
||||
- Root verification discovers all intended package tests or explicitly documents separate lanes.
|
||||
- Full server route/performance gate is deterministic, with realistic budgets or isolated perf lanes.
|
||||
- Noisy logs are reduced, filtered, or documented so real failures stand out.
|
||||
- SDK/server/worker developer journeys have happy-path and error-path evidence, or are explicitly deferred from the five-persona score.
|
||||
|
||||
## Ops, Deployment, CI, Benchmarks, and Judging
|
||||
|
||||
Dedicated supplement: `docs/audits/2026-07-08-ops-deploy-ci-judging-t18-analysis.md`.
|
||||
|
||||
User jobs:
|
||||
|
||||
- Release reviewers trust GitHub Actions, Docker, Render, and local Compose as honest shipping gates.
|
||||
- Operators can validate deployment configuration without leaking local secrets into logs.
|
||||
- Hosted deploy target is coherent: local sidecar demo versus team Postgres server is an explicit decision.
|
||||
- Infra-dependent suites have a known, runnable lane.
|
||||
- Benchmark and judge harnesses used for product claims compile and test through documented commands.
|
||||
- Historical judging screenshots/reports are not mistaken for current five-persona score evidence.
|
||||
|
||||
Current command evidence, 2026-07-08 continuation:
|
||||
|
||||
| Check | Result | Interpretation |
|
||||
|---|---:|---|
|
||||
| YAML parse for `docker-compose.yml`, `docker-compose.production.yml`, `render.yaml`, `litellm-config.yaml`, and all 7 `.github/workflows/*.yml` | Pass | The deployment, LiteLLM, and workflow YAML files are syntactically valid. |
|
||||
| `docker --version` | Pass | Docker is available in this environment for config-level checks. |
|
||||
| `docker compose ps --format json` | Fail | Docker CLI is installed, but the Docker Desktop Linux engine pipe is not reachable, so live Compose services and `npm run test:infra` could not be run here. |
|
||||
| `docker compose -f docker-compose.yml config` and `docker compose -f docker-compose.production.yml config` | Pass with secret-log caveat | Compose can expand both files, but raw expansion reads local env values and can print secret values. Use `--no-interpolate` or a sanitized env for shareable logs. |
|
||||
| `docker compose -f docker-compose*.yml config --no-interpolate` targeted scan | Pass | The non-interpolated command is safer for evidence because it keeps variable references instead of local secret values. |
|
||||
| `git ls-files` / `git check-ignore` for `.env`, `env.local`, `AI API KEYS.txt`, and `apps/www/.env.local` | Pass for tracking hygiene | Only `.env.example` files are tracked; local secret-bearing files checked here are ignored. |
|
||||
| `npx tsc --noEmit --project benchmarks/harness/tsconfig.json` | Pass | Benchmark harness TypeScript compiles. |
|
||||
| `npm run test --prefix benchmarks/harness` | Fail | Package-local Vitest tries to load `benchmarks/harness/vitest.setup.ts`; root-run tests pass. |
|
||||
| `npx vitest run benchmarks/harness/tests --config vitest.config.ts --reporter=dot` | Pass, 29 files / 325 tests | Benchmark harness unit/smoke/stats/judge tests pass through the root runner, with noisy turn/bench logs. |
|
||||
| `judging/` inventory | Historical only | Current files are prior June 2026 rounds/screenshots, not the requested July five-persona scorecards. |
|
||||
|
||||
Open ops/judging risks:
|
||||
|
||||
| ID | Risk | Evidence | Why it matters |
|
||||
|---|---|---|---|
|
||||
| OPS-1 | Shareable Compose validation can leak local secrets if run in the obvious way. | `docker compose config` expands local ignored `.env` values; ignored-file checks show those values are not tracked, but command output can still expose them. | Release/ops evidence must be secret-safe; logs are part of the UX of operating the system. |
|
||||
| OPS-2 | Render deployment target is still ambiguous. | `render.yaml` provisions Postgres and Redis but starts `packages/server/src/local/start.ts --skip-litellm`, the local SQLite sidecar path, not the team Postgres server. | A hosted deploy should either be a local-sidecar demo or a team server; mixed infra confuses operators and can hide broken team workflows. |
|
||||
| OPS-3 | CI E2E remains advisory and infra tests are not in CI; local live infra could not be run in this environment. | `ci.yml` marks the E2E job `continue-on-error: true`; no workflow starts Postgres/Redis or runs `npm run test:infra`; `vitest.infra-suites.ts` lists 19 suites that require live infra; local `docker compose ps` failed because the Docker Desktop engine pipe was missing. | Green CI can still miss route, browser, and Postgres/Redis regressions, and this packet only proves config-level Docker validity. |
|
||||
| OPS-4 | Production Compose uses development-style default credentials unless explicitly overridden. | `docker-compose.production.yml` has fallback values for Postgres and MinIO credentials. | Defaults are convenient locally, but production operators need fail-closed secret requirements or prominent warning evidence. |
|
||||
| OPS-5 | Benchmark harness package-local test script fails even though root-run tests pass. | `npm run test --prefix benchmarks/harness` fails on missing local setup file; root-run benchmark tests pass. | Benchmark/judge credibility depends on commands that future reviewers can run without knowing hidden root-run shape. |
|
||||
| OPS-6 | Existing `judging/` artifacts are stale for the current goal. | `judging/round2`, `round3`, `screenshots`, and crops are June 2026 artifacts; current scorecards are not filled. | The requested 9/10 result must be generated from current source after fixes, not inherited from older screenshots. |
|
||||
|
||||
Recommended ticket: T18, Ops, Deployment, CI, Benchmark, and Judging Evidence Gate.
|
||||
|
||||
Suggested acceptance:
|
||||
|
||||
- Secret-safe ops validation commands are documented and used for shareable evidence.
|
||||
- Render target is decided and verified: hosted local-sidecar demo or team Postgres server.
|
||||
- Docker/Compose production path is built or explicitly deferred, with fail-closed secret expectations.
|
||||
- CI has a blocking smoke lane or the advisory E2E/infra gap is explicitly accepted before scoring.
|
||||
- `npm run test:infra` has a documented Docker/migration lane and evidence, or is deferred.
|
||||
- Benchmark harness package-local command shape is fixed or documented; root-run benchmark evidence remains green.
|
||||
- Five-persona judging artifacts are generated from current post-fix source and replace historical screenshots for scoring.
|
||||
|
||||
## Phase Placement
|
||||
|
||||
Phase 1 remains unchanged and should stay focused on the installed app P0s:
|
||||
|
||||
- T1 auth/CSP/accountless console health.
|
||||
- T2 mobile Settings.
|
||||
- T3 Solo/Teams/Enterprise copy in active app UI.
|
||||
- T4 shortcut and Workspace Switcher route contract.
|
||||
- T5 visual snapshot triage.
|
||||
- T11 route evidence for thin in-app routes.
|
||||
|
||||
T13, T14, T15, T16, T17, T18, and T19 are not part of Phase 1 unless a Phase 1 verification command directly forces a tiny supporting fix. They are final-product gates after the in-app blockers are cleared.
|
||||
|
||||
## Final Goal Rule
|
||||
|
||||
Do not claim "complete UX" until:
|
||||
|
||||
- T13, T14, T15, T16, T17, T18, and T19 are fixed and verified, or
|
||||
- the user explicitly approves deferring launch/extension/desktop/utility/hook/developer/ops gates from the five-persona score.
|
||||
|
||||
The approval brief can still request Phase 1 implementation now, because Phase 1 removes known P0s in the main installed cockpit and is the fastest path to a credible judge dry run.
|
||||
81
docs/audits/2026-07-08-ux-packet-index.md
Normal file
81
docs/audits/2026-07-08-ux-packet-index.md
Normal file
@@ -0,0 +1,81 @@
|
||||
# UX Audit Packet Index
|
||||
|
||||
Status: analysis packet is approval-ready. Full goal remains open until approved fixes and five-persona judge scoring are complete.
|
||||
|
||||
## Start Here
|
||||
|
||||
1. `docs/audits/2026-07-08-ux-approval-brief.md` - shortest decision doc; approve Phase 1 from here.
|
||||
2. `docs/audits/2026-07-08-analysis-completion-audit.md` - proves what the analysis phase covers and what remains unfinished.
|
||||
3. `docs/superpowers/plans/2026-07-08-ux-phase-1-corrections.md` - implementation plan to execute after approval.
|
||||
|
||||
## Evidence Files
|
||||
|
||||
| File | Purpose |
|
||||
|---|---|
|
||||
| `docs/audits/2026-07-08-complete-ux-usage-audit.md` | Full evidence, findings, phases, tickets, and verification results. |
|
||||
| `docs/audits/2026-07-08-ux-route-scenario-manifest.md` | Registered routes, overlays, scenarios, and evidence owner requirements. |
|
||||
| `docs/audits/2026-07-08-ux-state-failure-scenario-matrix.md` | Account, tier, model, data, offline, mobile, and failure-state combinations that judge runs must sample. |
|
||||
| `docs/audits/2026-07-08-ux-non-main-surface-scope.md` | Public landing/download/checkout/legal, desktop wrapper, installer/update, sidecar-startup, utility, hook, developer/substrate, ops/deployment, CI, benchmark, and judging scope decisions. |
|
||||
| `docs/audits/2026-07-08-five-persona-judge-scorecards.md` | Five judge personas, scoring model, score caps, and required screenshots/evidence. |
|
||||
| `docs/audits/2026-07-08-five-persona-judge-runbook.md` | Executable judge protocol: state bundles, route sequences, required screenshots, score caps, deferral rules, and evidence folder shape. |
|
||||
| `docs/audits/2026-07-08-ux-correction-register.md` | Master row-by-row tracker mapping findings to tickets, phases, files, judges, and closure evidence. |
|
||||
| `docs/audits/2026-07-08-ux-approval-brief.md` | Concise approval decision and Phase 1 scope boundaries. |
|
||||
| `docs/audits/2026-07-08-analysis-completion-audit.md` | Requirement trace from the original goal to current evidence and remaining blockers. |
|
||||
| `docs/audits/2026-07-08-ux-post-phase-1-roadmap.md` | Sequencing for Phase 2/3 work after Phase 1; not approved for implementation yet. |
|
||||
| `docs/audits/2026-07-08-web-guidelines-line-findings.md` | Line-level Web Interface Guidelines supplement for native dialogs, focus, forms, animation, images, and locale formatting. |
|
||||
| `docs/audits/2026-07-08-runtime-a11y-t10-analysis.md` | Focused runtime axe/DOM accessibility smoke for core desktop/mobile routes and Command Center; adds rendered T10 evidence for unnamed controls, select labels, scroll regions, workspace semantics, and mobile command-label overflow. |
|
||||
| `docs/audits/2026-07-08-visual-t5-classification.md` | Fresh visual-regression rerun and 14-snapshot classification: current screenshots are coherent, active baselines are stale, duplicate baseline families need cleanup, and baseline updates must wait for approval. |
|
||||
| `docs/audits/2026-07-08-first-run-onboarding-t1-t2-t12-analysis.md` | Focused clean-data first-run onboarding smoke with desktop completion, mobile 390 x 844 profile evidence, Clerk/CSP console reproduction, import-risk findings, and T1/T2/T12 closure updates. |
|
||||
| `docs/audits/2026-07-08-source-inventory-consistency-audit.md` | Current-source route/app/package inventory check; adds Browser Companion T19 and command-query evidence notes. |
|
||||
| `docs/audits/2026-07-08-browser-companion-t19-analysis.md` | Focused Browser Companion extension analysis with manifest/syntax/typecheck evidence, disconnected/connected popup screenshots, direct sidecar save evidence, current toolbar/extraction probes, Memory UI confirmation, CORS/auth-pairing findings, and T19 line-level findings. |
|
||||
| `docs/audits/2026-07-08-desktop-wrapper-t14-analysis.md` | Focused Tauri desktop wrapper analysis with Rust/app/server checks, native event consumer matrix, release/tray source hardening, and T14 line-level findings. |
|
||||
| `docs/audits/2026-07-08-admin-cli-utility-t15-analysis.md` | Focused admin web, CLI, marketplace, memory MCP, hive-mind MCP, and hive-mind CLI analysis with package test/build/help and MCP protocol smoke evidence. |
|
||||
| `docs/audits/2026-07-08-ai-tool-hook-t16-analysis.md` | Focused AI-tool hook lifecycle analysis with Launcher/source evidence, rendered Browser smoke artifacts, hook package/root-run tests, package typechecks, compiled bin help smokes, and T16 correction candidates. |
|
||||
| `docs/audits/2026-07-08-developer-substrate-t17-analysis.md` | Focused developer API, server, worker, WaggleDance, SDK, and substrate verification analysis with package/root command evidence and T17 correction candidates. |
|
||||
| `docs/audits/2026-07-08-ops-deploy-ci-judging-t18-analysis.md` | Focused ops, deployment, CI, benchmark, and judging analysis with YAML/Compose/secret-safe checks, benchmark evidence, and T18 correction candidates. |
|
||||
| `docs/audits/2026-07-08-launch-funnel-t13-analysis.md` | Focused public launch funnel analysis with www test/build/typecheck/route smoke evidence, rendered Browser checkout-recovery evidence, live apex/www DNS failure, download/release check, local checkout/auth recovery fixes, controlled download status page, legal placeholder guard, and remaining T13 correction candidates. |
|
||||
| `docs/audits/2026-07-08-route-evidence-t11-analysis.md` | Focused installed-app route evidence analysis with route registry, direct test-reference matrix, app-test command evidence, zero-hit route findings, current all-route built-preview smoke evidence, and T11 correction candidates. |
|
||||
| `docs/audits/2026-07-08-state-failure-t12-analysis.md` | Focused T12 analysis with state-slice test evidence, native dialog source evidence, five-persona state-bundle contract, and T12 correction candidates. |
|
||||
| `docs/audits/2026-07-08-mobile-executive-t2-t12-analysis.md` | Fresh 390 x 844 Mobile Executive smoke evidence showing Settings visible clipping despite clean document scroll width, Memory/chat tab overflow, Command Center mobile close/label risks, and stronger T2/T12 acceptance criteria. |
|
||||
| `docs/audits/2026-07-08-shell-overlays-t10-t12-analysis.md` | Focused shell-overlay smoke/source evidence for Notification Inbox, Create Workspace, Spawn Agent, Workspace Switcher, Persona Switcher, Keyboard Shortcuts, Context Rail, Onboarding Tooltips, Upgrade Modal, and Trial Expired Modal. |
|
||||
| `docs/superpowers/plans/2026-07-08-ux-phase-1-corrections.md` | Bite-sized test-first implementation plan for Phase 1. |
|
||||
|
||||
## Current Decision
|
||||
|
||||
Approve Phase 1 before coding.
|
||||
|
||||
Recommended execution mode: Subagent-Driven.
|
||||
|
||||
Phase 1 includes only:
|
||||
|
||||
- Local auth/Clerk/CSP console health.
|
||||
- Mobile Settings and first-run onboarding responsive layout.
|
||||
- Solo/Teams/Enterprise copy cleanup.
|
||||
- `Ctrl+Shift+N` and Workspace Switcher route contract.
|
||||
- Visual snapshot triage.
|
||||
- Route evidence for thin judge paths.
|
||||
|
||||
## Not Ready To Claim
|
||||
|
||||
Do not mark the goal complete until:
|
||||
|
||||
- P0 findings are closed.
|
||||
- Phase 1 and any judge-blocking P1 items are verified.
|
||||
- Mobile evidence proves critical visible controls stay in-bounds, not merely that document-level horizontal overflow is absent.
|
||||
- First-run onboarding evidence proves accountless clean-data setup has no Clerk/CSP console errors and mobile Profile primary action is visible or clearly reachable.
|
||||
- Every registered route and major overlay has an evidence owner.
|
||||
- State and failure-mode bundles are declared for each judge run.
|
||||
- The five-persona judge runbook is filled with current evidence, not merely referenced.
|
||||
- Launch funnel and desktop wrapper gates are fixed or explicitly deferred from the final score.
|
||||
- Admin web, CLI launcher, marketplace CLI, and memory/MCP utility gates are fixed or explicitly deferred from the final score. Current admin-web rendered artifacts: `output/playwright/admin-web-t15-57795/`.
|
||||
- AI-tool hook lifecycle gates are fixed or explicitly deferred from the final score.
|
||||
- Developer API, background worker, and substrate verification gates are fixed or explicitly deferred from the final score.
|
||||
- Ops, deployment, CI, benchmark, and judging gates are fixed or explicitly deferred from the final score.
|
||||
- Web Guidelines line findings are fixed or explicitly assigned to T7/T8/T10/T12 deferrals.
|
||||
- Runtime T10 axe/DOM findings are fixed or explicitly assigned to judge-route deferrals.
|
||||
- T5 visual baseline decisions are approved after the fresh classification note; stale baselines are updated only after Phase 1 UI fixes land.
|
||||
- Command Center mobile close/label-fit evidence is fixed or explicitly deferred from the Mobile Executive and Engineer score paths.
|
||||
- Shell overlay T10/T12 findings are fixed or explicitly deferred, especially Notification Inbox Escape/semantics, Create Workspace semantics/mobile hierarchy, Context Rail landmarking, coach-mark semantics, and named close/icon controls.
|
||||
- Source-inventory supplement findings, including Browser Companion T19, are fixed or explicitly deferred.
|
||||
- Five persona scorecards are filled from current evidence.
|
||||
- All five personas score at least 9/10.
|
||||
572
docs/audits/2026-07-08-ux-post-phase-1-roadmap.md
Normal file
572
docs/audits/2026-07-08-ux-post-phase-1-roadmap.md
Normal file
@@ -0,0 +1,572 @@
|
||||
# UX Post-Phase-1 Roadmap
|
||||
|
||||
Status: analysis artifact. This roadmap sequences the remaining correction work after Phase 1. It is not approved for implementation yet.
|
||||
|
||||
## Purpose
|
||||
|
||||
Phase 1 removes the P0 blockers that currently cap the judge score. This roadmap keeps the remaining P1/P2 work visible so the product can move from "credible enough to judge" toward the requested 9/10 across five personas.
|
||||
|
||||
## Phase 2: Trust, Determinism, and Accessibility
|
||||
|
||||
Goal: remove P1 items that directly affect trust, recovery, or high-frequency workflows.
|
||||
|
||||
### R2-1: Trust-critical dialogs
|
||||
|
||||
Source finding: P1-1 / T7.
|
||||
|
||||
Line evidence: `docs/audits/2026-07-08-web-guidelines-line-findings.md` historically listed 20 high-confidence native browser dialog calls; the current follow-up scan finds 0 after the Create Workspace, Approvals, Artifact, Memory Center, Wiki export, Settings telemetry/backup/restore, BackupApp restore, Automation delete, compliance template delete, and admin member-removal fixes.
|
||||
|
||||
Current evidence update:
|
||||
|
||||
- Create Workspace custom-template delete is fixed with an in-app confirmation and focused component coverage.
|
||||
- Approvals revoke-all is fixed with the shared in-app `ApprovalModal`; `p7-b1-approvals-error.test.tsx` covers the no-native-confirm contract, `J3d` passes a rendered `/approvals` route path on port `34159`, and the expanded user-journey suite passed 20/20 on port `34160` before the Artifact `J3e` addition.
|
||||
- Artifact permanent delete is fixed with the shared in-app `ApprovalModal`; `artifact-center-trust.test.tsx` covers the no-native-confirm contract, `J3e` passes a rendered `/artifacts` route path on port `34161`, and the expanded user-journey suite passes 21/21 on port `34162`.
|
||||
- Memory Center permanent delete, GDPR erase, and allow re-import are fixed with the shared in-app `ApprovalModal`; `memory-center-trust.test.tsx` covers the no-native-confirm contract, `J3f` passes a rendered `/memory?tab=memories` route path on port `34164`, and the expanded user-journey suite passes 22/22 on port `34165`.
|
||||
- Wiki Obsidian and Notion exports are fixed with in-app form dialogs; `wiki-export-trust.test.tsx` covers the no-native-prompt contract, `J3g` passes a rendered `/memory?tab=wiki` route path on port `34167`, and the path remains included in the latest 24/24 user-journey suite on port `34173`.
|
||||
- Settings telemetry clear and Settings backup/restore are fixed with in-app approval/status states; `settings-trust.test.tsx` covers the no-native-dialog contract, `J3h` passes a rendered `/settings?tab=backup` route path on port `34171`, and the expanded user-journey suite passes 24/24 on port `34173`.
|
||||
- Standalone `BackupApp.tsx` restore is fixed with the shared in-app `ApprovalModal`; `p1b-authgate-surfaces.test.tsx` covers the no-native-confirm contract.
|
||||
- Automation delete is fixed with the shared in-app `ApprovalModal`; `phase3b-automation-center.test.tsx` covers the no-native-confirm contract.
|
||||
- Compliance template delete is fixed with the shared in-app `ApprovalModal`; `compliance-template-trust.test.tsx` covers the no-native-confirm contract.
|
||||
- Admin-web member removal is fixed with an in-app confirmation panel; `admin-pages.test.ts` covers the no-native-confirm contract.
|
||||
|
||||
Surfaces:
|
||||
|
||||
- No current high-confidence production native dialogs remain; broader T7 work should now focus on persona-state evidence, error states, and keyboard/screen-reader polish for the fixed flows.
|
||||
- Remaining workspace/template destructive actions, if any, outside the fixed custom-template delete path.
|
||||
|
||||
Acceptance:
|
||||
|
||||
- No native `confirm`, `alert`, or `prompt` in trust-critical flows.
|
||||
- In-app modal or inline confirmation names the object, consequence, reversibility, and next step.
|
||||
- Result state appears in-app after completion or failure.
|
||||
|
||||
Judge impact:
|
||||
|
||||
- Team admin.
|
||||
- Researcher.
|
||||
- Mobile executive where destructive flows are reachable.
|
||||
|
||||
### R2-2: Marketplace/local-first determinism
|
||||
|
||||
Source finding: P1-3 / T6.
|
||||
|
||||
Surfaces:
|
||||
|
||||
- Marketplace browse/search.
|
||||
- `/api/marketplace/sync`.
|
||||
- Marketplace background sync.
|
||||
|
||||
Acceptance:
|
||||
|
||||
- Standard UX audit does not call live external marketplace sources.
|
||||
- Live sync is moved to a named live-integration lane or guarded by explicit environment behavior.
|
||||
- Marketplace search and browse pass without retry flakes.
|
||||
|
||||
Judge impact:
|
||||
|
||||
- Engineer.
|
||||
- Solo founder.
|
||||
|
||||
### R2-3: Form, focus, and icon-button accessibility
|
||||
|
||||
Source finding: P1-5 / T10.
|
||||
|
||||
Line evidence: `docs/audits/2026-07-08-web-guidelines-line-findings.md` lists high-confidence focus misses, image-dimension misses, and a form metadata audit queue.
|
||||
|
||||
Surfaces:
|
||||
|
||||
- Settings.
|
||||
- Profile.
|
||||
- Workspace creation.
|
||||
- Workspace actions dialogs.
|
||||
- Agent/persona creation.
|
||||
- Mission Control and agent cards.
|
||||
- Compliance templates.
|
||||
- Onboarding forms.
|
||||
|
||||
Acceptance:
|
||||
|
||||
- High-traffic forms have accessible names, associated labels, and inline errors.
|
||||
- Focus order is predictable.
|
||||
- First invalid field receives focus on submit where practical.
|
||||
- Icon-only controls have accessible names.
|
||||
- Modal initial focus and focus return are covered for high-traffic dialogs.
|
||||
|
||||
Judge impact:
|
||||
|
||||
- Mobile executive.
|
||||
- Team admin.
|
||||
- Engineer.
|
||||
- Solo founder.
|
||||
|
||||
### R2-4: Test warning hygiene
|
||||
|
||||
Source finding: P1-2 / T10.
|
||||
|
||||
Surfaces:
|
||||
|
||||
- Component tests emitting repeated `act(...)` warnings outside the now-quiet focused admin-web lane.
|
||||
- Standard verification lane output.
|
||||
|
||||
Acceptance:
|
||||
|
||||
- Warnings that mask real failures are eliminated, scoped, or documented.
|
||||
- Standard verification output remains readable enough for reviewers.
|
||||
|
||||
Judge impact:
|
||||
|
||||
- Indirect, all personas.
|
||||
|
||||
### R2-5: Route evidence completion
|
||||
|
||||
Source finding: P1-7 / T11.
|
||||
|
||||
Current evidence:
|
||||
|
||||
- `docs/audits/2026-07-08-route-evidence-t11-analysis.md` now includes a current all-route built-preview smoke on port 3457.
|
||||
- Artifacts live under `output/playwright/route-evidence-3457/`.
|
||||
- The smoke proves 33 desktop route navigations plus 11 mobile route spot-checks return 200, produce screenshots, and have no document-level horizontal overflow.
|
||||
- `/payment-cancelled` redirects to `/settings?tab=billing`.
|
||||
- T11 remains open because this evidence is ad hoc, every sampled route logs the Clerk development-key warning, `/launcher?watch=1` logs detectTools network noise, `/settings/usage` logs a 403 resource error, the catch-all route logs its expected 404 as a console error, and several judge-route controls still lack accessible names/labels.
|
||||
|
||||
Surfaces:
|
||||
|
||||
- Any route still marked Thin or Missing after Phase 1.
|
||||
- Major overlays not exercised by Phase 1.
|
||||
|
||||
Acceptance:
|
||||
|
||||
- Every route and major overlay has a smoke, visual, or persona evidence owner.
|
||||
- Deferrals are explicit and approved.
|
||||
|
||||
Judge impact:
|
||||
|
||||
- All personas.
|
||||
|
||||
### R2-6: Shell overlay semantics and create-workspace hierarchy
|
||||
|
||||
Source findings: P1-19 / T10/T12 and P1-20 / T10/T12/T7.
|
||||
|
||||
Focused supplement: `docs/audits/2026-07-08-shell-overlays-t10-t12-analysis.md`.
|
||||
|
||||
Surfaces:
|
||||
|
||||
- Notification Inbox.
|
||||
- Create Workspace.
|
||||
- Context Rail.
|
||||
- Onboarding Tooltips.
|
||||
- Upgrade and Trial Expired modals.
|
||||
|
||||
Current evidence update:
|
||||
|
||||
- Fresh desktop/mobile overlay smoke passes the high-level open/close path for Keyboard Shortcuts, Persona Switcher, Workspace Switcher, Spawn Agent, and Upgrade Modal.
|
||||
- Notification Inbox now has a named dialog contract, focus trap, Escape close, and named mark-all/close actions.
|
||||
- Create Workspace now has named primary/subdialog contracts, focus traps, Escape close, named sampled template/share actions, in-app custom-template delete confirmation, and focused 390 x 844 rendered evidence that required setup precedes optional templates.
|
||||
- Context Rail now has a labelled complementary contract and named close action at component level; route-specific rendered states still need evidence.
|
||||
- Onboarding Tooltips now has an explicit non-modal dialog contract and Escape dismissal at component level; rendered first-run/mobile evidence still needs refresh.
|
||||
- Upgrade and Trial Expired close icons now have accessible names; the event-driven Upgrade modal close path has rendered `J3c` evidence.
|
||||
- Browser-plugin spot check for this slice failed on the in-app Browser DOM snapshot path (`incrementalAriaSnapshot is not a function`), so Playwright remains the reliable rendered evidence lane until P2-2 is fixed.
|
||||
|
||||
Acceptance:
|
||||
|
||||
- Modal overlays share dialog semantics, focus behavior, Escape close, focus return, and named close/icon actions.
|
||||
- Non-modal panels use labelled landmarks and predictable keyboard reachability.
|
||||
- Create Workspace prioritizes required name/storage/create controls on the sampled mobile path, with templates progressive or summarized.
|
||||
- Template destructive actions use in-app confirmation and result states.
|
||||
- Desktop and 390 x 844 overlay screenshots plus DOM/keyboard evidence are attached; current gap is refreshed screenshot/state coverage beyond the focused Create Workspace path.
|
||||
|
||||
Judge impact:
|
||||
|
||||
- Mobile executive.
|
||||
- Engineer.
|
||||
- Solo founder.
|
||||
- Team admin when workspace/template management is in scope.
|
||||
|
||||
### R2-7: Admin web and CLI/MCP utility evidence
|
||||
|
||||
Source finding: P1-11 / T15.
|
||||
|
||||
Surfaces:
|
||||
|
||||
- `packages/admin-web`.
|
||||
- `packages/launcher` / `npx waggle`.
|
||||
- `packages/cli`.
|
||||
- `packages/marketplace` CLI.
|
||||
- `packages/memory-mcp` and `packages/hive-mind-mcp-server`.
|
||||
- `packages/hive-mind-cli`.
|
||||
|
||||
Current evidence update:
|
||||
|
||||
- T15 deep-dive: `docs/audits/2026-07-08-admin-cli-utility-t15-analysis.md`.
|
||||
- Targeted source tests, no-emit TypeScript checks, and package builds pass for the inspected admin/CLI/MCP utility surfaces.
|
||||
- Admin-web built-render smoke covers all 7 desktop pages and all 7 mobile pages under `output/playwright/admin-web-t15-57795/`; typed happy paths render with no current-port console errors. A newer package-owned rendered Playwright gate builds the admin web and passes all seven pages across desktop/mobile widths, hash deep links, browser back/forward traversal, `aria-current`, labelled table scroll regions, rendered control labels, mobile shell keyboard reachability, page-level keyboard traversal from connection fields into the covered admin pages, desktop/mobile visual snapshots, capability governance forms, malformed analytics response recovery, all-page initial API-failure recovery, rendered mutation/destructive-failure recovery for capability policy save, capability override create/remove, capability request decision, member invite, member role change, member removal, and team settings save, local bearer-auth wrong-token/valid-token behavior, and React warning cleanup.
|
||||
- Built legacy `waggle-memory-mcp` read-only/write-scope protocol startup plus installed read-only startup, built hive-mind MCP read-only/write-scope protocol startup plus installed read-only startup, built/packed/local package-closure installed `@waggle/cli` `npx` help plus installed local REPL startup/slash-command/exit and streamed chat/provider plumbing, launcher built help/invalid-port/occupied-port recovery plus packed first-command, clean installed occupied-port startup recovery, and clean installed long-running `/health` startup, built/installed marketplace CLI help/invalid-command behavior plus package manifest/packed-file alignment, and local package-closure installed `hive-mind-cli` help are locally fixed, but still need registry-only proof after internal package publication.
|
||||
- Built hive-mind MCP passes official MCP client read-only registration and write-scope save/recall roundtrips with 9 tools and 4 resources.
|
||||
- Package-local test commands remain inconsistent outside the now-documented hive-mind CLI and hook/shim lanes. Root Vitest does not discover hive-mind CLI colocated tests under `src/`, but `npm run test --workspace @waggle/hive-mind-cli -- --reporter=dot` now owns that lane. Marketplace invalid-command behavior and sampled hive-mind CLI subcommand help are now locally fixed and guarded.
|
||||
|
||||
Acceptance:
|
||||
|
||||
- Admin-web rendered desktop/mobile happy-path, browser-history, page-level keyboard, visual snapshot, malformed analytics recovery, initial API-failure, mutation/destructive-failure, and local bearer-auth states stay green in the package-owned gate.
|
||||
- Built utility entries run from a clean environment, are explicitly deprecated, or are deferred.
|
||||
- Utility CLIs have happy-path and error-path command evidence, or are deferred.
|
||||
- Hive-mind CLI package-local test discovery remains in the documented verification lane; the remaining package-local script gaps are fixed or have documented separate verification lanes.
|
||||
|
||||
Judge impact:
|
||||
|
||||
- Team admin.
|
||||
- Engineer.
|
||||
- Solo founder setup path.
|
||||
|
||||
### R2-8: AI-tool hook lifecycle evidence
|
||||
|
||||
Source finding: P1-12 / T16.
|
||||
|
||||
Surfaces:
|
||||
|
||||
- Launcher hook install/verify/uninstall UI.
|
||||
- `packages/hive-mind-hooks-claude-code`.
|
||||
- `packages/hive-mind-hooks-codex`.
|
||||
- `packages/hive-mind-hooks-codex-desktop`.
|
||||
- `packages/hive-mind-hooks-cursor`.
|
||||
- `packages/hive-mind-hooks-hermes`.
|
||||
- `packages/hive-mind-hooks-openclaw`.
|
||||
- `packages/hive-mind-hooks-claude-desktop` unsupported/stub copy.
|
||||
|
||||
Current evidence update:
|
||||
|
||||
- Shared/agent/server T16 route-contract plus package-runtime tests pass 7 files / 96 tests, and the focused registry-aware adapter launch slice passes 5 files / 108 tests.
|
||||
- Packed-package `npx --yes @waggle/hive-mind-hooks-<id> install/verify/uninstall` lifecycle passes for all six hook-capable packages from a clean temp project with the local package closure installed.
|
||||
- Launcher/prompt/adapter tests pass 3 files / 28 tests.
|
||||
- Root-run hook/shim package tests pass 60 files / 573 tests with 1 skipped.
|
||||
- Hook/shim package-local `npm run test --workspace ...` scripts now pass for hook core, all six hook-capable packages, and shim core; the shim lane also verifies the CLI ESM resolver fix for the MCP server entry.
|
||||
- Official hook/shim package typechecks pass 8/8, the Claude Desktop stub build passes, and compiled hook-bin help smokes pass 6/6.
|
||||
- Focused rendered Launcher Browser smoke artifacts live under `output/playwright/launcher-t16-54147/`; they prove mixed hook states, prompt summary, install success, verify failure, and running output against a mock local API, while exposing the original generic/dropped-detail result copy, missing Claude Desktop launch-only copy, and prompt metadata gaps. Duplicate live-output replay after exit, hook stdout/stderr detail visibility, Backup/Recovery labels for covered install output, Check-failed/manual-approval rows for Verify output, uninstall restore/cleanup labels, More-output summarization, structured hook failure preservation, empty-output Verify recovery copy, and Claude Desktop launch-only copy are now covered by focused web regressions; a fresh real `/launcher` Browser smoke now renders `verify failed (exit 1)` with retry/uninstall/reinstall guidance instead of `HTTP 400`, Browser-rendered mocked states now cover Claude Desktop launch-only copy, Codex Verify manual approval, and Codex Uninstall cleanup, a codified rendered Playwright spec covers standard install changed-file/pointer/backup/recovery labels, all six hook-capable install/verify/uninstall transitions, sidecar-offline Retry, long-stderr summarization, and non-built-in adapter launch-only/prompt behavior, a gated real-tool Playwright spec covers OpenClaw rendered detection plus observed launch/output/exit/process cleanup, and a gated route Playwright spec covers real `/api/tools/hooks` install/verify/uninstall for all six hook-capable tools against an isolated profile. Packaged desktop hook-status transitions remain open rather than a generic status leak.
|
||||
- Tailwind motion-token warning hygiene is partially fixed: ambiguous `duration-[var(--mo-*)]` / `ease-[var(--mo-ease)]` classes are replaced by named utilities and guarded by `motion-class-hygiene.test.ts`; the `shape-selection.ts` dynamic/static import warning is also fixed and guarded by `build-warning-hygiene.test.ts`; the Vite large-chunk warning is fixed by lazy-loading routes, closed shell overlays, ChatHost, and PostHog analytics. Remaining warning hygiene still includes color-env, embedding, and hook negative-path noise.
|
||||
- Focused evidence and open gaps are recorded in `docs/audits/2026-07-08-ai-tool-hook-t16-analysis.md`.
|
||||
|
||||
Acceptance:
|
||||
|
||||
- Rendered Launcher evidence covers detected/not-detected, standard install changed-file/pointer labels, verify, uninstall, all six hook-capable install/verify/uninstall transitions, sidecar offline, long-stderr, error, unsupported Claude Desktop states, and at least one real observed CLI launch; packaged desktop hook-status evidence is still required for T16 closure.
|
||||
- Command lifecycle evidence covers six hook-capable packages through the production-like package-name invocation path; route lifecycle evidence covers all six hook-capable tools through safe isolated install/verify/uninstall smokes.
|
||||
- Registry-only proof is captured after actual hook package publication.
|
||||
- Observed live output does not duplicate replay after terminal exit, hook result stdout/stderr/Backup-Recovery/Check-failed/structured-failure/empty-output recovery details are preserved, and Claude Desktop launch-only copy is explicit; current focused evidence passes in the Launcher/web suite.
|
||||
- Third-party adapter detection/launch behavior is fixed or explicitly out of scope.
|
||||
- Expected fail-open/log noise is quiet enough or explicitly documented.
|
||||
|
||||
Judge impact:
|
||||
|
||||
- Engineer.
|
||||
- Solo founder.
|
||||
- Team admin.
|
||||
|
||||
### R2-9: Developer API, background worker, and substrate verification evidence
|
||||
|
||||
Source finding: P1-13 / T17.
|
||||
|
||||
Dedicated supplement: `docs/audits/2026-07-08-developer-substrate-t17-analysis.md`.
|
||||
|
||||
Surfaces:
|
||||
|
||||
- `packages/sdk`.
|
||||
- `packages/server`.
|
||||
- `packages/worker`.
|
||||
- `packages/waggle-dance`.
|
||||
- `packages/agent`, `packages/core`, and `packages/shared`.
|
||||
- `packages/hive-mind-core`, `packages/hive-mind-shim-core`, `packages/hive-mind-wiki-compiler`, and `packages/wiki-compiler`.
|
||||
- Package-local test scripts and root Vitest discovery.
|
||||
|
||||
Current evidence update:
|
||||
|
||||
- Direct no-emit TypeScript passes for 13 remaining backend/developer/substrate workspaces.
|
||||
- Agent tests pass 194 files / 3079 tests.
|
||||
- Core, optimizer, and weaver tests pass through package scripts.
|
||||
- Root-run SDK, substrate, shared, worker, WaggleDance, and compiler tests pass.
|
||||
- Current full server route suite passes, but it is slow/noisy; earlier packet evidence saw one workspace-listing perf budget failure under full-suite load while the same benchmark passed in isolation.
|
||||
- Several non-hook package-local `npm test` scripts fail by command shape even though root-run tests pass.
|
||||
- Marketplace sync behavior leaks into the normal server verification lane and should be made hermetic or moved to a named live-integration lane.
|
||||
|
||||
Acceptance:
|
||||
|
||||
- Package-local scripts either pass or clearly delegate to the correct root/project-reference lane.
|
||||
- Full server suite is deterministic, or perf assertions are separated into an explicit perf lane with a stable threshold.
|
||||
- SDK/server/worker/WaggleDance/substrate package verification is discoverable and quiet enough for release review.
|
||||
- Developer API/background/substrate evidence is attached or explicitly deferred from the five-persona score.
|
||||
|
||||
Judge impact:
|
||||
|
||||
- Engineer.
|
||||
- Team admin.
|
||||
- Release confidence for all personas.
|
||||
|
||||
### R2-10: Ops, deployment, CI, benchmark, and judging evidence
|
||||
|
||||
Source finding: P1-14 / T18.
|
||||
|
||||
Dedicated supplement: `docs/audits/2026-07-08-ops-deploy-ci-judging-t18-analysis.md`.
|
||||
|
||||
Surfaces:
|
||||
|
||||
- `.github/workflows`.
|
||||
- `Dockerfile`.
|
||||
- `docker-compose.yml` and `docker-compose.production.yml`.
|
||||
- `render.yaml`.
|
||||
- `litellm-config.yaml` and `ops/litellm`.
|
||||
- `vitest.infra.config.ts` and `vitest.infra-suites.ts`.
|
||||
- `benchmarks/harness`.
|
||||
- `judging`.
|
||||
|
||||
Current evidence update:
|
||||
|
||||
- YAML parse passes for Compose, Render, LiteLLM, and all workflow files.
|
||||
- Docker CLI is available and Compose config expansion succeeds for development and production files, but the Docker Desktop engine is unavailable in this environment, so live infra tests were not run.
|
||||
- Raw Compose config can print local ignored env secrets; use non-interpolated or sanitized output for shareable evidence.
|
||||
- Only `.env.example` files are tracked among the checked env/key files; local secret-bearing files are ignored.
|
||||
- Render currently provisions Postgres/Redis but starts the local sidecar path.
|
||||
- CI E2E is advisory, and no CI lane runs the 19 infra-dependent suites.
|
||||
- Benchmark harness TypeScript passes; root-run benchmark tests pass 29 files / 325 tests; package-local benchmark tests fail by command shape.
|
||||
- Current `judging/` artifacts are historical, not the final July five-persona score evidence.
|
||||
- The focused T18 supplement also records the `apps/www` GitHub Pages artifact mismatch, production Compose default credentials, and LiteLLM live-routing gap.
|
||||
|
||||
Acceptance:
|
||||
|
||||
- Secret-safe ops validation commands are documented and used.
|
||||
- Docker/Render target and entrypoint choices are coherent with the product mode being claimed.
|
||||
- CI blocking/advisory semantics are explicit; infra lane is run, added, or deferred with Docker-engine availability noted.
|
||||
- Benchmark harness commands are fixed or documented.
|
||||
- Current five-persona judging artifacts are regenerated after approved fixes land.
|
||||
|
||||
Judge impact:
|
||||
|
||||
- Engineer.
|
||||
- Team admin.
|
||||
- Release confidence for all personas.
|
||||
|
||||
### R2-11: Browser Companion extension evidence
|
||||
|
||||
Source finding: P1-15 / T19.
|
||||
|
||||
Surfaces:
|
||||
|
||||
- `apps/browser-ext/manifest.json`.
|
||||
- `apps/browser-ext/popup.html`.
|
||||
- `apps/browser-ext/popup.js`.
|
||||
- `apps/browser-ext/background.js`.
|
||||
- `apps/browser-ext/content.js`.
|
||||
- `packages/server/src/local/routes/browser-ext.ts`.
|
||||
- `packages/server/src/local/cors-config.ts`.
|
||||
|
||||
Current evidence update:
|
||||
|
||||
- Source inventory confirms `apps/browser-ext` is a Chrome MV3 extension surface, not an npm workspace.
|
||||
- Server exposes `/api/browser-ext/health` and CORS env gates for extension origins.
|
||||
- Direct sidecar save evidence proves selection-shaped and page-shaped Browser Companion frames can be created, duplicate detection works, and Memory UI renders the saved imported frame.
|
||||
- A real unpacked Chromium extension run originally proved background save failed by default with generic `HTTP 500` when the extension origin was not allowlisted. The current secure-default live smoke now proves content-script extraction from a normal page, MV3 no-Origin header handling, token bootstrap during save, background save through `chrome.runtime.sendMessage`, and `/api/memory/frames` imported-frame confirmation under `WAGGLE_BROWSER_EXT_IDS=<extension id>`.
|
||||
- The direct popup keyboard/click live smoke now proves popup Tab order, visible Save selection focus, Enter-to-save selection, Save page click, saved feedback, Memory frame creation through the secure sidecar path, and `/api/memory/search` `source: import` when supplied the active target tab that the native toolbar popup would receive from Chrome. Context-menu registration/handler behavior is regression-covered.
|
||||
- Native toolbar-bubble exposure, native context-menu clicks, packaged pairing, and any separately scored global-recall provenance shape are still unproven.
|
||||
- Historical `/api/memory/frames` versus `/api/memory/search` source/provenance disagreement is fixed for imported Browser Companion captures.
|
||||
|
||||
Acceptance:
|
||||
|
||||
- Connected/disconnected popup states are screenshot or test evidenced.
|
||||
- Save selection and save page create Memory frames from popup keyboard/click interactions or have an approved deferral; native toolbar-bubble proof can be manual if automation cannot expose it.
|
||||
- Native context-menu save creates a Memory frame or has an approved deferral.
|
||||
- CORS/auth-denied/unpaired-extension state gives a clear setup path instead of a generic `HTTP 500` or raw `MISSING_TOKEN`; focused coverage exists, with live screenshot evidence still optional for the judge packet.
|
||||
- Memory provenance remains consistent between frame list, search result, UI, and any separate global-recall result shape included in scoring.
|
||||
- Popup keyboard/focus basics have live evidence; status announcement behavior is separately checked or deferred if scored.
|
||||
- Browser Companion is either evidenced or explicitly deferred from the five-persona score.
|
||||
|
||||
Judge impact:
|
||||
|
||||
- Researcher.
|
||||
- Solo founder.
|
||||
- Mobile executive where browser capture is part of the scenario.
|
||||
|
||||
### R2-12: State and failure bundle evidence
|
||||
|
||||
Source finding: P1-8 / T12.
|
||||
|
||||
Surfaces:
|
||||
|
||||
- `docs/audits/2026-07-08-ux-state-failure-scenario-matrix.md`.
|
||||
- Five-persona scorecards.
|
||||
- Routes and overlays exercised under account, tier, disclosure, model, data, offline, and viewport variants.
|
||||
|
||||
Acceptance:
|
||||
|
||||
- Each judge scorecard declares the exact state bundle it exercised.
|
||||
- Account mode, billing tier, disclosure tier, model state, data state, offline/error state, and viewport are present in the evidence.
|
||||
- Skipped state bundles have explicit deferral rationale before scoring.
|
||||
|
||||
Judge impact:
|
||||
|
||||
- All personas.
|
||||
|
||||
### R2-13: Public launch funnel repair and evidence
|
||||
|
||||
Source finding: P0-L1 / T13.
|
||||
|
||||
Focused supplement: `docs/audits/2026-07-08-launch-funnel-t13-analysis.md`.
|
||||
|
||||
Surfaces:
|
||||
|
||||
- `apps/www` homepage.
|
||||
- Canonical `waggle-os.ai` DNS/deployment.
|
||||
- Download CTA.
|
||||
- GitHub Releases or replacement download/status page.
|
||||
- Sign-in, sign-up, account.
|
||||
- Team checkout and checkout cancel recovery.
|
||||
- Legal/privacy/cookies/EU AI Act pages.
|
||||
- `deploy-www.yml` or replacement deployment target.
|
||||
|
||||
Acceptance:
|
||||
|
||||
- Download CTA leads to valid Windows/macOS artifacts or a controlled availability/status page; mobile and unsupported OS labels are honest.
|
||||
- `https://waggle-os.ai/` and `https://www.waggle-os.ai/` resolve and serve the selected public deployment or redirect coherently.
|
||||
- Local production route/API smoke uses the correct host binding and serves the public launch, legal, auth, account, methodology, and checkout routes without 500s, timeouts, unexpected 404s, or Clerk redirect-loop spam.
|
||||
- Launch funnel is verified with desktop/mobile route evidence, or explicitly deferred from the five-persona score.
|
||||
- Signed-out Team checkout guides the user into auth/checkout recovery instead of a JSON-shaped error.
|
||||
- Checkout cancel returns to a real pricing recovery state.
|
||||
- Public legal/trust pages contain no placeholder launch text and no active Pro copy.
|
||||
- Deployment target matches the app shape.
|
||||
|
||||
Current evidence:
|
||||
|
||||
- www tests, direct TypeScript, and Next build pass.
|
||||
- Current external refresh cannot resolve `waggle-os.ai` or `www.waggle-os.ai` from the audit environment; DNS reports non-existent domain/no A or CNAME.
|
||||
- Fresh localhost route/API smoke serves core routes, but `/pricing?checkout=cancelled` still 404s.
|
||||
- Fresh rendered Browser smoke under `output/playwright/www-t13-3491/` captures desktop/mobile homepage, mobile menu, signed-out pricing error, route/API probes, and clean sampled console logs; it still reproduces broken Download and signed-out checkout recovery.
|
||||
- Public download currently points to GitHub Releases, where the repo has no releases.
|
||||
- Current workflow uploads `apps/www/dist` while the app builds `.next`.
|
||||
|
||||
Judge impact:
|
||||
|
||||
- Solo founder.
|
||||
- Team admin.
|
||||
- Mobile executive.
|
||||
|
||||
### R2-14: Desktop wrapper and release UX evidence
|
||||
|
||||
Source finding: P1-10 / T14.
|
||||
|
||||
Surfaces:
|
||||
|
||||
- Packaged Tauri launch.
|
||||
- Sidecar readiness and failure recovery.
|
||||
- Tray menu actions.
|
||||
- Close-to-tray and global shortcut.
|
||||
- Installer/signing expectations.
|
||||
- Update notification or deliberate hidden/disabled update state.
|
||||
|
||||
Current evidence update:
|
||||
|
||||
- Static Tauri config/update/tray tests, app TypeScript, web typecheck, Rust `cargo check`, sidecar resource preflight, app service E2E, release workflow ordering, and web-side Tauri binding tests now pass; Rust source directly handles tray Open/focus, close-to-tray, `Ctrl+Shift+W`, and Quit.
|
||||
- Settings routes through a tested `/settings` desktop bridge; Pause Agents and About Waggle are hidden; remaining unconsumed native events are update/service status and still need visible UX or deferral.
|
||||
- This remains Phase 2/launch work unless a Phase 1 verification command directly needs a small supporting fix.
|
||||
|
||||
Acceptance:
|
||||
|
||||
- Installed app launch and service recovery are verified, or explicitly deferred from the five-persona score.
|
||||
- Tray/update/service events are either consumed by UI, handled natively, or hidden until supported; Quit actually exits or is not shown as a supported command.
|
||||
- Release channel expectations are clear for signed/trusted install.
|
||||
|
||||
Judge impact:
|
||||
|
||||
- All personas.
|
||||
|
||||
## Phase 3: Performance, Cost Semantics, and Polish
|
||||
|
||||
Goal: improve speed, clarity, and polish after core trust/flow issues are under control.
|
||||
|
||||
### R3-1: Initial payload and route loading
|
||||
|
||||
Source finding: P1-4 / T8.
|
||||
|
||||
Surfaces:
|
||||
|
||||
- App shell.
|
||||
- Deep Settings/Marketplace/admin surfaces.
|
||||
- Keep `shape-selection.ts` import hygiene guarded while reducing the remaining initial payload.
|
||||
- Large persona/logo assets.
|
||||
|
||||
Acceptance:
|
||||
|
||||
- Startup JS stays below the Vite 500 kB warning threshold.
|
||||
- Deep apps, closed overlays, chat, and analytics load after core shell where feasible.
|
||||
- Defeated dynamic import and oversized-JS warnings remain resolved or explicitly justified.
|
||||
- Large persona/logo assets are optimized or deferred.
|
||||
|
||||
Judge impact:
|
||||
|
||||
- Mobile executive.
|
||||
- Engineer.
|
||||
- Solo founder first impression.
|
||||
|
||||
### R3-2: Local model pricing semantics
|
||||
|
||||
Source finding: P1-6 / T9.
|
||||
|
||||
Surfaces:
|
||||
|
||||
- `packages/agent/src/cost-tracker.ts`.
|
||||
- Usage & Cost UI.
|
||||
- No-LLM/degraded-mode tests.
|
||||
|
||||
Acceptance:
|
||||
|
||||
- Local/unpriced models are labeled honestly.
|
||||
- No standard audit warning estimates local model cost as paid-provider cost without explanation.
|
||||
- Tests that claim no-LLM behavior actually force provider unavailability or are renamed.
|
||||
|
||||
Judge impact:
|
||||
|
||||
- Engineer.
|
||||
- Team admin.
|
||||
|
||||
### R3-3: Stale comments and legacy terminology
|
||||
|
||||
Source finding: P2-1.
|
||||
|
||||
Surfaces:
|
||||
|
||||
- Touched billing/tier files.
|
||||
- Touched route/shortcut files.
|
||||
- Touched Marketplace/MCP/skills copy files.
|
||||
|
||||
Acceptance:
|
||||
|
||||
- Comments in touched files reflect current Solo/Teams/Enterprise and route behavior.
|
||||
- No unrelated cleanup outside touched files.
|
||||
|
||||
Judge impact:
|
||||
|
||||
- Indirect.
|
||||
|
||||
### R3-4: Browser plugin DOM snapshot tooling
|
||||
|
||||
Source finding: P2-2.
|
||||
|
||||
Surfaces:
|
||||
|
||||
- Audit tooling, not product.
|
||||
|
||||
Acceptance:
|
||||
|
||||
- Keep Playwright CLI as the reliable audit path until Browser `domSnapshot()` mismatch is fixed.
|
||||
- If Browser tooling is fixed, update the audit process and evidence packet.
|
||||
|
||||
Judge impact:
|
||||
|
||||
- Audit ergonomics only.
|
||||
|
||||
## Re-Scoring Sequence
|
||||
|
||||
1. Finish Phase 1 and update the correction register.
|
||||
2. Attach route, state/failure, launch-funnel, Browser Companion, desktop-wrapper, utility/admin, hook lifecycle, developer/substrate, and ops/deployment/judging evidence owners to the five scorecards.
|
||||
3. Run a dry five-persona scorecard pass.
|
||||
4. If any persona is capped by a Phase 2 item, execute only the relevant Phase 2 lane.
|
||||
5. Repeat scorecard pass.
|
||||
6. Execute Phase 3 only after trust, routing, accessibility, route evidence, state-bundle evidence, and any non-deferred launch/extension/desktop/utility/hook/developer/ops gates stop capping scores.
|
||||
|
||||
## Approval Boundary
|
||||
|
||||
This roadmap is not a request to implement Phase 2/3 now. It exists so the remaining correction work is sequenced after Phase 1 and no P1/P2 item is lost.
|
||||
319
docs/audits/2026-07-08-ux-route-scenario-manifest.md
Normal file
319
docs/audits/2026-07-08-ux-route-scenario-manifest.md
Normal file
@@ -0,0 +1,319 @@
|
||||
# Waggle OS UX Route and Scenario Manifest
|
||||
|
||||
Companion artifact for `docs/audits/2026-07-08-complete-ux-usage-audit.md`.
|
||||
|
||||
Five-persona scoring packet: `docs/audits/2026-07-08-five-persona-judge-scorecards.md`.
|
||||
|
||||
Five-persona execution runbook: `docs/audits/2026-07-08-five-persona-judge-runbook.md`.
|
||||
|
||||
Master correction register: `docs/audits/2026-07-08-ux-correction-register.md`.
|
||||
|
||||
State and failure matrix: `docs/audits/2026-07-08-ux-state-failure-scenario-matrix.md`.
|
||||
|
||||
Focused T11 route evidence supplement: `docs/audits/2026-07-08-route-evidence-t11-analysis.md` (now includes a current all-route built-preview smoke on port 3457 and codified `J-route-coverage` evidence on port `34200`).
|
||||
|
||||
Focused Mobile Executive T2/T12 supplement: `docs/audits/2026-07-08-mobile-executive-t2-t12-analysis.md`.
|
||||
|
||||
Focused runtime accessibility T10 supplement: `docs/audits/2026-07-08-runtime-a11y-t10-analysis.md`.
|
||||
|
||||
Focused first-run onboarding T1/T2/T12 supplement: `docs/audits/2026-07-08-first-run-onboarding-t1-t2-t12-analysis.md`.
|
||||
|
||||
Focused shell overlay T10/T12 supplement: `docs/audits/2026-07-08-shell-overlays-t10-t12-analysis.md`.
|
||||
|
||||
Non-main surface scope: `docs/audits/2026-07-08-ux-non-main-surface-scope.md`.
|
||||
|
||||
Source inventory consistency audit: `docs/audits/2026-07-08-source-inventory-consistency-audit.md`.
|
||||
|
||||
Phase 1 implementation plan and verification log: `docs/superpowers/plans/2026-07-08-ux-phase-1-corrections.md`.
|
||||
|
||||
Purpose: make the final 9/10 UX claim auditable. Every registered route, major overlay, and cross-cutting scenario needs an evidence owner before the five-persona judge gate is run.
|
||||
|
||||
Current Phase 1 evidence update: the approved Phase 1 corrections are implemented and the full combined browser gate passed 156/156 on port `34150`. This includes accountless console health, mobile Settings, mobile first-run onboarding, `Ctrl+Shift+N`, Workspace Switcher open/close behavior, thin route smokes, and visual snapshots. Phase 2 has also begun: Notification Inbox and Create Workspace overlay contracts passed component and rendered journey evidence on ports `34151`/`34152`; the next overlay slice adds component evidence for Context Rail, Onboarding Tooltips, and tier close labels plus rendered `J3c` tier-modal evidence on port `34153`, with the earlier expanded full user-journey suite passing 18/18 on port `34155`. The Create Workspace mobile hierarchy slice adds component evidence, focused 390 x 844 rendered evidence on port `34157`, and expanded full user-journey evidence passing 19/19 on port `34158`. Rows below keep `Mixed` where deeper workflow, destructive-dialog, accessibility, mobile-overlay, or non-main-surface evidence is still required before the final 9/10 judge pass.
|
||||
|
||||
Status key:
|
||||
|
||||
- `Strong`: current tests or rendered inspection exercise the route and at least one primary interaction.
|
||||
- `Mixed`: route has coverage, but known failures, flakes, visual drift, or thin interaction depth remain.
|
||||
- `Thin`: route is registered and has only shallow/static/string evidence.
|
||||
- `Missing`: route has no direct route-string coverage in `tests/` or `apps/web/src/test`.
|
||||
|
||||
Global route acceptance checks:
|
||||
|
||||
1. Route loads a meaningful surface, not a blank shell.
|
||||
2. No framework overlay or critical console error.
|
||||
3. Primary visible control works and produces an observable state change.
|
||||
4. Empty, loading, error, and permission states are understandable.
|
||||
5. Keyboard focus is visible and ordered through primary controls.
|
||||
6. Runtime axe/DOM findings on judge routes have no critical or serious unapproved violations.
|
||||
7. Mobile 390 px layout has no horizontal overflow or clipped critical controls; critical visible element bounds must be checked because document scroll width alone can miss clipping.
|
||||
8. First-run onboarding is checked on a clean data dir without skip flags for console health, mobile primary-action reachability, and terminal first-task handoff.
|
||||
9. User-facing copy matches current Solo, Teams, Enterprise strategy.
|
||||
10. Destructive or trust-critical actions use in-app confirmation and result states.
|
||||
11. URL state is stable enough for reload/back/forward on tabs, filters, and route context.
|
||||
12. Route has an evidence owner in the final judge packet.
|
||||
|
||||
## Route Manifest
|
||||
|
||||
| Area | Route | Primary component | User job | Current evidence | Status | Must prove before 9/10 |
|
||||
|---|---|---|---|---|---|---|
|
||||
| Auth | `/auth` | `AuthRoute`, `WaggleClerkProvider` | Sign in or continue accountless | 9 route-string hits; Phase 1 console health passes with Clerk disabled by default unless `VITE_WAGGLE_ENABLE_CLERK=1` | Mixed | Auth mode still needs an intentional-Clerk enabled lane; accountless local mode is clean in the standard audit lane. |
|
||||
| Shell index | `/` -> `/home` | `IndexRedirect`, `AppShell` | Land in the product | Covered by shell/user journeys | Strong | Redirect is stable and does not flash invalid shell state. |
|
||||
| Home | `/home` | `HomeCockpit` | Continue work, Start Here, create/open workspace | 81 route-string hits; rendered desktop/mobile inspection | Strong | Preserve Home as the continuity anchor; no new surface for existing Home jobs. |
|
||||
| Workspaces | `/workspaces` | `AllWorkspacesApp` | Browse, create, manage workspaces | 115 route-string hits | Mixed | Workspace creation/deletion/manage flows have in-app confirmations and mobile fit. |
|
||||
| Workspace | `/workspaces/:workspaceId/:tab?` | `WorkspaceDesktopApp`, `ChatApp`, `TasksTab` | Chat, files, tasks, workspace memory | 41 nested route hits; chat tests; Phase 1 `Ctrl+Shift+N` and active-workspace fallback pass in combined browser gate | Mixed | Mobile workspace tabs are in-bounds or intentionally scrollable; deeper files/tasks/memory states remain evidence work. |
|
||||
| Memory | `/memory/:mindScope?` | `MemoryCenterApp` | Search, trust, provenance, wiki, timeline | 138 route-string hits; T5 visual failure classified mostly as stale-baseline drift; fresh 390 x 844 smoke shows mobile tab-strip overflow; rendered `J3f` delete/erase/re-import confirmations pass on port `34164`; rendered `J3g` Wiki export forms pass on port `34167`; latest expanded user journeys pass 24/24 on port `34173` | Mixed | Memory Center destructive trust flow and Wiki export destinations use in-app consequence/form modals; mobile tab-strip behavior, visual baseline, failure states, and broader persona states still need final evidence. |
|
||||
| Artifacts | `/artifacts` | `ArtifactCenterApp` | Review, archive, delete produced artifacts | Codified `J-route-coverage: priority thin routes` renders Artifact/Library shell; rendered `J3e` permanent-delete confirmation passes on port `34161`; latest expanded user journeys pass 24/24 on port `34173` | Mixed | Permanent delete uses in-app consequence modal; archive, empty/error, and broader artifact persona states still need final evidence. |
|
||||
| Files | `/files` | `StorageAndFilesApp` | Browse, upload, preview, download files | Codified `J-route-coverage: priority thin routes` renders storage/files shell; passed 2026-07-08 on port `34200` | Mixed | Upload, preview, empty/error states and keyboard navigation are covered. |
|
||||
| Agents | `/agents` | `AgentsApp` | Create/manage agents and groups | 12 route-string hits | Mixed | Agent creation forms have labels, errors, and no noisy `act` warnings. |
|
||||
| Automations | `/automations` | `AutomationCenterApp` | Build and manage scheduled jobs | Codified `J-route-coverage: priority thin routes` renders Automation Center shell; delete confirmation has focused component evidence in `phase3b-automation-center.test.tsx` | Mixed | Builder validation, logs, pause/resume, failure, and broader persona states remain. |
|
||||
| Skills | `/skills` | `CapabilitiesApp` | Inspect/install skills | 50 route-string hits; T5 visual failure classified mostly as stale-baseline drift with T10 row/action follow-up | Mixed | Pro copy removed; install/update/error states are deterministic; approved visual baseline passes. |
|
||||
| Room | `/room` | `RoomApp` | Observe parallel agent work | 12 route-string hits | Mixed | Empty/running/completed states and explanatory affordances are covered. |
|
||||
| WaggleDance | `/waggle-dance` | `WaggleDanceApp` | Understand signal sharing/swarm behavior | Codified `J-route-coverage: priority thin routes` renders signal-sharing shell; passed 2026-07-08 on port `34200` | Mixed | Value clarity and live signal states are added. |
|
||||
| Approvals | `/approvals` | `ApprovalsApp` | Review grants/actions and revoke permissions | Rendered `J3d` revoke-all in-app confirmation path passes on port `34159`; latest expanded user journeys pass 24/24 on port `34173` | Mixed | Revoke-all uses in-app consequence modal and result state; approve/deny and individual revoke states still need final persona-bundle evidence. |
|
||||
| Connectors | `/connectors` | `ConnectorsApp` | Connect external systems | 35 route-string hits | Mixed | Credential setup, revoke, error recovery, and no-secrets display are tested. |
|
||||
| MCP Hub | `/mcps` | `MCPHubApp` | Manage MCP servers | Codified `J-route-coverage: priority thin routes` renders MCP Hub installed/catalog/custom shell; passed 2026-07-08 on port `34200` | Mixed | Solo/Team copy, custom MCP form labels, scope dialog, install/verify states covered. |
|
||||
| Marketplace | `/marketplace` | `MarketplaceApp` | Browse/install extensions | 53 route-string hits; one flake | Mixed | Standard audit does not hit live external sync; search/browse are stable. |
|
||||
| Launcher | `/launcher` | `LauncherApp` | Launch AI tools and manage hooks | Codified `J-route-coverage: priority thin routes` renders `/launcher` and `/launcher?watch=1`; passed 2026-07-08 on port `34200` | Mixed | Clean-install runtime verification, launch-with-prompt, hook install/verify/uninstall covered. |
|
||||
| Team | `/team` | `TeamGovernanceApp` | Manage team governance | 20 route-string hits | Mixed | Tier gating, permissions, audit trail, and empty Solo state are understandable. |
|
||||
| Settings | `/settings` | `SettingsApp` | Configure models, billing, general, backup | 66 route-string hits; Phase 1 390 px tests pass for general, models, billing, and profile with visible-control bounds checks; rendered `J3h` Settings backup/restore trust path passes on port `34171`; latest expanded user journeys pass 24/24 on port `34173` | Mixed | Settings telemetry clear and backup/restore now use in-app approval/status states; form labels, deeper mobile overlay paths, and broader persona-state evidence remain. |
|
||||
| Vault | `/settings/vault` | `VaultApp` | Store API keys and secrets | 7 route-string hits | Mixed | Success/error states do not leak secrets and are keyboard accessible. |
|
||||
| Profile | `/settings/profile` | `UserProfileApp` | Manage identity and preferences | Codified `J-route-coverage: priority thin routes` renders profile form shell; passed 2026-07-08 on port `34200` | Mixed | Form labels, save/error states, and mobile layout covered. |
|
||||
| Mission Control | `/settings/mission-control` | `CockpitApp` | Inspect health/cost/activity | 10 route-string hits; T5 visual failure classified as low-risk drift with connector-list scroll/affordance review | Mixed | Visual baseline, connector-list fit, and local model cost semantics fixed or explicitly deferred. |
|
||||
| Timeline | `/settings/timeline` | `TimelineApp` | Review activity history | Codified `J-route-coverage: priority thin routes` renders timeline/activity shell; passed 2026-07-08 on port `34200` | Mixed | Empty, filtered, long-list, and date formatting states covered. |
|
||||
| Events | `/settings/events` | `EventsApp` | Inspect logs/events | 6 route-string hits; T5 visual failure classified as low-risk drift | Mixed | Filters, empty state, long log lines, and approved visual baseline covered. |
|
||||
| Usage | `/settings/usage` | `TelemetryApp` | Understand spend/tokens | Codified `J-route-coverage: priority thin routes` renders usage/cost shell; passed 2026-07-08 on port `34200` | Mixed | Unknown local model pricing is explicit; cost tables use tabular numbers and Intl formatting. |
|
||||
| Benchmarks | `/benchmarks` | `BenchmarkApp` | Inspect benchmark capability | Component tests plus codified `J-route-coverage: thin utility routes` render the route; passed 2026-07-08 on port `34200` | Mixed | Discovery path and benchmark interpretation are clear. |
|
||||
| Platform | `/platform` | `PlatformApp` | Understand platform/roadmap | Component tests plus codified `J-route-coverage: thin utility routes` render the route; passed 2026-07-08 on port `34200` | Mixed | Decide if this command-palette surface is included in judged app scope. |
|
||||
| Payment success | `/payment-success` | `PaymentSuccessApp` | Recover from successful checkout | Codified `J-route-coverage: priority thin routes` renders checkout/no-confirmation fallback; passed 2026-07-08 on port `34200` | Mixed | Legacy Pro copy is constrained to historical billing state; Teams success path is clear. |
|
||||
| Payment cancelled | `/payment-cancelled` | `Navigate` to billing settings | Recover from cancelled checkout | Codified `J-route-coverage: thin utility routes` proves redirect to `/settings?tab=billing` and billing recovery copy; passed 2026-07-08 on port `34200` | Mixed | Billing explains the recovery action and checkout retry path. |
|
||||
| Not found | `*` | `NotFound` | Recover from bad route | Existing stress coverage | Mixed | Recovery link is visible and returns to Home without overlay traps. |
|
||||
|
||||
## Embedded and Retired Surface Inventory
|
||||
|
||||
This table prevents the final "complete UX" claim from silently ignoring app files that are not top-level routes.
|
||||
|
||||
| Surface/file | Current state from source inventory | UX audit consequence |
|
||||
|---|---|---|
|
||||
| `DashboardApp.tsx` | Present in `apps/web/src/components/os/apps`, but not imported by a route; `dashboard` app id retargets to `/home`. | Treat Home as the judged dashboard/continuity surface. Do not add DashboardApp evidence unless the surface is resurrected. |
|
||||
| `VoiceApp.tsx` | Present as a "Coming Soon" component; `voice` app id retargets to `/home`. | If voice becomes reachable, it needs a route, copy, and judge scenario. Current final packet may exclude it as retired/unrouted. |
|
||||
| `MissionControlApp.tsx` | Present but route comments mark the legacy app id as killed; `/settings/mission-control` renders `CockpitApp`. | Judge Mission Control through `CockpitApp`; stale comments/import references should not confuse future scoring. |
|
||||
| `BackupApp.tsx` | No top-level route; backup lives inside Settings and now has focused Settings plus standalone component coverage. | Judge backup primarily through `/settings?tab=backup`; standalone `BackupApp.tsx` restore also uses in-app approval if this embedded component is surfaced again. |
|
||||
| `StorageApp.tsx`, `FilesAppTabs.tsx`, `FilesApp.tsx` | Embedded under `StorageAndFilesApp`, which is routed at `/files`. | Judge the unified `/files` A/B storage/files surface, including workspace query state. |
|
||||
| `BenchmarkApp.tsx`, `PlatformApp.tsx` | Top-level routes exist; component tests and codified `J-route-coverage` prove they render. | Route existence is closed under T11; discovery/value clarity remains a scenario-depth item. |
|
||||
|
||||
## Command-Query Destination Inventory
|
||||
|
||||
Source inventory found two active command destinations whose base routes are covered, but whose query-state UX still needs explicit evidence:
|
||||
|
||||
| Destination | Source | UX audit consequence |
|
||||
|---|---|---|
|
||||
| `/launcher?watch=1` | Command Center action: Watch a coding agent live | Cover as a T11/T16 Launcher watch-mode state, not as a new top-level route. |
|
||||
| `/settings?tab=billing` | Command Center action: Upgrade to Team | Cover as a T3/T11/T13 billing deep-link and checkout-recovery state, not as a new top-level route. |
|
||||
| `/motion-spec` | Development-only motion specification route | Exclude from production score unless developer visual tooling is explicitly brought into scope. |
|
||||
|
||||
## Overlay Manifest
|
||||
|
||||
| Overlay | Entry | User job | Current concern | Must prove before 9/10 |
|
||||
|---|---|---|---|---|
|
||||
| Command Center | `Ctrl+K`, sidebar Search | Find routes/actions quickly | Pro wording in pinned group; action labels need consistency; fresh mobile smoke found long-label overflow, missing dialog description warning, and Escape close not proven | Search, select, gated action prompt, long-label fit, accessible dialog naming, and keyboard/touch close all pass. |
|
||||
| Workspace Switcher | Sidebar workspace, fallback shortcut | Switch/create workspace | Phase 1 route contract passes: switcher opens/closes and no longer blocks route-changing traversal; focused create-workspace mobile hierarchy now passes at 390 x 844 | Selection, Escape, outside-click, route-changing nav, and broader create-workspace states all need final judge evidence. |
|
||||
| Persona Switcher | `Ctrl+Shift+P` | Change active persona | Focus/labels need review | Current persona, available modes, custom persona creation, and close behavior covered. |
|
||||
| Spawn Agent | Sidebar/New Agent | Start an agent run | Model fallback clarity and form labels | Create, cancel, missing model, and workspace selection covered. |
|
||||
| Onboarding Wizard | First run or forced wizard | First launch setup | Phase 1 clean first-run console smoke passes and mobile Profile primary action remains reachable at 390 px | Model gate ready copy, memory import consequence clarity, template, first task, and post-completion chat handoff still need final judge screenshots. |
|
||||
| Login Briefing | Post-auth/local briefing | Explain next steps | Pro/Teams copy mismatch | Copy aligns with Solo/Teams/Enterprise and Home Start Here. |
|
||||
| Upgrade/Trial modals | Tier gates | Explain access boundaries | Strategy copy must match tier model | No Pro upgrade copy in active flows. |
|
||||
| Notification Inbox | Status/sidebar entry | Review alerts | Phase 2 partial fix: named dialog, Escape close, focus trap, named mark-all/close actions, rendered `J3b` journey passes | Notification content states and mobile screenshot refresh still need final judge evidence. |
|
||||
| Create Workspace | Workspace Switcher New workspace | Start a new workspace | Phase 2 partial fix: named primary/subdialog contracts, Escape close, focus trap, named sampled template/share actions, in-app custom-template delete confirmation, and focused 390 x 844 hierarchy with templates behind progressive disclosure | Full mobile screenshot proof, keyboard-only create path, and broader create-workspace state coverage remain. |
|
||||
| Context Rail | Chat/context action | Inspect selected context | Phase 2 partial fix: source/component evidence now gives the side rail a labelled complementary landmark, named close action, and expandable item state | Route-specific rendered open/close, loading, empty, long content, and focus order evidence remain. |
|
||||
| Onboarding Tooltips | Post-onboarding coach mark | Dismiss or advance first-run tips | Phase 2 partial fix: named non-modal dialog with Escape dismissal and stored dismissed state has component coverage | Rendered first-run/mobile state evidence and proof of suppression around other overlays remain. |
|
||||
| Upgrade / Trial tier modals | Tier-gated actions and expired trial state | Recover from paywall interruption | Phase 2 partial fix: tier modal close labels have component coverage; event-driven Upgrade modal has rendered `J3c` evidence | Full billing, checkout, expired-trial account state, and mobile evidence remain. |
|
||||
| Erase Data dialog | Settings/system | Delete local data safely | High-risk destructive flow | Exact consequence copy, typed confirmation if needed, result state covered. |
|
||||
|
||||
## Usage Scenario Catalog
|
||||
|
||||
### First-run and account scenarios
|
||||
|
||||
- First launch with no account, no local model, and no imported memory.
|
||||
- First launch with a valid Clerk key intentionally enabled.
|
||||
- Continue accountless from `/auth`.
|
||||
- Start trial and then fall back to Solo behavior.
|
||||
- Return after onboarding with remembered Home Start Here state.
|
||||
- Accountless mode with network disabled.
|
||||
|
||||
### Workspace scenarios
|
||||
|
||||
- Create first workspace from Home.
|
||||
- Create workspace from All Workspaces.
|
||||
- Switch workspace from sidebar switcher.
|
||||
- Open workspace chat from Home Start Here.
|
||||
- Open workspace chat through `Ctrl+Shift+N`.
|
||||
- Delete/archive workspace template or workspace-adjacent artifact with consequence copy.
|
||||
- Long workspace names and empty workspace lists.
|
||||
|
||||
### Chat and agent scenarios
|
||||
|
||||
- Send first chat message with real local model.
|
||||
- Send first chat message with model unavailable.
|
||||
- Retry a failed assistant response.
|
||||
- Tool-use block renders, expands, and does not leak broken JSON.
|
||||
- Model switch block explains fallback.
|
||||
- Spawn Agent with a workspace and model available.
|
||||
- Spawn Agent when no model is configured.
|
||||
- Agent group creation, detail view, execution view, and cancellation.
|
||||
|
||||
### Memory scenarios
|
||||
|
||||
- Empty Memory Center.
|
||||
- Memory list with many records and long titles.
|
||||
- Search memory.
|
||||
- Inspect memory provenance/trust.
|
||||
- Archive/delete memory with in-app confirmation.
|
||||
- Wiki export to local path without native prompt.
|
||||
- Import reminder banner dismissed.
|
||||
- Timeline and evolution tabs empty, loading, error, and populated.
|
||||
|
||||
### Extend and integration scenarios
|
||||
|
||||
- Marketplace browse/search with local catalog only.
|
||||
- Marketplace sync live-integration lane, separate from standard UX audit.
|
||||
- Skill install/update/error.
|
||||
- MCP server install/verify/uninstall.
|
||||
- Custom MCP form with invalid command, missing fields, and scope selection.
|
||||
- Connector connect/revoke/error without secret exposure.
|
||||
- Launcher tool detect, launch with prompt, hook install/verify/uninstall.
|
||||
- Browser Companion extension connected/disconnected popup, save selection, save page, CORS-denied recovery, and memory-frame result.
|
||||
|
||||
### Trust, admin, and billing scenarios
|
||||
|
||||
- Billing settings in Solo, Trial, Teams, Enterprise, and legacy Pro subscriber state.
|
||||
- Payment success for Teams.
|
||||
- Payment cancelled redirect to billing.
|
||||
- Approvals review, approve, deny, revoke all.
|
||||
- Vault create/update/delete API key without exposing value.
|
||||
- Backup create, restore, server unreachable, restore success.
|
||||
- Team governance visible/hidden according to tier.
|
||||
- Compliance template create/edit/delete.
|
||||
|
||||
### Responsive and accessibility scenarios
|
||||
|
||||
- Desktop 1440 x 900 for every route.
|
||||
- Mobile 390 x 844 for Home, Settings general/models/billing/profile, Memory, Chat, Marketplace, Billing, Profile, Command Center, Workspace Switcher, and the sampled Create Workspace path.
|
||||
- Critical visible control bounds at 390 x 844, not only document-level overflow.
|
||||
- Narrow viewport for every modal/overlay that can open from mobile.
|
||||
- Keyboard-only pass through sidebar, Command Center, Workspace Switcher, Settings tabs, Chat composer, and destructive dialogs.
|
||||
- Screen reader naming pass for icon-only buttons, form controls, tabs, dialogs, and toasts.
|
||||
- Reduced-motion pass for boot, route transitions, Home cards, and onboarding.
|
||||
|
||||
### Error, offline, and performance scenarios
|
||||
|
||||
- Sidecar unavailable.
|
||||
- Marketplace external source unavailable.
|
||||
- LLM provider unavailable.
|
||||
- File upload failure.
|
||||
- API returns 401/403/404/500.
|
||||
- Large memory list and large event log.
|
||||
- Main chunk and route-level lazy loading after performance fixes.
|
||||
- Console error budget: zero critical app/auth/CSP errors in the standard audit lane.
|
||||
|
||||
### Public launch funnel scenarios
|
||||
|
||||
- Current evidence update: focused T13 supplement confirms local production route/API smoke must use `--hostname localhost` on Windows for this Next middleware setup. Corrected-host smoke renders the homepage, legal pages, auth pages, `/docs/methodology`, unauthenticated account redirect, and signed-out GET checkout redirect; it still exposes `/pricing?checkout=cancelled` 404, signed-out POST/pricing checkout dead-end copy, legal placeholders, download target with no GitHub releases, and deploy mismatch. Treat these as T13/P0-L1 until fixed or explicitly deferred.
|
||||
- Homepage desktop and mobile first viewport.
|
||||
- Mobile menu open, navigate, and close.
|
||||
- Download CTA resolves to the intended release/download target.
|
||||
- Signed-out Team checkout guides to sign-in/sign-up and then checkout recovery.
|
||||
- Checkout cancel returns to a real pricing/billing recovery state.
|
||||
- Account route redirects unauthenticated users to sign-in.
|
||||
- `/docs/methodology` is the canonical methodology route; public links should not point users at missing `/methodology`.
|
||||
- Legal/privacy/cookies/EU AI Act pages contain no launch placeholders and no active Pro copy.
|
||||
- Deployment target serves the actual Next app shape or is explicitly replaced/deferred.
|
||||
|
||||
### Desktop wrapper and release scenarios
|
||||
|
||||
- Packaged Tauri app launches to Home or a clear service-recovery state.
|
||||
- Sidecar healthy, degraded, unavailable, and port-conflict states surface in user language.
|
||||
- Tray Open, Settings, and Quit actions are verified in a packaged smoke; About and Pause Agents remain hidden until implemented.
|
||||
- Close-to-tray and global shortcut behavior are verified on the target OS lane.
|
||||
- Update event is either wired to visible update UX or disabled/hidden until supported.
|
||||
- Installer/signing warning expectations match the release channel.
|
||||
|
||||
Current T14 source evidence update:
|
||||
|
||||
- Tray Open/focus, close-to-tray, `Ctrl+Shift+W`, and Quit are source-wired in Rust, while Settings routes through a tested `/settings` desktop bridge; none of these is yet proved through a packaged app smoke.
|
||||
- Remaining unconsumed native event families are update/service status: `waggle://update-available`, `waggle://service-status`, and `waggle://service-restart-needed`.
|
||||
- About and Pause Agents tray actions are hidden until there is real product behavior behind them.
|
||||
|
||||
### Admin web and CLI/MCP utility scenarios
|
||||
|
||||
- Admin web connects with team slug + token, then renders Dashboard, Analytics, Members, Capabilities, Jobs, Audit Log, and Team Settings.
|
||||
- Admin web empty, loading, API failure, invalid token, long team/user names, mobile width, keyboard navigation, and focus states are verified or explicitly deferred.
|
||||
- `npx waggle --help`, default startup, `--port`, invalid port, port conflict, `--skip-litellm`, `--no-open`, first-run setup, and service failure copy are verified or explicitly deferred.
|
||||
- Waggle CLI command parsing, auth failure, normal chat/repl entry, `/help`, `/model`, `/clear`, `/identity`, and graceful exit are verified or explicitly deferred.
|
||||
- Marketplace CLI help, search/list/info/install/audit/scan, invalid package, dangerous `--force-insecure`, and JSON/human output states are verified or explicitly deferred.
|
||||
- Memory MCP and hive-mind MCP stdio startup, scope enforcement, erase, missing data dir, invalid JSON args, and host integration errors are verified or explicitly deferred.
|
||||
- Hive-mind CLI help, init/status/recall/save/harvest/maintenance, missing env, invalid path, `--json`, and Windows shim/postinstall path are verified or explicitly deferred.
|
||||
- Root verification either discovers hive-mind CLI colocated tests or cites the separate command that owns them.
|
||||
|
||||
### AI-tool hook lifecycle scenarios
|
||||
|
||||
- Launcher shows all 7 launchable AI tools and exposes hook install/verify/uninstall only for the 6 hook-capable tools.
|
||||
- Claude Desktop is launchable but clearly marked as not hook-capable.
|
||||
- Codex Desktop shares Codex hook state and copy makes that understandable.
|
||||
- Hook install, verify, uninstall, already-installed, not-detected, missing CLI, corrupt config, backup restore, and sidecar-unreachable fail-open states are verified or explicitly deferred.
|
||||
- Hook logs are quiet enough for standard audit output, or expected fail-open warnings are documented and filtered from failure triage.
|
||||
- Real-tool or hermetic config evidence proves that hook install does not corrupt existing settings and uninstall is byte-identical or removes only Waggle-managed files.
|
||||
|
||||
### Developer API, background worker, and substrate scenarios
|
||||
|
||||
- SDK skill/plugin validation, install, runtime, invalid package, and bad metadata states are verified or explicitly deferred.
|
||||
- Server route suite passes deterministically as a standard audit command, or performance assertions are moved to an isolated documented lane.
|
||||
- Worker job processing, failed handler, retry/recovery, and dispatch states have command or UI evidence.
|
||||
- WaggleDance protocol dispatch, invalid signal, and unavailable peer/server states have evidence.
|
||||
- Hive-mind core, shim core, wiki compiler, optimizer, shared, and compiler package tests are discoverable through documented root or package-local commands.
|
||||
- Package-local `npm test` scripts either pass or clearly delegate to root/project-reference commands.
|
||||
- Standard package verification output is quiet enough that real failures stand out, or expected warnings are documented.
|
||||
|
||||
### Ops, deployment, CI, benchmark, and judging scenarios
|
||||
|
||||
- GitHub Actions CI, release, Tauri PR, public-site deploy, and hive-mind CLI workflows parse and have explicit blocking/advisory semantics.
|
||||
- Dockerfile, Compose, Render, and LiteLLM config parse and have secret-safe validation evidence.
|
||||
- Compose validation uses non-interpolated or sanitized output when evidence may be copied into docs, issues, or PRs.
|
||||
- Render deployment target is explicitly local-sidecar demo or team Postgres server; provisioned services match that decision.
|
||||
- Infra-dependent tests have a Docker/migration lane, and missing CI coverage is explicitly accepted or fixed.
|
||||
- Benchmark harness typecheck and tests are runnable through documented commands.
|
||||
- Historical `judging/` rounds are treated as reference material only; final scorecards are generated from current post-fix source.
|
||||
|
||||
## Five-Persona Coverage Map
|
||||
|
||||
| Persona | Journey | Required route/overlay coverage |
|
||||
|---|---|---|
|
||||
| Solo founder | First run -> accountless -> Home Start Here -> workspace -> chat -> return later | `/auth`, onboarding, `/home`, `/workspaces`, workspace chat, Settings model gate. |
|
||||
| Researcher | Memory search -> provenance -> wiki/timeline -> export/delete trust flow -> optional Browser Companion capture | `/memory`, memory tabs, `/settings/timeline`, native prompt replacements, T19 extension evidence if capture is in scope. |
|
||||
| Engineer | Command Center -> shortcuts -> Launcher/MCP -> files -> logs -> CLI utility, hook smoke, and developer verification | Command Center, `/launcher`, `/mcps`, `/files`, `/settings/events`, `Ctrl+Shift+N`, selected T15 CLI/MCP, T16 hook, T17 developer/substrate evidence, and T19 extension sidecar contract evidence if not deferred. |
|
||||
| Team admin | Billing -> Vault -> Approvals -> Backup -> Team governance -> admin web | `/settings`, `/settings/vault`, `/approvals`, backup section, `/team`, payment success/cancelled, selected T15 admin-web evidence if not deferred. |
|
||||
| Mobile executive | Narrow Home -> Settings -> billing/profile -> memory glance -> theme | Mobile `/home`, `/settings`, `/settings/profile`, `/memory`, theme controls. |
|
||||
|
||||
## Final Judge Evidence Packet
|
||||
|
||||
Before claiming the goal is achieved, collect or generate:
|
||||
|
||||
- Current route manifest with every row marked Strong or explicitly deferred.
|
||||
- Standard verification command output.
|
||||
- Rendered desktop screenshots for all primary routes.
|
||||
- Mobile screenshots for core mobile routes and every overlay used on mobile.
|
||||
- Overlay evidence includes Notification Inbox, Create Workspace, Context Rail, Onboarding Tooltips, and tier modals, not only Command Center/Workspace Switcher. Current update: the core contract and sampled Create Workspace mobile hierarchy have focused coverage, but less common states still need rendered/mobile owners.
|
||||
- Console error summary from the standard browser lane.
|
||||
- Visual baseline decision log: updated baseline versus fixed UI.
|
||||
- Five persona scorecards with scores and notes for all six dimensions in the main audit.
|
||||
- State/failure bundle coverage for each of the five judge personas.
|
||||
- T13 public launch funnel evidence or approved deferral.
|
||||
- T14 desktop wrapper/release evidence or approved deferral.
|
||||
- T15 admin web and CLI/MCP utility evidence or approved deferral.
|
||||
- T16 AI-tool hook lifecycle evidence or approved deferral.
|
||||
- T17 developer API/background/substrate verification evidence or approved deferral.
|
||||
- T18 ops/deployment/CI/benchmark/judging evidence or approved deferral.
|
||||
- Completed `docs/audits/2026-07-08-five-persona-judge-scorecards.md` table with evidence and score caps applied.
|
||||
- Completed `docs/audits/2026-07-08-five-persona-judge-runbook.md` evidence folders or equivalent current evidence links.
|
||||
- Deferral list, if any, approved before scoring.
|
||||
|
||||
Current T11 evidence note: app-local route-table tests pass 29/29 and Benchmark/Platform/Payment component tests pass 17/17. Fresh built-app route smokes on ports 3407 and 3411 proved the previous zero/thin route set renders/redirects as URLs. A current all-route built-preview smoke on port 3457 proves 33 desktop route navigations plus 11 mobile route spot-checks return 200, produce meaningful screenshots, and have no document-level horizontal overflow; `/payment-cancelled` redirects to `/settings?tab=billing`. Codified `J-route-coverage` tests now pass 2/2 on port `34200`, so route-existence ownership is closed. Remaining issues such as Launcher detect noise, Usage 403 semantics, expected catch-all logging, and accessible-name gaps stay in T9/T10/T12/T16 rather than T11 route ownership.
|
||||
98
docs/audits/2026-07-08-ux-state-failure-scenario-matrix.md
Normal file
98
docs/audits/2026-07-08-ux-state-failure-scenario-matrix.md
Normal file
@@ -0,0 +1,98 @@
|
||||
# Waggle OS UX State and Failure Scenario Matrix
|
||||
|
||||
Companion artifacts:
|
||||
|
||||
- `docs/audits/2026-07-08-complete-ux-usage-audit.md`
|
||||
- `docs/audits/2026-07-08-ux-route-scenario-manifest.md`
|
||||
- `docs/audits/2026-07-08-five-persona-judge-scorecards.md`
|
||||
- `docs/audits/2026-07-08-five-persona-judge-runbook.md`
|
||||
- `docs/audits/2026-07-08-ux-correction-register.md`
|
||||
- `docs/audits/2026-07-08-source-inventory-consistency-audit.md`
|
||||
- `docs/audits/2026-07-08-state-failure-t12-analysis.md`
|
||||
- `docs/audits/2026-07-08-mobile-executive-t2-t12-analysis.md`
|
||||
- `docs/audits/2026-07-08-runtime-a11y-t10-analysis.md`
|
||||
- `docs/audits/2026-07-08-first-run-onboarding-t1-t2-t12-analysis.md`
|
||||
|
||||
Status: analysis-only. No product code is approved or changed by this file.
|
||||
|
||||
Purpose: route coverage proves that screens load; it does not prove the product is usable across the states real users hit. This matrix defines the state and failure combinations that must be sampled, tested, or explicitly deferred before the five-persona 9/10 judge gate.
|
||||
|
||||
## Source Inventory
|
||||
|
||||
| System axis | Source of truth inspected | UX consequence |
|
||||
|---|---|---|
|
||||
| Billing tiers | `packages/shared/src/tiers.ts` | Canonical tiers are `TRIAL`, `FREE` displayed as Solo, `TEAMS`, and `ENTERPRISE`; legacy `PRO` maps to `FREE`. |
|
||||
| UI disclosure tiers | `apps/web/src/lib/dock-tiers.ts` | Navigation depth changes across `simple`, `professional`, `power`, and `admin`; billing tier also hides Team/Approvals entries below Teams. |
|
||||
| Settings visibility | `apps/web/src/lib/settings-tier-filter.ts` | Settings tabs vary by disclosure tier; Team tab has a separate Teams billing gate. |
|
||||
| Auth/account mode | `apps/web/src/lib/clerk.ts`, `apps/web/src/providers/WaggleClerkProvider.tsx` | Clerk is optional; no key or malformed key must yield honest accountless mode, not a blank shell. |
|
||||
| Shell runtime state | `apps/web/src/providers/ShellContext.tsx`, `apps/web/src/components/os/AppShell.tsx` | Shell owns workspaces, tier, trial, onboarding, notifications, offline, overlays, and context rail. |
|
||||
| Workspace state | `apps/web/src/hooks/useWorkspaces.ts` | Workspace selection is explicit; failed loads preserve prior data and surface errors. Create has a local fallback, while delete/patch mutate only on server success. |
|
||||
| Offline state | `apps/web/src/hooks/useOfflineStatus.ts` | Offline flips after two failed health checks and rechecks on online, visible, and focus events. |
|
||||
| Model readiness | `apps/web/src/hooks/useHasWorkingModel.ts`, `apps/web/src/components/os/model-gate/ModelGate.tsx`, `NoModelBanner.tsx` | A working model means at least one keyed cloud provider or detected local model; live probes distinguish verified, rejected, and unverified keys. |
|
||||
| Adapter/API surface | `apps/web/src/lib/adapter.ts`, `packages/server/src/local/index.ts` | UI flows span workspaces, chat, memory, artifacts, local inference, skills, marketplace, agents, automations, notifications, approvals, settings, providers, connectors, MCP, vault, team, cost, backup, files, Stripe, harvest, wiki, identity, compliance, and local inference route families. |
|
||||
| Destructive workflows | `BackupApp.tsx`, `EraseDataDialog.tsx`, `ApprovalsApp.tsx`, `ArtifactCenterApp.tsx`, `CreateWorkspaceDialog.tsx`, `MemoryCenterTab.tsx`, `WikiTab.tsx`, `SettingsApp.tsx` | Typed confirmation exists for erase data; Create Workspace template delete, Approvals revoke-all, Artifact permanent delete, Memory Center delete/erase/re-import, Wiki export destinations, and Settings telemetry/backup/restore now use in-app confirmations/forms/status; several other trust-critical flows still use native browser dialogs or thin result states. |
|
||||
|
||||
## State Matrix
|
||||
|
||||
| State axis | Variants to sample before 9/10 | Primary surfaces | Current evidence | Gap or correction owner |
|
||||
|---|---|---|---|---|
|
||||
| Account and auth | No Clerk key, malformed Clerk key, valid Clerk key, accountless continue, session-token bootstrap failure, Clerk network/CSP failure | `/auth`, `WaggleClerkProvider`, `AuthRoute`, full shell | Auth/component tests plus rendered console failure evidence | T1: local accountless lane must emit no Clerk/CSP errors; auth-enabled lane must remain intentional. |
|
||||
| Onboarding lifecycle | Fresh install, skipped boot, incomplete wizard, completed onboarding, forced wizard, returning after absence, high-volume import detected, model ready/no-model, first task auto-send | `OnboardingWizard`, `BootScreen`, `LoginBriefing`, `/home`, workspace chat | Unit/E2E coverage exists for boot, onboarding gates, first task, briefing; focused first-run smoke completed desktop onboarding and captured mobile Welcome/Profile | T1/T2/T12: first-run clean-data lane still has Clerk/CSP console errors, mobile Profile Continue below viewport, import CTA risk, and model/auto-send polish gaps. |
|
||||
| Billing tier | Trial active, trial expired, Solo, Team, Enterprise, legacy Pro subscriber, tier lookup error, Stripe unconfigured | Settings billing, `PlanCards`, `UpgradeModal`, `TrialExpiredModal`, Team/Approvals nav, `/payment-success`, `/payment-cancelled` | Tier tests and API tests exist; active UI still has Pro-copy findings | T3 plus T11/T12: current copy and payment recovery states must be judge-ready. |
|
||||
| UI disclosure tier | Simple, professional, power, admin; same billing tier with different disclosure tier | Sidebar, Command Center, Settings tabs, pinned nav | Route/unit coverage exists for route table and settings filter | T12: judge evidence must include at least simple and power/admin shell screenshots so hidden vs discoverable depth is intentional. |
|
||||
| Workspace list and selection | Empty list, loading, load error, many workspaces, long names, archived entries, no active workspace, stale active workspace id, active workspace deleted | Home, Workspace Switcher, `/workspaces`, `/workspaces/:id`, Chat shortcut | Hook code distinguishes errors; rendered failures cover Workspace Switcher and shortcut | T4/T11/T12: route-changing nav, `Ctrl+Shift+N`, and empty/error workspace states need current evidence. |
|
||||
| Workspace content | Empty workspace, populated workspace, workspace with pending tasks, workspace files, long activity history, permission denied, not found | `WorkspaceDesktopApp`, `TasksTab`, files/storage, Home Start Here | Mixed route and unit coverage | T11/T12: each workspace tab needs either persona journey coverage or explicit deferral. |
|
||||
| Model readiness | No model, cloud key saved but unverified, cloud key verified, rejected key, local Ollama present, local runtime unavailable, LiteLLM unavailable, unpriced local model | Onboarding model gate, Settings Models, Home banner, Chat, Spawn Agent, Usage & Cost | ModelGate and no-model tests exist; local model cost warning found | T9/T12: no-model and local-model lanes must be scored honestly. |
|
||||
| Chat runtime | First message, streaming response, retry after failure, abort, session search, session export, artifact block, tool-use block, model switch block, missing provider | Chat, `ChatHost`, `ChatWindowInstance`, chat blocks, workspace sessions | Chat block and live-chat tests exist | Final judge evidence must include failure/retry and artifact-open-to-Files loop. |
|
||||
| Agent and automation runtime | No agents, many agents, create agent, ambiguous workspace, run/pause/cancel, automation builder validation, scheduled run logs, engine unavailable | `/agents`, `/automations`, Spawn Agent, Room | Unit/E2E coverage exists, but warning noise remains | T10/T12: form validation, loading/error, and run-result states need clean verification output. |
|
||||
| Memory and provenance | Empty memory, many frames, search no-results, search hit, source missing/404, trace, trust confirm, archive, delete, erase/suppression, all-minds vs workspace scope | `/memory`, Memory Center tabs, Context Rail, Erase Data | Memory route and trust tests exist; Memory Center delete/erase/re-import and Wiki export destinations now have in-app coverage | T7/T10/T12: export failure/result states and screen-reader labels need final pass. |
|
||||
| Browser capture extension | Extension not loaded, connected, disconnected/CORS denied, save selection, save page, context menu save, content script unavailable, active workspace missing, resulting memory frame, provenance mismatch | `apps/browser-ext` popup/background/content, `/api/browser-ext/session-token`, `/api/browser-ext/health`, `/api/memory/frames`, Memory provenance/search | Source inventory, direct sidecar save, secure-default loaded-extension content extraction/background save, stable packaged-ID pairing, token bootstrap during save, duplicate handling, Memory frame confirmation, popup keyboard/focus/Enter save, Save page click, restricted-page disabled-state/recovery behavior, Memory search `source: import` provenance, existing chat `auto_recall`/catch-up imported provenance, and rendered Memory UI after secure popup save now exist; native toolbar bubble, native context menu, signed release-package proof if scored, and any future scored recall result shape remain unproven. | T19: verify native toolbar/context-menu UX and CORS recovery, plus future recall shapes if scored, or explicitly defer them from final scoring. |
|
||||
| Files and artifacts | No workspace, workspace chosen by query, root directory, nested folder, upload, preview, download, move/copy/delete, traversal input, artifact delete/archive | `/files`, `/artifacts`, chat artifact block | Deep-link and path-normalization tests exist | T11/T12: rendered Files/Artifacts destructive and empty/error states need evidence. |
|
||||
| Skills, marketplace, connectors, MCP | Local-only catalog, live marketplace sync, search empty, install success, install 403, install failure, connector connect/revoke/sync error, custom MCP invalid command, MCP permission scope | `/skills`, `/marketplace`, `/connectors`, `/mcps` | Many component/API tests; marketplace sync flake found | T6/T7/T10/T12: determinism, forms, and branded confirmation states. |
|
||||
| Launcher and external tools | No tools detected, supported tools detected, launch with prompt, launch failure, process running, hook install/verify/uninstall, hook unsupported | `/launcher`, tool output pane | Component tests exist; route coverage is thin | T11/T12: clean-install runtime verification and route-level evidence. |
|
||||
| Team and governance | Solo hidden state, Team visible state, Enterprise/KVARK CTA, Approvals present, Team governance empty/populated, audit trail | `/team`, `/approvals`, Settings Team/Enterprise, Cockpit compliance | Tier/API coverage exists; current high-confidence native-dialog scan is clean | T7/T12: Team admin judge must see clear gating, branded trust flows, and failure-state evidence. |
|
||||
| Backup, restore, and erasure | No backup, metadata 404 empty, metadata 500 error, create success, create failure, restore file selected, restore cancel, restore success, erase phrase mismatch, erase success receipt | Settings Backup, `BackupApp`, `EraseDataDialog` | Settings backup failure/restore success and standalone `BackupApp` restore now have in-app approval/status evidence; Backup status classifier and erase modal exist | T7/T12: final judge evidence must cover typed/explicit consequence copy, restore result states, and broader backup failure variants. |
|
||||
| Offline and API failure | Sidecar unavailable, `/health` failing, session token 401, route 403 tier gate, detail 404, route 500, SSE disconnect, external marketplace unavailable, Stripe unavailable | StatusBar offline pill, Home, Settings, Command Center, Marketplace, Notifications, Events | Failure-injection tests and hook logic exist | T1/T6/T12: standard audit lane needs zero critical console errors and clear offline recovery copy. |
|
||||
| Responsive layout | Desktop 1440x900, tablet 1024x768, mobile 390x844, landscape mobile, modal on narrow screen, long copy, long names | All routes and overlays; especially Settings, onboarding, Memory, Marketplace, Chat, Create Workspace | Fresh 390 x 844 smoke renders Home, Settings, Profile, Memory, workspace chat, Command Center, and Workspace Switcher; Settings still fails through visible clipping/squeezing despite no document overflow; first-run mobile Profile hides Continue below the viewport; Memory/chat tab strips overflow; Create Workspace fits horizontally but the first mobile viewport is template-heavy; visual suite failed | T2/T5/T10/T12: mobile screenshots, critical element-bounds checks, overlay close checks, creation-flow hierarchy, and visual baseline decision log are mandatory. |
|
||||
| Accessibility and keyboard | Sidebar tab order, Command Center keyboard, Workspace Switcher, Notification Inbox, Create Workspace, Context Rail, Settings tabs, forms, icon-only buttons, dialogs, toasts, reduced motion | Shell, overlays, forms, ModelGate, destructive dialogs, Mission Control, agent cards, Launcher, Approvals, Files, workspace chat | Mixed unit coverage and guidelines pass; static inspection found representative unlabeled controls; fresh runtime axe/DOM smoke found critical unnamed controls/selects, a serious Files keyboard-scroll issue, workspace semantics/image-alt findings, and Command Center dialog/label-fit warnings; shell overlay smoke found Notification Inbox and Create Workspace lacking semantics/Escape close and Create Workspace exposing 16 unnamed visible icon buttons | T10/T12: keyboard-only, screen-reader naming, modal focus-return, runtime axe, overlay semantics/close, named icon actions, and reduced-motion passes must be evidence-backed. |
|
||||
| Scale and performance | Large memory list, large event log, 50+ marketplace items, many agents, large files, first bundle, lazy deep apps | Memory, Events, Marketplace, Agents, Files, app shell | Build warning shows large main chunk | T8/T12: performance budget and list behavior need current evidence. |
|
||||
|
||||
## Judge State Bundles
|
||||
|
||||
These bundles turn the matrix into five concrete scoring runs. A persona should not receive 9/10 if their bundle skips the state that matters to their job.
|
||||
|
||||
| Judge persona | Required state bundle | Minimum evidence before scoring |
|
||||
|---|---|---|
|
||||
| Solo founder | Accountless or Solo, simple disclosure, no or one workspace, clean-data first-run onboarding, model gate/no-model recovery, Home Start Here, workspace chat, marketplace local-only | Desktop/mobile onboarding, desktop and mobile Home, Settings Models, first chat, no critical console errors. |
|
||||
| Researcher | Populated memory plus empty search, provenance/trust, wiki/timeline, source missing or archive/delete, export/error result state, Browser Companion capture if in scope | Memory screenshots, keyboard path through tabs, trust/destructive confirmation evidence, T19 extension evidence or deferral. |
|
||||
| Engineer | Power/admin disclosure, Command Center, `Ctrl+Shift+N`, MCP custom invalid/valid, Launcher hook lifecycle, files artifact deep-link, events logs, extension-sidecar contract if in scope | Shortcut proof, route-level Launcher/MCP/Files evidence, console health, T19 contract evidence or deferral. |
|
||||
| Team admin | Team billing tier, Team/Approvals visible, Vault, Backup, Team governance, payment success/cancelled, legacy Pro collapsed to Solo where relevant | Billing/tier screenshots, branded confirmations, backup/restore and approvals evidence. |
|
||||
| Mobile executive | Mobile 390 x 844, simple disclosure, Settings/Profile/Billing, Memory glance, theme toggle, notification/overlay close | Mobile screenshots for routes and overlays, no horizontal overflow, critical visible controls in-bounds, visible focus, readable copy, and selected overlay close proof. |
|
||||
|
||||
## Evidence Rules
|
||||
|
||||
- A component unit test is not enough for a routed state unless the route shell, side effects, and viewport are irrelevant to the claim.
|
||||
- A route smoke is not enough for a stateful workflow unless it exercises the state transition and observes the result.
|
||||
- A failing external service can pass UX only if the user sees a clear, branded recovery state and the standard audit lane avoids avoidable live external dependency.
|
||||
- Native browser dialogs do not satisfy trust-critical UX unless explicitly approved as a temporary exception.
|
||||
- Any final judge score must cite the state bundle it actually exercised, not merely the route it visited.
|
||||
- The exact state bundle must be recorded in the judge runbook evidence folder before the persona score is accepted.
|
||||
|
||||
## Corrections Implied By This Matrix
|
||||
|
||||
This matrix does not replace the existing tickets. It clarifies their evidence scope:
|
||||
|
||||
- T1 must prove accountless/auth-enabled console health, not only update CSP strings.
|
||||
- T2/T5 must collect mobile/visual evidence for stateful surfaces, not only default route screenshots or document scroll-width checks.
|
||||
- T6 must separate local marketplace UX from live sync UX.
|
||||
- T7 must cover branded confirmations and result states for destructive flows.
|
||||
- T8 must define scale/performance evidence for large lists and app payload.
|
||||
- T9 must label local/unpriced model cost states honestly.
|
||||
- T10 must include keyboard, labels, icon-only action names, modal focus return, inline errors, runtime axe/DOM findings, and warning hygiene.
|
||||
- T11 must own route-level evidence for every registered route and embedded/retired surface classification.
|
||||
- T12 should be added as the cross-state judge evidence ticket: every five-persona judge run must declare which account, tier, disclosure, model, data-volume, offline, and viewport states it exercised.
|
||||
- Focused T12 supplement `docs/audits/2026-07-08-state-failure-t12-analysis.md` is the current evidence record for the state-slice command run, native dialog scan, persona bundle fields, and T12-A through T12-G correction candidates.
|
||||
- Focused mobile supplement `docs/audits/2026-07-08-mobile-executive-t2-t12-analysis.md` is the current evidence record for Mobile Executive 390 x 844 screenshots, Settings visible clipping, Memory/chat tab overflow, and Command Center mobile overlay risks.
|
||||
- Focused shell overlay supplement `docs/audits/2026-07-08-shell-overlays-t10-t12-analysis.md` is the current evidence record for Notification Inbox/Create Workspace semantics and close failures, Create Workspace mobile hierarchy, Context Rail/coach-mark semantics, and tier-modal close naming.
|
||||
- Focused first-run supplement `docs/audits/2026-07-08-first-run-onboarding-t1-t2-t12-analysis.md` is the current evidence record for clean-data onboarding, desktop completion, mobile Profile reachability, import risk, and first-task handoff behavior.
|
||||
- T19 remaining proof is native toolbar/context-menu capture, clear CORS/setup recovery screenshots if scored, signed release-package proof if scored, and any future scored recall result shape; loaded-extension UX, popup keyboard/focus basics, stable packaged-ID pairing, restricted-page disabled-state recovery, rendered Memory confirmation, consistent `/api/memory/search` provenance, and existing chat `auto_recall`/catch-up imported provenance now have evidence.
|
||||
96
docs/audits/2026-07-08-visual-t5-classification.md
Normal file
96
docs/audits/2026-07-08-visual-t5-classification.md
Normal file
@@ -0,0 +1,96 @@
|
||||
# T5 Visual Snapshot Classification
|
||||
|
||||
Status: verified fixed for the tracked visual lane. The original classification below is retained for history; the current worktree includes canonical baseline updates plus a stabilization-only visual-spec change for volatile Home text. No product UI rollback was required.
|
||||
|
||||
## Command
|
||||
|
||||
Fresh-port run from `D:\Projects\waggle-os`:
|
||||
|
||||
```powershell
|
||||
$env:WAGGLE_E2E_PORT='3463'
|
||||
$env:WAGGLE_E2E_BASE_URL='http://127.0.0.1:3463'
|
||||
$env:WAGGLE_E2E_DATA_DIR=(Join-Path $pwd 'output/playwright/visual-t5-3463/data')
|
||||
$env:WAGGLE_TRUST_LOCALHOST='1'
|
||||
$env:WAGGLE_DISABLE_MARKETPLACE_SYNC='1'
|
||||
$env:EMBEDDING_PROVIDER='mock'
|
||||
$env:VITE_CLERK_PUBLISHABLE_KEY=''
|
||||
$env:CLERK_SECRET_KEY=''
|
||||
node node_modules/playwright/cli.js test tests/visual/views.spec.ts --project=chromium --reporter=list --output=output/playwright/visual-t5-3463/test-results
|
||||
```
|
||||
|
||||
Historical result: failed as expected; all 14 snapshot comparisons were red before the approved baseline/spec update.
|
||||
|
||||
## Current Verification
|
||||
|
||||
Fresh-port verification from `D:\Projects\waggle-os`:
|
||||
|
||||
```powershell
|
||||
$env:WAGGLE_E2E_PORT='34199'
|
||||
$env:WAGGLE_E2E_BASE_URL='http://localhost:34199'
|
||||
node node_modules/playwright/cli.js test tests/visual/views.spec.ts --project=chromium --reporter=line
|
||||
```
|
||||
|
||||
Result: passed, 14/14 visual snapshot comparisons.
|
||||
|
||||
Current implementation notes:
|
||||
|
||||
- The runner still covers the same seven desktop views: chat, memory, events, capabilities, cockpit/Home, mission-control, and settings.
|
||||
- `tests/visual/views.spec.ts` now masks volatile Home cockpit facts/workspace text so snapshots compare stable layout and presentation instead of live workspace copy.
|
||||
- The active expected paths remain the ASCII-hyphen `tests/visual/baselines/Visual-Regression---...` family.
|
||||
- Duplicate historical baseline families remain present but documented below; they are not the family used by the current runner.
|
||||
|
||||
Artifacts:
|
||||
|
||||
- `output/playwright/visual-t5-3463/test-results/`
|
||||
- Actual and diff PNGs are stored below per-test folders.
|
||||
- The generated sidecar data directory was disposable and should not be kept as product evidence.
|
||||
|
||||
## Summary
|
||||
|
||||
The visual suite is not failing because snapshots are missing. The active expected paths resolve under the ASCII-hyphen `tests/visual/baselines/Visual-Regression---...` family.
|
||||
|
||||
The current screenshots are generally coherent in desktop dark and light mode. The diffs mostly reflect stale baselines after intentional surface changes: Home continuity now has a Start Here card, Chat has the newer workspace setup/composer state, Memory has the newer trust/correct/forget presentation, Settings has the newer provider-card and Model Pilot layout, and Skills Hub has the newer list/action treatment.
|
||||
|
||||
Do not update baselines before Phase 1 approval and the other Phase 1 UI changes land. Rebaseline only after screenshots are reviewed again.
|
||||
|
||||
## Failure Table
|
||||
|
||||
| Snapshot | Pixels | Ratio | Classification | Decision |
|
||||
|---|---:|---:|---|---|
|
||||
| `chat-dark.png` | 21,258 | 0.03 | Intentional drift | Keep current workspace chat direction; rebaseline after Phase 1 approval unless T4 changes the route/composer state. |
|
||||
| `chat-light.png` | 18,337 | 0.02 | Intentional drift | Same as dark. |
|
||||
| `memory-dark.png` | 48,585 | 0.06 | Intentional drift, plus T10/T12 follow-up | Current Memory actual is coherent; update baseline only after Memory mobile/tab/accessibility follow-ups are either fixed or explicitly deferred. |
|
||||
| `memory-light.png` | 44,402 | 0.05 | Intentional drift, plus T10/T12 follow-up | Same as dark. |
|
||||
| `events-dark.png` | 9,088 | 0.01 | Low-risk drift | Rebaseline after approval; no immediate UI fix found in the inspected actual. |
|
||||
| `events-light.png` | 3,297 | 0.01 | Low-risk drift | Rebaseline after approval; no immediate UI fix found in the inspected actual. |
|
||||
| `capabilities-dark.png` | 12,221 | 0.02 | Intentional drift, plus T10 follow-up | Current Skills Hub actual is coherent; keep T10 coverage for icon/action accessible names and long-row truncation. |
|
||||
| `capabilities-light.png` | 9,325 | 0.02 | Intentional drift, plus T10 follow-up | Same as dark. |
|
||||
| `cockpit-dark.png` | 27,356 | 0.03 | Intentional drift | Preserve the current Home Start Here continuity surface; rebaseline after approval. |
|
||||
| `cockpit-light.png` | 22,423 | 0.03 | Intentional drift | Same as dark. |
|
||||
| `mission-control-dark.png` | 8,036 | 0.01 | Low-risk drift, plus layout review | Current cockpit actual is coherent, but the connector list reaches the viewport edge in the cropped capture; keep a scroll/affordance check before final scoring. |
|
||||
| `mission-control-light.png` | 3,304 | 0.01 | Low-risk drift, plus layout review | Same as dark. |
|
||||
| `settings-dark.png` | 42,572 | 0.05 | Intentional drift, plus Phase 1 dependency | Current Settings actual is coherent, but T2/T3 may alter Settings; rebaseline only after mobile Settings and Solo/Teams copy fixes. |
|
||||
| `settings-light.png` | 34,466 | 0.04 | Intentional drift, plus Phase 1 dependency | Same as dark. |
|
||||
|
||||
## Baseline Ownership Finding
|
||||
|
||||
`tests/visual/baselines/` currently contains three naming families:
|
||||
|
||||
- `Visual-Regression---...` - active family used by the current run.
|
||||
- `Visual-Regression-...` with Unicode dash characters - duplicate historical family.
|
||||
- `Visual-baselines-...` with title-case snapshot names - older duplicate historical family.
|
||||
|
||||
Correction after approval:
|
||||
|
||||
1. Keep a single canonical baseline family.
|
||||
2. Remove or archive duplicate historical baseline folders only as part of an explicit test-readiness change.
|
||||
3. Re-run the suite with `--update-snapshots` only after the current actual screenshots are approved.
|
||||
|
||||
## T5 Decision
|
||||
|
||||
T5 is verified fixed for the tracked visual lane:
|
||||
|
||||
- Product UI did not need to be rolled back to old baselines.
|
||||
- The current canonical baselines represent the approved Phase 1 desktop screenshots.
|
||||
- The visual suite passes 14/14 on port `34199`.
|
||||
- A later test-readiness cleanup can remove or archive duplicate historical baseline folders, but they no longer block this P0 because the active runner family is documented and green.
|
||||
234
docs/audits/2026-07-08-web-guidelines-line-findings.md
Normal file
234
docs/audits/2026-07-08-web-guidelines-line-findings.md
Normal file
@@ -0,0 +1,234 @@
|
||||
# Web Guidelines Line Findings - 2026-07-08
|
||||
|
||||
Status: analysis supplement with implementation notes.
|
||||
|
||||
Source rule set: Vercel Web Interface Guidelines, fetched on 2026-07-08 from `https://raw.githubusercontent.com/vercel-labs/web-interface-guidelines/main/command.md`.
|
||||
|
||||
Scope scanned:
|
||||
|
||||
- `apps/web/src/components/os`
|
||||
- `packages/admin-web/src`
|
||||
- `apps/www/app`
|
||||
- Test/spec files excluded for the count summaries.
|
||||
|
||||
This file supplements the main UX audit with line-level Web Interface Guidelines findings. It does not change the Phase 1 approval boundary. It strengthens T7, T8, and T10 in the correction register.
|
||||
|
||||
## Summary
|
||||
|
||||
| Area | Finding | Ticket |
|
||||
|---|---:|---|
|
||||
| Native browser dialogs | Historical audit found 20 high-confidence production calls in destructive/admin/export flows; current follow-up scan finds 0 after Create Workspace template delete, Approvals revoke-all, Artifact delete, Memory Center destructive-action, Wiki export, Settings telemetry/backup/restore, BackupApp restore, Automation delete, compliance template delete, and admin member-removal fixes | T7 |
|
||||
| Broad animation transitions | 0 `transition-all` hits in scoped production UI after the focused transition-scoping sweep | T8/T10 |
|
||||
| Focus suppression | 57 `outline-none` / `focus:outline-none` hits in the original scan; 0 current high-confidence weak/missing focus replacements remain in the reviewed list below after focused fixes | T10 |
|
||||
| Raw images | 11 `<img>` hits in cockpit UI; 0 lack explicit `width`/`height` attributes after the `SuggestedAgentCards`, `AgentCenterRow`, `ReadyStep`, `BootScreen`, `StatusBar`, `LoginBriefing`, `ChatApp`, and `SpawnAgentDialog` fixes | T8/T10 |
|
||||
| Form metadata | Loose JSX scan found 145 text-like controls missing `name`, 107 missing `autoComplete`, and 115 without an obvious same-tag `id`/ARIA label hook | T10 |
|
||||
| Locale/date handling | 114 `new Date(...)` / `toLocale*` hits in scoped production UI; user-visible date/number output needs an `Intl.*` pass | T10/T12 |
|
||||
| Paste/zoom blockers | No paste-blocking `onPaste` and no zoom-disabling viewport rules found | pass |
|
||||
|
||||
The form metadata scan is intentionally treated as an audit queue, not a compiler-grade count, because JSX attributes often span lines and custom `Input` components need contextual label inspection.
|
||||
|
||||
## T7 Native Dialogs
|
||||
|
||||
These should move to in-app confirmation/result patterns with clear object name, consequence, reversibility, and post-action state.
|
||||
|
||||
Current high-confidence production scan: no remaining browser-native `confirm`, `alert`, or `prompt` calls. Remaining broad scan hits are markdown sanitizer test payloads only.
|
||||
|
||||
Implementation update 2026-07-08:
|
||||
|
||||
- `CreateWorkspaceDialog.tsx` template delete now uses an in-app confirmation.
|
||||
- `ApprovalsApp.tsx` revoke-all grants now uses the shared in-app `ApprovalModal`.
|
||||
- `ArtifactCenterApp.tsx` permanent delete now uses the shared in-app `ApprovalModal`; `artifact-center-trust.test.tsx` and rendered `J3e` cover the no-native-confirm contract.
|
||||
- `MemoryCenterTab.tsx` permanent delete, GDPR erase, and allow re-import now use the shared in-app `ApprovalModal`; `memory-center-trust.test.tsx` and rendered `J3f` cover the no-native-confirm contract.
|
||||
- `WikiTab.tsx` Obsidian and Notion exports now use an in-app form dialog; `wiki-export-trust.test.tsx` and rendered `J3g` cover the no-native-prompt contract.
|
||||
- `SettingsApp.tsx` telemetry clear and Settings backup/restore now use in-app approval/status states; `settings-trust.test.tsx` and rendered `J3h` cover the no-native-dialog contract.
|
||||
- `BackupApp.tsx` restore now uses the shared in-app `ApprovalModal`; `p1b-authgate-surfaces.test.tsx` covers the no-native-confirm contract.
|
||||
- `AutomationCenterApp.tsx` delete now uses the shared in-app `ApprovalModal`; `phase3b-automation-center.test.tsx` covers the no-native-confirm contract.
|
||||
- `ComplianceTemplateModal.tsx` delete now uses the shared in-app `ApprovalModal`; `compliance-template-trust.test.tsx` covers the no-native-confirm contract.
|
||||
- `packages/admin-web/src/pages/Members.tsx` member removal now uses an in-app confirmation panel; `admin-pages.test.ts` covers the no-native-confirm contract.
|
||||
|
||||
## T10 Focus Findings
|
||||
|
||||
These are the high-confidence focus risks from the scoped `outline-none` scan. Lines that already include a clear same-element `focus-visible:ring-*` replacement were not listed here.
|
||||
|
||||
Implementation update 2026-07-08 T10: `AllWorkspacesApp` search, the Settings Prompt Shape select, and `WikiTab` search now have visible focus-ring replacements and are removed from the open list below.
|
||||
|
||||
Implementation update 2026-07-09 T10: `WorkspaceActionsMenu` rename and delete confirmation inputs now have explicit `label`/`htmlFor` wiring, `name`, `autocomplete="off"`, and token-based `focus-visible` rings guarded by `workspace-actions-menu.test.tsx`; they are removed from the open list below.
|
||||
|
||||
Implementation update 2026-07-09 T10: `CommandCenter` search now has a token-based `focus-visible` ring guarded by `p7-b3-command-center.test.tsx`; it is removed from the open list below.
|
||||
|
||||
Implementation update 2026-07-09 T10: warm `AskBar` input now has `name`, `autocomplete="off"`, and a token-based `focus-visible` ring guarded by `warm-primitives.test.tsx`; it is removed from the open list below.
|
||||
|
||||
Implementation update 2026-07-09 T10: workspace `TasksTab` add-task input now has `aria-label`, `name`, `autocomplete="off"`, and a token-based `focus-visible` ring guarded by `workspace-tasks-tab.test.tsx`; it is removed from the open list below.
|
||||
|
||||
Implementation update 2026-07-09 T10: `TimelineApp` event-type filter select now has `aria-label`, `name`, hidden decorative icon semantics, and a token-based `focus-visible` ring guarded by `timeline-app.test.tsx`; it is removed from the open list below.
|
||||
|
||||
Implementation update 2026-07-09 T10: `CreateWorkspaceDialog` template creator Description and Starter Memory textareas now have associated labels, `name`, `autocomplete="off"`, and token-based `focus-visible` rings guarded by `shell-overlay-contracts.test.tsx`; they are removed from the open list below.
|
||||
|
||||
Implementation update 2026-07-09 T10: `WorkspaceSwitcher` focus movement, Tab trap, Escape close, and focus return are now guarded by `shell-overlay-contracts.test.tsx`; it is removed from the open list below.
|
||||
|
||||
Implementation update 2026-07-09 T10: `ConnectorCard` row actions and Jira credential setup fields now have stable names/autocomplete metadata, email/token semantics, disabled spellcheck, and token-based focus rings guarded by `phase4b-connector-hub.test.tsx`; the ConnectorCard credential rows are removed from the open list below.
|
||||
|
||||
Implementation update 2026-07-09 T10: `ExtensionCard` marketplace inline connector-token paste now has an accessible connector-specific label, stable `name`, `autocomplete="off"`, disabled spellcheck, and token focus-ring coverage guarded by `phase4b-marketplace-extend.test.tsx`.
|
||||
|
||||
Implementation update 2026-07-09 T10: `InstallAuditPanel` marketplace audit type filter now has stable `name`/autocomplete metadata and token-based focus rings guarded by `phase4b-marketplace-extend.test.tsx`.
|
||||
|
||||
Implementation update 2026-07-09 T10: `TelemetryApp` daily budget input now has stable `name` and `autocomplete="off"` metadata guarded by `TelemetryApp.test.tsx`.
|
||||
|
||||
Implementation update 2026-07-09 T10: `SkillEditorDrawer` markdown textarea now has stable `name`, `autocomplete="off"`, and disabled spellcheck metadata guarded by `phase3c-skill-builder.test.tsx`.
|
||||
|
||||
Implementation update 2026-07-09 T10: `CreateAgentForm`, `CreateGroupForm`, and `GroupDetail` template controls now have associated labels or accessible names, stable `name` values, `autocomplete="off"`, token-based focus rings, and the group execution strategy exposes `aria-pressed`; `AgentTemplateForms.test.tsx` guards the flow.
|
||||
|
||||
Implementation update 2026-07-09 T10: `TemplatesView` search now exposes contextual accessible names, stable `name` values, and `autocomplete="off"` for persona/group template search; `phase3b-agent-center.test.tsx` guards the persona-template path.
|
||||
|
||||
Implementation update 2026-07-09 T10: `GroupCard` now uses separate select/delete buttons instead of nesting a delete button inside the card button; delete is named per group, both actions have token-based focus rings, and `GroupCard.test.tsx` guards the behavior.
|
||||
|
||||
Implementation update 2026-07-09 T10: `AgentCard` now uses separate select/delete buttons instead of an interactive delete control inside a selectable `role="button"` card; custom-agent delete stays named, both actions have token-based focus rings, and `AgentCard.test.tsx` guards the behavior.
|
||||
|
||||
Current high-confidence production scan: no remaining focus findings in this reviewed list. The broad `outline-none` count above remains historical until an AST/lint rescan.
|
||||
|
||||
## T8/T10 Transition Findings
|
||||
|
||||
Guideline rule: never use `transition-all`; list properties explicitly and prefer transform/opacity for compositor-friendly animation.
|
||||
|
||||
Implementation update 2026-07-09 T8/T10: `CreateGroupForm` strategy buttons now use `transition-colors`, `GroupCard`/`AgentCard` select/delete controls now use explicit transition properties, and `SuggestedAgentCards` now uses explicit color/transform transition properties for the browse affordance, so the prior `CreateGroupForm.tsx:81`, `GroupCard.tsx`, `AgentCard.tsx`, and `SuggestedAgentCards.tsx:144` rows are removed.
|
||||
|
||||
Implementation update 2026-07-09 T8/T10: `LoginBriefing` workspace rows, `SpawnAgentDialog` workspace/persona buttons, and the `ChatApp` session sidebar now use explicit color or width transitions guarded by `wave-w-briefing-entrance.test.tsx`, `SpawnAgentDialog.test.tsx`, and `wave-u-chat-action-row.test.tsx`, so the prior `LoginBriefing.tsx:344`, `SpawnAgentDialog.tsx:215`, `SpawnAgentDialog.tsx:270`, and `ChatApp.tsx:948` rows are removed.
|
||||
|
||||
Implementation update 2026-07-09 T8/T10: `CreateWorkspaceDialog` chip, template, storage, persona, and agent-group controls now use explicit color/transform transitions guarded by `shell-overlay-contracts.test.tsx`, so the prior `CreateWorkspaceDialog.tsx:163`, `:631`, `:653`, `:911`, `:949`, `:967`, `:1109`, `:1198`, and `:1218` rows are removed.
|
||||
|
||||
Implementation update 2026-07-09 T8/T10: `WorkspaceSwitcher` rows and `PersonaSwitcher` persona/group cards now use explicit color transitions guarded by `shell-overlay-contracts.test.tsx`, so the prior `WorkspaceSwitcher.tsx:59` and `PersonaSwitcher.tsx:188`/`:320` rows are removed.
|
||||
|
||||
Implementation update 2026-07-09 T8/T10: `ConnectorCard` rows, `BrandTile` shadows, and `McpCatalog` distribution/category controls now use explicit color, shadow, filter, or background/color/box-shadow transitions guarded by `phase4b-connector-hub.test.tsx` and `phase4b-mcp-hub.test.tsx`, so the prior `ConnectorCard.tsx:115`, `BrandTile.tsx:50`, and `McpCatalog.tsx:146`/`:207`/`:221` rows are removed.
|
||||
|
||||
Implementation update 2026-07-09 T8/T10: `TelemetryApp` daily budget meter, `SurfaceToggle` knob, and workspace `TasksTab` delete action now use explicit width, left/background-color, or opacity/color transitions guarded by `TelemetryApp.test.tsx`, `power-primitives.test.tsx`, and `workspace-tasks-tab.test.tsx`, so the prior `TelemetryApp.tsx:222`, `power-primitives.tsx:89`, and `TasksTab.tsx:60` rows are removed.
|
||||
|
||||
Implementation update 2026-07-09 T8/T10: `ArtifactCenterApp` artifact cards, `DashboardApp` workspace tiles, `HomeCockpit` recent workspace cards, and `MarketplaceApp` shelf chips now use explicit border/transform/shadow or background/color/box-shadow transitions guarded by `artifact-center-trust.test.tsx`, `DashboardApp.test.tsx`, `p2-home-desktop.test.tsx`, and `phase4b-marketplace-extend.test.tsx`, so the prior `ArtifactCenterApp.tsx:344`, `DashboardApp.tsx:207`, `HomeCockpit.tsx:449`, and `MarketplaceApp.tsx:373` rows are removed.
|
||||
|
||||
```text
|
||||
No remaining production `transition-all` hits in the scoped cockpit/admin/www scan.
|
||||
```
|
||||
|
||||
## T8/T10 Image Findings
|
||||
|
||||
All inspected image hits have `alt` or `alt=""` and explicit `width` and `height` attributes after focused fixes. The `AppShell` wallpaper at `apps/web/src/components/os/AppShell.tsx:415` already had explicit dimensions.
|
||||
|
||||
Implementation update 2026-07-09 T8/T10: `SuggestedAgentCards` persona avatars and roster thumbnails now have explicit `width`/`height` attributes guarded by `SuggestedAgentCards.test.tsx`, so the prior `SuggestedAgentCards.tsx:78` and `SuggestedAgentCards.tsx:124` rows are removed.
|
||||
|
||||
Implementation update 2026-07-09 T8/T10: `AgentCenterRow` list avatars now have explicit `width`/`height` attributes guarded by `phase3b-agent-center.test.tsx`, so the prior `AgentCenterRow.tsx:46` row is removed.
|
||||
|
||||
Implementation update 2026-07-09 T8/T10: onboarding `ReadyStep` logo media now has explicit `width`/`height` attributes guarded by `ReadyStep.test.tsx`, so the prior `ReadyStep.tsx:19` row is removed.
|
||||
|
||||
Implementation update 2026-07-09 T8/T10: `BootScreen` and `StatusBar` logo media now have explicit `width`/`height` attributes guarded by `r20-boot-reduced-motion-glow.test.tsx` and `StatusBar.test.tsx`, so the prior `BootScreen.tsx:132` and `StatusBar.tsx:111` rows are removed.
|
||||
|
||||
Implementation update 2026-07-09 T8/T10: `LoginBriefing`, the `ChatApp` empty state, and `SpawnAgentDialog` persona media now have explicit `width`/`height` attributes guarded by `wave-w-briefing-entrance.test.tsx`, `wave-u-chat-action-row.test.tsx`, and `SpawnAgentDialog.test.tsx`, so the remaining `LoginBriefing.tsx:228`, `ChatApp.tsx:1070`, and `SpawnAgentDialog.tsx:276`/`:388` rows are removed.
|
||||
|
||||
```text
|
||||
No remaining high-confidence image-dimension findings in the scoped cockpit scan.
|
||||
```
|
||||
|
||||
## T10 Form Metadata Queue
|
||||
|
||||
Loose JSX scan results from the pre-fix snapshot:
|
||||
|
||||
- 145 text-like controls missing `name`.
|
||||
- 107 controls missing `autoComplete`.
|
||||
- 115 controls without an obvious same-tag `id`, `aria-label`, or `aria-labelledby`.
|
||||
|
||||
Implementation update 2026-07-08 T10: representative high-traffic metadata fixes landed for Settings model/trust/team/KVARK controls, Profile identity fields, Launcher refresh/prompt, Approvals refresh/revoke grant buttons, All Workspaces search, and Wiki search. The broad counts above are retained as historical scan output and should be regenerated by an AST/lint pass before the final T10 closeout.
|
||||
|
||||
Implementation update 2026-07-09 T10: template custom-agent, agent-group creator, and group task-runner fields now have label associations, stable metadata, focus rings, and strategy pressed state, guarded by `AgentTemplateForms.test.tsx`.
|
||||
|
||||
Implementation update 2026-07-09 T10: `SpawnAgentDialog` launch task/new-workspace fields, `McpCatalog` catalog search, `ArtifactCenterApp` detail editor fields, `ModelGate` cloud-key/local-pull fields, inline `CapabilityRequestCard` connector token entry, and `TelegramDigestCard` credential fields now have associated labels or accessible names plus stable `name` and `autocomplete` metadata. Focused coverage: `SpawnAgentDialog.test.tsx`, `phase4b-mcp-hub.test.tsx`, `artifact-center-trust.test.tsx`, `ModelGate.test.tsx`, `pr4-inline-capability.test.tsx`, and `TelegramDigestCard.test.tsx`.
|
||||
|
||||
Implementation update 2026-07-09 T10: first-run onboarding profile name/role, workspace-name, and first-task controls now expose stable `name` and `autocomplete` metadata guarded by `WhoAreYouStep.test.tsx`, `WorkspaceCreateStep.test.tsx`, and `FirstTaskStep.test.tsx`.
|
||||
|
||||
Implementation update 2026-07-09 T10: `EraseDataDialog` destructive confirmation now associates its label with the phrase field, adds stable `name`/`autocomplete="off"` metadata, and uses a token `focus-visible` ring guarded by `EraseDataDialog.test.tsx`.
|
||||
|
||||
Implementation update 2026-07-09 T10: `McpScopeDialog` target-workspace select now has stable `name`/`autocomplete="off"` metadata and a token `focus-visible` ring guarded by `phase4b-mcp-hub.test.tsx`.
|
||||
|
||||
Implementation update 2026-07-09 T10: `CreateWorkspaceDialog` visible setup fields, template search, template-creator AI/name fields, and folder-picker new-folder field now expose stable `name`/`autocomplete="off"` metadata and accessible labels/names guarded by `shell-overlay-contracts.test.tsx`.
|
||||
|
||||
Implementation update 2026-07-09 T10: `ComplianceDashboard` report template/date controls and `ComplianceTemplateModal` create/edit fields now expose associated labels, stable `name`/`autocomplete` metadata, and token focus rings guarded by `ComplianceDashboard.test.tsx` and `compliance-template-trust.test.tsx`.
|
||||
|
||||
Implementation update 2026-07-09 T10: `FilesApp` transient new-folder and inline rename fields now expose accessible names, stable `name`/`autocomplete="off"` metadata, disabled spellcheck for file names, and token focus rings guarded by `StorageAndFilesApp.test.tsx`.
|
||||
|
||||
Implementation update 2026-07-09 T10: `FilesApp` bulk Move and file Properties dialogs now name their icon-only close controls, add token focus rings, and preserve the row-level Properties context menu instead of falling through to the empty-space menu; `StorageAndFilesApp.test.tsx` guards both flows.
|
||||
|
||||
Implementation update 2026-07-09 T10: `AutomationCenterApp` template workspace and assist-mode controls now expose stable `name`/`autocomplete` metadata and token focus rings guarded by `phase3b-automation-center.test.tsx`.
|
||||
|
||||
Implementation update 2026-07-09 T10: `MemoryTrustManage` search and correction editor now expose stable `name`/`autocomplete` metadata, and the correction editor has a token `focus-visible` ring guarded by `pr35-memory-trust-manage.test.tsx`.
|
||||
|
||||
Implementation update 2026-07-09 T10: `TimelineTab` search, filter toggle, and minimum-importance slider now expose accessible names/label association, stable metadata, and token focus rings guarded by `p7-b5-error-threading.test.tsx`.
|
||||
|
||||
Implementation update 2026-07-09 T10: `EvolutionTab` proposal review note now associates its visible label with the textarea, exposes stable `name`/`autocomplete` metadata, and uses a token focus ring guarded by `EvolutionTab.test.tsx`.
|
||||
|
||||
Implementation update 2026-07-09 T10: `EvolutionTab` New Run modal now names the close icon, associates Target Kind, Target Name, Baseline, and Schema baseline controls with stable `id`/`name` metadata, sets select and textarea `autocomplete="off"`, and uses token focus rings guarded by `EvolutionTab.test.tsx`.
|
||||
|
||||
Implementation update 2026-07-09 T10: `MemoryCard` selection checkboxes now expose memory-specific accessible names plus stable `name`/`value` metadata guarded by `MemoryCard.test.tsx`.
|
||||
|
||||
Implementation update 2026-07-09 T10: `SettingsApp` Prompt Shape and `TimelineApp` event-type selects now set `autocomplete="off"` with focused coverage in `settings-trust.test.tsx` and `timeline-app.test.tsx`.
|
||||
|
||||
Implementation update 2026-07-09 T10: `WikiTab` search now has an input-level token focus ring, and Obsidian/Notion export target fields now expose target-specific `name`, `autocomplete="off"`, and token focus rings guarded by `wiki-export-trust.test.tsx`.
|
||||
|
||||
Implementation update 2026-07-09 T10: `UserProfileApp` Analyze Style action now has an explicit button type and token focus ring guarded by `UserProfileApp.test.tsx`.
|
||||
|
||||
Implementation update 2026-07-09 T10: `ComplianceDashboard` report options and `ComplianceTemplateModal` template fields now use token `focus-visible:ring-2` focus rings guarded by `ComplianceDashboard.test.tsx` and `compliance-template-trust.test.tsx`.
|
||||
|
||||
Implementation update 2026-07-09 T10: `MemoryTrustManage` search now pairs its stable metadata with a token focus ring guarded by `pr35-memory-trust-manage.test.tsx`.
|
||||
|
||||
Implementation update 2026-07-09 T10: `ChatApp` message composer now pairs its stable metadata with a token focus ring guarded by `lane-c-input-power.test.tsx`.
|
||||
|
||||
Implementation update 2026-07-09 T10: `ModelPilotCard` budget threshold slider now exposes an accessible name, stable `name`, and token focus ring while preserving update behavior guarded by `ModelPilotCard.test.tsx`.
|
||||
|
||||
Implementation update 2026-07-09 T10: `AllWorkspacesApp` search now pairs its stable metadata with an input-level token focus ring guarded by `AllWorkspacesApp.test.tsx`.
|
||||
|
||||
Implementation update 2026-07-09 T10: `ArtifactCenterApp` detail Kind select now pairs its stable metadata with a token focus ring guarded by `artifact-center-trust.test.tsx`.
|
||||
|
||||
Implementation update 2026-07-09 T10: `LauncherApp` optional launch prompt now pairs its stable metadata with a token focus ring guarded by `launcher-a11y.test.tsx`.
|
||||
|
||||
Representative high-traffic rows:
|
||||
|
||||
```text
|
||||
apps/web/src/components/os/apps/VaultApp.tsx:329 - secret name input lacks name/autocomplete and explicit label hook
|
||||
apps/web/src/components/os/apps/VaultApp.tsx:361 - secret type select lacks name and explicit label hook
|
||||
apps/web/src/components/os/apps/VaultApp.tsx:373 - username/email input lacks name/autocomplete and should disable spellcheck
|
||||
apps/web/src/components/os/apps/VaultApp.tsx:379 - secret value password lacks name/autocomplete
|
||||
apps/web/src/components/os/apps/ChatApp.tsx:1589 - chat composer textarea lacks name/autocomplete
|
||||
apps/web/src/components/os/apps/agents/AgentBuilder.tsx:230 - agent name input lacks name/autocomplete
|
||||
apps/web/src/components/os/apps/automations/AutomationBuilder.tsx:305 - automation name input lacks name/autocomplete
|
||||
apps/web/src/components/os/apps/memory/MemoryCenterTab.tsx:406 - memory search input lacks name/autocomplete and explicit label hook
|
||||
packages/admin-web/src/App.tsx:76 - team slug input lacks name/autocomplete and explicit label association
|
||||
```
|
||||
|
||||
Recommended correction pattern:
|
||||
|
||||
- Add `id` plus `htmlFor`, or a clear `aria-label` where visual labels are intentionally absent.
|
||||
- Add stable `name` values for all real form controls.
|
||||
- Add `autoComplete="off"` for non-auth/system fields and meaningful autocomplete tokens for URL, email, username, and current/new password fields.
|
||||
- Use correct `type`, `inputMode`, and `spellCheck={false}` for URLs, emails, codes, tokens, and usernames.
|
||||
- Keep placeholders as examples, not as the only label.
|
||||
|
||||
## Locale And Date Queue
|
||||
|
||||
Scoped scan found 114 `new Date(...)` / `toLocale*` hits. Some are internal sorting/parsing and some already use explicit locale constants, but the pass should standardize user-visible formatting through helpers backed by `Intl.DateTimeFormat` / `Intl.NumberFormat`.
|
||||
|
||||
High-priority examples:
|
||||
|
||||
```text
|
||||
apps/web/src/components/os/apps/HomeCockpit.tsx:98 - user-visible date formatting is test-coupled; keep locale explicit if changed
|
||||
apps/web/src/components/os/apps/TimelineApp.tsx - top scoped file by date/locale hits
|
||||
apps/web/src/components/os/StatusBar.tsx - user-visible time/date area needs hydration and locale review
|
||||
apps/web/src/components/os/apps/FilesApp.tsx - file dates/sizes should share one formatting helper
|
||||
apps/web/src/components/os/apps/cockpit/ComplianceDashboard.tsx - compliance date filters and report dates need consistent locale semantics
|
||||
```
|
||||
|
||||
## Commands Used
|
||||
|
||||
```powershell
|
||||
rg -n "\b(confirm|alert|prompt)\s*\(" apps/web/src apps/www packages/admin-web/src --glob '*.ts' --glob '*.tsx'
|
||||
rg -n "transition-all" apps/web/src/components/os packages/admin-web/src apps/www/app --glob '*.tsx' --glob '*.ts'
|
||||
rg -n "outline-none|focus:outline-none" apps/web/src/components/os packages/admin-web/src apps/www/app --glob '*.tsx' --glob '*.ts'
|
||||
rg -n -A6 "<img" apps/web/src/components/os --glob '*.tsx'
|
||||
```
|
||||
|
||||
The form metadata counts came from a loose PowerShell JSX tag scanner over `<Input>`, `<input>`, `<textarea>`, and `<select>` and should be regenerated or replaced by an AST-based lint before implementation.
|
||||
83
docs/audits/2026-07-11-channels-ux-hardening.md
Normal file
83
docs/audits/2026-07-11-channels-ux-hardening.md
Normal file
@@ -0,0 +1,83 @@
|
||||
# Channels UX and Reliability Hardening
|
||||
|
||||
**Date:** 2026-07-11
|
||||
**Branch:** `codex/channels-ux-hardening` from `origin/main` (`89329f99`)
|
||||
**Scope:** Telegram, Discord, Slack, WhatsApp settings, protected routes, inbound agent turns, pairing, workspace routing, approvals, persistence, responsive UI, clean install, and real local runtime.
|
||||
|
||||
## Gate status
|
||||
|
||||
The Channels implementation is code-complete and locally release-ready. All hermetic gates pass, the production UI was exercised against a real bearer-protected sidecar, and the real Baileys transport reached QR pairing without writing plaintext authentication state.
|
||||
|
||||
External release evidence is still required for real provider credentials and a real WhatsApp device scan. That evidence is intentionally not represented as complete below.
|
||||
|
||||
## Defects found and corrected
|
||||
|
||||
| Severity | Defect | Correction |
|
||||
|---|---|---|
|
||||
| P0 | Settings used raw `fetch`, so protected channel routes returned 401. | Added typed authenticated adapter methods and moved the UI entirely onto them. |
|
||||
| P0 | Channel-to-agent loopback called protected `/api/chat` without a bearer token. | Threaded the sidecar session token through `ChannelManager` and `runChannelChatTurn`; added a real Fastify auth integration test. |
|
||||
| P0 | `package-lock.json` did not describe the merged Channels dependency graph; clean install failed. | Regenerated the lock, aligned `tsx`/`esbuild`, added the Testing Library peer, and proved normal `npm ci --legacy-peer-deps`. |
|
||||
| P0 | Baileys persisted WhatsApp account/session keys as plaintext JSON. | Replaced multi-file auth persistence with one encrypted Vault entry, added one-time legacy migration, removed plaintext only after the encrypted write succeeds, and wipes both stores on logout. Corrupt state now fails closed and is preserved for recovery. |
|
||||
| P1 | Headless channel turns could wait five minutes for an SSE approval nobody was watching. | Channel turns now use `proposeHeld`; gated actions enter the durable approval queue and the channel receives the app-approval response immediately. |
|
||||
| P1 | Config writes could partially save secrets before a later field failed validation. | Validate the entire request, secret key set, values, enabled flag, and workspace before any Vault write. |
|
||||
| P1 | Config changes were not represented in the audit stream. | Added `channel_config_change`; audit payloads contain secret key names only, never values. |
|
||||
| P1 | Duplicate provider deliveries could execute a turn twice; simultaneous messages in one chat could interleave. | Added bounded 24-hour message-ID deduplication and a per-chat promise queue. |
|
||||
| P1 | `/workspace` required opaque IDs and `/status` exposed raw IDs. | Resolve names or IDs, persist the stable ID, and show `Name (id)` in status output. |
|
||||
| P1 | Channel controls were unreadable in narrow windows because the nested Settings rail consumed most of the viewport. | Settings tabs become a horizontal scroller below 640 px; active tabs scroll into view; channel forms, selectors, and actions stack without page overflow. |
|
||||
| P2 | A deep link to a disclosure-hidden tab stayed on General after the user enabled Standard. | Re-resolve `?tab=channels` when the disclosure tier changes. |
|
||||
| P2 | Failed saves cleared or obscured useful state and several actions had weak busy/error feedback. | Preserve secret drafts on failure, expose alert/status semantics, and consistently disable busy or prerequisite-blocked actions with visible reasons. |
|
||||
| P2 | Baileys' default logger printed verbose device-pairing payloads. | Pass a silent internal logger and retain only curated Waggle lifecycle/error messages. |
|
||||
|
||||
## Scenario coverage
|
||||
|
||||
| Scenario | Evidence | Result |
|
||||
|---|---|---|
|
||||
| Unauthenticated management request | Real Fastify security middleware | 401, no state change |
|
||||
| Authenticated browser bootstrap | Production app + fresh sidecar | Session token fetched; all Channels GET/POST requests 200 with bearer header |
|
||||
| Atomic invalid config | Route integration | Unknown secret or workspace rejected; earlier fields are not persisted |
|
||||
| Secret save | Real browser + isolated Vault | Draft clears only after success; masked saved state appears; Start becomes available |
|
||||
| Bad provider credential | Real Telegram start | Honest `Unauthorized` error, transport remains stoppable |
|
||||
| Deny-by-default pairing | Manager/pairing tests | Unknown senders receive silence; only valid one-use code pairs |
|
||||
| Pair expiry/case/reuse | Pairing tests | Ten-minute TTL, case-insensitive entry, single use |
|
||||
| Workspace name/id routing | Manager tests | Human name resolves to stable ID; per-chat override remains scoped |
|
||||
| Duplicate inbound delivery | Manager tests | Same provider message ID executes once |
|
||||
| Concurrent same-chat turns | Manager tests | Replies remain in arrival order |
|
||||
| Injection rejection | Loopback client test | Scanner 400 becomes a safe user-facing channel error |
|
||||
| Held tool approval | Chat route + client/manager tests | Durable approval notification; no unwatched five-minute wait |
|
||||
| Platform send limits/reconnects | Telegram/Discord/Slack/WhatsApp adapter tests | Chunking, recovery, stop, and error states pass |
|
||||
| WhatsApp encrypted persistence | Auth-state tests | Credentials and signal keys round-trip through Vault; no plaintext files |
|
||||
| Existing WhatsApp pairing migration | Auth-state test | Multi-file state imported, encrypted, then legacy directory removed |
|
||||
| Real Baileys startup | Fresh sidecar on port 34292 | Outbound connection succeeds, QR appears, no plaintext auth dir, verbose upstream logs suppressed |
|
||||
| Desktop UI | Playwright at 1280x720 | No overlap; full workflow remains scannable |
|
||||
| Narrow-window UI | Playwright at 390x844 | No document overflow; active tab visible; fields/actions stack and remain usable |
|
||||
| Deep-link disclosure recovery | Real browser | Essential falls back to General; switching to Standard opens requested Channels tab |
|
||||
|
||||
## Verification evidence
|
||||
|
||||
- `npm ci --legacy-peer-deps`: pass, 1,769 packages installed.
|
||||
- Backend Channels lane: 9 files, **89/89 tests pass**.
|
||||
- Web Channels/deep-link lane: 2 files, **12/12 tests pass**.
|
||||
- `npx tsc --noEmit --project packages/server/tsconfig.json`: pass.
|
||||
- `npm run build:packages`: pass.
|
||||
- `npm run build`: pass.
|
||||
- `git diff --check`: pass.
|
||||
- Fresh sidecar boot with a new data directory: pass.
|
||||
- Real browser authenticated Vault save/start/error/stop flow: pass.
|
||||
- Real Baileys unpaired QR flow: pass.
|
||||
|
||||
## Remaining release evidence
|
||||
|
||||
These are not code failures, but they must be completed before calling Channels production-proven:
|
||||
|
||||
1. Telegram bot: real token, pair from owner account, inbound answer, workspace switch, duplicate-delivery observation, revoke.
|
||||
2. Discord bot: real app/guild install, Message Content intent, DM and channel message, reconnect, revoke.
|
||||
3. Slack app: real Socket Mode app/bot tokens, DM and channel message, reconnect, revoke.
|
||||
4. WhatsApp: scan with a secondary number, restart persistence, inbound/outbound message, unlink/logout wipe, re-pair.
|
||||
5. Trigger one genuinely gated tool from a real channel and approve it in the desktop Approvals surface.
|
||||
|
||||
The founder-approved WhatsApp Terms-of-Service and ban risk remains visible and is not treated as solved by the credential hardening.
|
||||
|
||||
## Integration notes outside this branch
|
||||
|
||||
- `origin/main` still emits the known Tailwind arbitrary-duration warnings, the shape-selection chunk warning, and a >500 kB bundle warning. The broader UX branch already contains related build-polish work; do not duplicate it here.
|
||||
- The served page reports a CSP rejection for the inline theme bootstrap script. It did not block Channels, but it remains global browser-console debt and should be resolved in the broader UX integration branch.
|
||||
90
docs/audits/2026-07-12-goal-integration-checkpoint.md
Normal file
90
docs/audits/2026-07-12-goal-integration-checkpoint.md
Normal file
@@ -0,0 +1,90 @@
|
||||
# Goal Integration Checkpoint - 2026-07-12
|
||||
|
||||
## Scope
|
||||
|
||||
Clean integration branch: `codex/goal-integration-2026-07-12`
|
||||
|
||||
Base: `origin/main` at `89329f99`
|
||||
|
||||
This checkpoint combines the current mainline with:
|
||||
|
||||
- hardened Channels from `015e20e4`
|
||||
- durable Rooms and external-tool collaboration from `39511843`
|
||||
- provider-native model discovery and managed LiteLLM runtime routing
|
||||
- recoverable API-key setup in onboarding and Settings
|
||||
- mobile onboarding, CSP, and dependency fixes found during combined validation
|
||||
|
||||
The original dirty UX checkout was not merged wholesale. Its unrelated working
|
||||
files remain untouched.
|
||||
|
||||
## Functional Result
|
||||
|
||||
- Provider model identities come from provider APIs, not a maintained model list.
|
||||
- Anthropic cursor and Google page-token pagination are exhausted.
|
||||
- Provider models use stable `provider/model` IDs through UI, Settings, Chat,
|
||||
explicit model selection, fleet execution, and LiteLLM runtime configuration.
|
||||
- Saving a key hydrates provider environment aliases, stores the key in Vault,
|
||||
scrubs legacy plaintext config, refreshes catalogs, and restarts the managed
|
||||
router when needed.
|
||||
- A model released while Waggle is open is pulled again on focus and becomes
|
||||
executable without a Waggle code change.
|
||||
- API-key success and router readiness are separate states. Router failure keeps
|
||||
an inline retry and does not claim that the model is ready.
|
||||
- Channels and durable Rooms coexist with the current MCP retrieval, marketplace,
|
||||
evolution, embedding-routing, and security changes on main.
|
||||
|
||||
## Verification
|
||||
|
||||
- `npm ci --legacy-peer-deps`: pass
|
||||
- `npm run build:packages`: pass
|
||||
- root `npm run build`: pass
|
||||
- server TypeScript project: pass
|
||||
- web TypeScript project: pass
|
||||
- app TypeScript project: pass
|
||||
- combined backend slice: 21 files, 221 tests passed
|
||||
- combined web slice: 14 files, 94 tests passed
|
||||
- agent, hook, shim, and worker slice: 14 files, 200 tests passed
|
||||
- hive-mind MCP server build and focused scope tests: 13 tests passed
|
||||
- fresh-port Playwright: 5/5 for app load, navigation, Settings/Models, Room,
|
||||
and zero console errors
|
||||
- 390 x 844 rendered inspection: onboarding/API-key, Settings/Models, and Channels
|
||||
fit without horizontal overflow; selected tabs remain visible; console errors 0
|
||||
- diff check: pass
|
||||
- added-line credential scan outside tests/docs: 0 candidates
|
||||
- production dependency audit: 0 high, 0 critical; 19 moderate remain
|
||||
- full workspace dependency audit: 0 high, 0 critical; 24 moderate remain
|
||||
|
||||
## Corrections Found During Integration
|
||||
|
||||
1. The root production build caught a stale frontend `Settings` type that omitted
|
||||
`defaultModel`; the contract now matches the model-save path.
|
||||
2. The production CSP blocked the inline pre-hydration theme script. It now runs
|
||||
as a same-origin classic script without weakening `script-src 'self'`.
|
||||
3. Tall onboarding steps were vertically centered on mobile, clipping content
|
||||
above the scroll origin. Mobile steps now align to the top and remain centered
|
||||
on larger viewports.
|
||||
4. `hono` 4.12.29, `vite` 6.4.3, `vitest` 3.2.7, and `form-data` 4.0.6 now
|
||||
resolve across the workspace, clearing every high- and critical-severity
|
||||
dependency advisory without a breaking application-code change.
|
||||
|
||||
## Open Build Hygiene
|
||||
|
||||
- Tailwind reports ambiguous arbitrary motion utility classes.
|
||||
- Vite reports mixed dynamic and static imports for shape selection.
|
||||
- The production build reports chunks larger than 500 kB.
|
||||
- Focused tests still emit known image-source and expected error-path stderr;
|
||||
the fresh browser run remains free of console errors.
|
||||
|
||||
## Remaining Goal Gates
|
||||
|
||||
- paid external-provider credential smoke against at least one live provider
|
||||
- live Telegram, Discord, and Slack credentials; WhatsApp scan/restart/unlink
|
||||
- one real held-tool approval arriving through a Channel
|
||||
- packaged Tauri desktop and signed installer evidence
|
||||
- remaining T13-T19 launch, utility, hook, Browser Companion, CI, and deployment
|
||||
evidence, or explicit scope deferrals
|
||||
- reconcile the broader dirty UX correction checkout in deliberate slices
|
||||
- run the final five-persona scorecards; no persona is yet formally confirmed at
|
||||
9/10
|
||||
|
||||
This branch is local and has not been pushed.
|
||||
144
docs/audits/2026-07-13-final-goal-verification.md
Normal file
144
docs/audits/2026-07-13-final-goal-verification.md
Normal file
@@ -0,0 +1,144 @@
|
||||
# Final UX Goal Verification - 2026-07-13
|
||||
|
||||
## Verdict
|
||||
|
||||
The five-persona in-product UX gate is met on commit `2a672bda`:
|
||||
|
||||
| Persona | Score | Verdict |
|
||||
|---|---:|---|
|
||||
| Solo founder | 9.4/10 | Pass |
|
||||
| Researcher | 9.5/10 | Pass |
|
||||
| Engineer / power user | 9.3/10 | Pass |
|
||||
| Team admin / security reviewer | 9.1/10 | Pass |
|
||||
| Mobile executive | 9.3/10 | Pass |
|
||||
|
||||
No persona is averaged away. All five independently clear 9/10, no normalized
|
||||
dimension is below 8/10, and no rubric score cap was triggered.
|
||||
|
||||
This is not a claim that public release operations are complete. Product UX and
|
||||
release availability are reported separately so an unavailable domain, missing
|
||||
signing, or an untested paid external account cannot be hidden inside a score.
|
||||
|
||||
## Scored Evidence
|
||||
|
||||
The final deterministic judge run passed 5/5. Its state bundles cover:
|
||||
|
||||
- 15 primary route states.
|
||||
- 25 offline, unavailable, slow, large-data, success, cancellation, and trust states.
|
||||
- 3 selected overlays with open, fit, Escape-close, and focus behavior.
|
||||
- Desktop 1440 x 900 and mobile 390 x 844 viewports.
|
||||
- 0 critical console errors, 0 page errors, and 0 unexpected critical network failures.
|
||||
- 0 visible horizontal-overflow findings in the captured route, failure, and overlay states.
|
||||
|
||||
The persona lanes were:
|
||||
|
||||
- Solo founder: Home continuity, Profile, chat-offline recovery, and Workspace Switcher.
|
||||
- Researcher: Memory, Artifacts, Timeline, unavailable/slow/large memory, large timeline, and failed export.
|
||||
- Engineer: Tool Launcher, MCP Hub, Files, degraded health, tool detection failure, slow/large agents, marketplace recovery, and file operations.
|
||||
- Team admin: Billing, Vault, Team settings, Approvals, active Team state, checkout success/cancel/unavailable, backup failures, and revoke-all confirmation.
|
||||
- Mobile executive: Home, Models/Settings, Memory, unavailable local runtime, chat-offline recovery, Notifications, and Command Center.
|
||||
|
||||
Screenshots were captured only after visible `aria-busy` loaders and known route
|
||||
loading labels settled. Animations were disabled during capture. The settled
|
||||
Home, Memory, Launcher, Approvals, mobile Settings, and mobile Command Center
|
||||
screens were inspected directly.
|
||||
|
||||
## Current-Head Quality Gates
|
||||
|
||||
All of these gates passed on the integrated branch:
|
||||
|
||||
- Root Vitest: 2,683 suites; 8,613 tests; 8,611 passed and 2 pending/skipped.
|
||||
- Full product browser audit: 204/204.
|
||||
- User journeys, network loss, and corrected Spawn Agent lane: 36/36.
|
||||
- Runtime accessibility: 2/2 desktop/mobile matrices with zero axe violations across the core route set, including Channels.
|
||||
- Visual and UAT regression: 27/27 in dark, light, desktop, mobile, and constrained viewports.
|
||||
- Performance regression: 13/13.
|
||||
- Provider model discovery: 43/43.
|
||||
- Channels: 105 backend tests and 7 focused UI tests; loopback manager/auth slice 29/29.
|
||||
- PostgreSQL/Redis integration: 19 files and 161 tests.
|
||||
- Admin: 42 unit tests and 14 rendered Chromium tests.
|
||||
- TypeScript: web, server, and Tauri app projects pass.
|
||||
- Lint: full repository pass.
|
||||
- Production build: `build:all` pass; web build transformed 2,616 modules.
|
||||
- Dependency audit for the standalone Tauri app: 0 vulnerabilities after lock refresh.
|
||||
|
||||
## Functional Corrections Included
|
||||
|
||||
The integrated work closes the main UX and runtime failures found during the
|
||||
complete audit:
|
||||
|
||||
- API-key onboarding and Settings use the same recoverable contract and keep key validation separate from router readiness.
|
||||
- Provider catalogs are fetched from provider APIs. New provider models can appear without a Waggle source-code release.
|
||||
- Anthropic cursor and Google page-token pagination are exhausted; Perplexity uses its current `/v1/models` catalog endpoint.
|
||||
- Stable provider/model identities reach Settings, Chat, Spawn Agent, fleet execution, and managed LiteLLM configuration.
|
||||
- Channels are integrated with protected loopback auth, encrypted WhatsApp state, atomic validation, durable approvals, duplicate suppression, same-chat ordering, mobile controls, and honest prerequisites/errors.
|
||||
- Persisted channel chat session IDs use a collision-resistant versioned encoding.
|
||||
- Home continuity, mobile onboarding, first-task handoff, overlay semantics, trust confirmations, focus metadata, light mode, visual baselines, and narrow-viewport layout are corrected and regression-locked.
|
||||
- The onboarding coachmark remains in bounds at constrained desktop width.
|
||||
- Memory, Launcher, Team, failure, and large-data judge states settle into inspectable UI rather than transient placeholders.
|
||||
|
||||
## Desktop Distribution Proof
|
||||
|
||||
The Tauri 2 debug package was built from the integrated source. The build produced:
|
||||
|
||||
- `app/src-tauri/target/debug/waggle.exe` (39.6 MiB).
|
||||
- `app/src-tauri/target/debug/bundle/msi/Waggle_0.2.0_x64_en-US.msi` (152.4 MiB).
|
||||
- `app/src-tauri/target/debug/bundle/nsis/Waggle_0.2.0_x64-setup.exe` (97.6 MiB).
|
||||
|
||||
The bundled application launched, created its tray, started the bundled Node
|
||||
sidecar, initialized the in-process embedder, and returned healthy from
|
||||
`/health`. An administrative MSI extraction produced 15,473 files and a 503.8
|
||||
MiB payload. The executable launched from that extracted MSI payload with a new
|
||||
data directory, used the packaged `resources/node.exe`, reached healthy, and
|
||||
released its port on shutdown.
|
||||
|
||||
This proves packaging and packaged startup. It does not prove production code
|
||||
signing or signed updater delivery.
|
||||
|
||||
## Dynamic Model Contract
|
||||
|
||||
Model availability is API-driven. The live endpoint audit returned expected
|
||||
authentication responses for Anthropic, OpenAI, Google, DeepSeek, xAI, Mistral,
|
||||
Alibaba, MiniMax, Zhipu, and Moonshot; OpenRouter and the current Perplexity
|
||||
catalog endpoint returned catalogs without a Waggle-maintained model allowlist.
|
||||
|
||||
Last-known catalogs remain available during a provider outage, and the UI says
|
||||
when refresh failed. Focus refresh makes newly released models available while
|
||||
Waggle is open. Empty maintained model arrays are intentional; they prevent a
|
||||
stale hardcoded catalog from becoming the product truth.
|
||||
|
||||
## Honest Release Boundary
|
||||
|
||||
The in-product five-persona goal is met. Public release readiness is not yet a
|
||||
9/10 claim because these external gates remain:
|
||||
|
||||
- `waggle-os.ai` and `www.waggle-os.ai` did not resolve during the live DNS check.
|
||||
- The available Vercel credential was invalid, no Vercel project/org secrets were present, and no Hostinger API token was available.
|
||||
- Production signing and signed updater artifacts are not available.
|
||||
- Paid/live third-party credentials were unavailable for Telegram, Discord, Slack, Stripe checkout, and a fully paired WhatsApp lifecycle.
|
||||
- A real provider-held tool approval arriving through a live Channel remains a credential-dependent smoke.
|
||||
|
||||
These do not invalidate the measured product UX score, but they block a claim
|
||||
that the public launch and every external integration are production-live. They
|
||||
require domain/deployment ownership, signing material, or third-party accounts;
|
||||
they cannot be completed honestly from repository code alone.
|
||||
|
||||
## Reproduction
|
||||
|
||||
Key commands used for the current-head evidence:
|
||||
|
||||
```powershell
|
||||
npm run lint
|
||||
npx tsc --noEmit --project packages/server/tsconfig.json
|
||||
npx tsc --noEmit --project apps/web/tsconfig.app.json
|
||||
npx tsc --noEmit --project app/tsconfig.json
|
||||
npm run build:all
|
||||
npm run test -- --run
|
||||
npx playwright test tests/e2e/runtime-a11y.spec.ts --project=chromium
|
||||
npx playwright test tests/e2e/visual-regression.spec.ts --project=chromium
|
||||
npx playwright test tests/e2e/five-persona-state-bundles.spec.ts --project=chromium
|
||||
```
|
||||
|
||||
The generated persona evidence lives under
|
||||
`output/playwright/five-persona-state-bundles/` during a local run and is not a
|
||||
source-controlled product artifact.
|
||||
Reference in New Issue
Block a user