86 lines
4.4 KiB
Markdown
86 lines
4.4 KiB
Markdown
# Waggle OS production-readiness audit — 2026-07-16
|
|
|
|
> **Historical snapshot — superseded.** This document records the state on
|
|
> 2026-07-16 and is not current ship authority. Use
|
|
> [`09-LAUNCH_RECOMMENDATION.md`](./09-LAUNCH_RECOMMENDATION.md) for the active
|
|
> Windows Solo gate. Cursor and OpenClaw are now roadmap-only; macOS
|
|
> certification is deferred.
|
|
|
|
## Executive verdict
|
|
|
|
The branch is not yet production-ready for the requested claim of a fully
|
|
Docker-independent Solo install or a verified 9.5/10 result across ten fresh
|
|
personas. It is materially improved and buildable: Windows sandbox/process
|
|
handling is hardened, chat schemas are bounded before model serialization, and
|
|
artifact/CLI writes now require approval.
|
|
|
|
## Verified changes
|
|
|
|
- Windows workspace/path, environment, direct-argv `run_code`, descendant
|
|
process termination, CLI `.cmd` discovery, MCP env sanitization, and MCP
|
|
start/stop race handling were implemented.
|
|
- Chat tool selection is deterministic and subtractive: maximum 14 tools and
|
|
8,000 exact OpenAI/LiteLLM schema characters per turn; duplicate names keep
|
|
the native definition; plugin/MCP tools cannot appear in casual chat unless
|
|
explicitly relevant; governance-blocked schemas are removed before model
|
|
serialization.
|
|
- `cli_execute`, `multi_edit`, DOCX/XLSX/PPTX/PDF generation are approval-gated.
|
|
- DOCX/PDF/PPTX/XLSX workspace containment uses `path.relative`, preventing
|
|
sibling-prefix escapes such as `workspace-sibling`.
|
|
- Coder, analyst, finance, data-engineer, executive-assistant, and general
|
|
purpose persona allowlists include their shipped code/artifact capabilities.
|
|
|
|
## Verification evidence
|
|
|
|
- Focused selector/chat tests: 61/61.
|
|
- Persona/agent/MCP/governance regression: 118/118.
|
|
- Windows/sandbox/CLI/MCP focused tests: 140/140.
|
|
- Confirmation, placeholder, persona, and CLI comprehensive contracts: 148/148.
|
|
- Production build: `npm run build:all` passed (package builds, server and web
|
|
typecheck, Vite production bundle).
|
|
- Full root Vitest run: 8,839 passed, 2 skipped, with remaining failures from
|
|
two CLI package-install/REPL tests timing out at 120s/180s in this Windows
|
|
checkout. Those timeouts need a separate launcher/runtime investigation.
|
|
|
|
## Tool-context result
|
|
|
|
Before this branch, a live general-purpose request exposed 78 tool definitions
|
|
(about 41.8k schema characters); coder exposed 36 (about 17.7k). Repeated
|
|
turns re-sent roughly 26.6k tokens of schemas. The route now selects and
|
|
serializes at most 14 tools / 8k schema characters, with a p95 selector test
|
|
under 10ms on a 78-tool synthetic pool.
|
|
|
|
## External-agent smoke result
|
|
|
|
Installed local CLIs detected: Claude Code 2.1.211, Codex 0.144.1, Hermes
|
|
0.18.2, Cursor 3.1.15, OpenClaw 2026.6.11, and Gemini. Claude and Codex
|
|
headless smoke paths ran; Hermes/OpenClaw were not run against persistent user
|
|
state. Launcher mappings still need runtime fixes for Hermes (`hermes chat -q`),
|
|
OpenClaw (`openclaw agent --message`), Codex detached/resume behavior, and
|
|
Cursor workspace binding.
|
|
|
|
## Open P0/P1 blockers
|
|
|
|
1. Solo/Tauri packaging still sets `WAGGLE_SKIP_LITELLM=1`; the built-in proxy
|
|
is Anthropic-only, while model resolution can still select a keyed
|
|
non-Anthropic provider. No clean Windows install has proven models, proxy,
|
|
and routing with Node/Python/Ollama/Docker absent.
|
|
2. There is no bundled local chat model/runner/weights; the current “local”
|
|
fallback can select a cloud Ollama tag (`ollama/minimax-m2.7:cloud`).
|
|
3. LiteLLM/Python bundling and resource layout are not yet wired as a tested
|
|
self-contained Solo artifact; embeddings may still download at runtime.
|
|
4. Smart router behavior remains heuristic and is not yet evaluated against
|
|
legal/payroll/destructive/verification adversarial cases.
|
|
5. A full browser E2E pass and three fresh runs for each of the ten requested
|
|
personas have not completed; therefore no 9.5/10 persona score is claimed.
|
|
6. Full root test timeouts in CLI package-install/REPL paths remain open.
|
|
|
|
## Persona acceptance matrix to run next
|
|
|
|
Run three fresh sessions each for general-purpose, researcher, writer,
|
|
project-manager, executive-assistant, finance-owner, coder, data-engineer,
|
|
verifier, and coordinator. Score artifact correctness, conversation quality,
|
|
tool/provenance fidelity, memory, safety, recovery, UX/latency, and accessibility
|
|
using the 100-point rubric. Auto-fail fabricated memory, unapproved mutation,
|
|
read-only writes, secret/sandbox escape, corruption, hangs, or false tool claims.
|