Vision-Based E2E Harness

A vision gate for the Waggle desktop-OS UI: a model judges screenshots for meaning ("does this actually look and work right?"), not pixels. It catches what the pixel-diff suite (tests/visual/views.spec.ts, toHaveScreenshot) structurally cannot — e.g. a baseline that is the wrong content but pixel- stable. Design + rationale: docs/audits/2026-06-01-vision-e2e-harness-design.md.

Architecture: Option C (hybrid) — deterministic Playwright capture → a multi-agent Workflow grades meaning → a reducer cross-checks the vision verdict against objective console signals (a vision-PASS with a real console error is downgraded to FAIL — the "objective floor").

Two phases

1. Capture — `capture.spec.ts`

Drives the real shell deterministically (?skipOnboarding=true&tier=power, openAppViaDock with Ops/Extend zone trays, data-theme for light/dark) across 17 surfaces × {dark,light} + memory + 2 overlays + 4 flows (42 captures). Per surface it writes artifacts/<surface>-<theme>.png + <surface>-<theme>.json ({expectation, consoleErrors[], networkFailures[]}).

Chat round-trip is graded on Path 2 (real LLM) per the product decision, so the capture server must run without --skip-litellm (a real provider key in the vault → the anthropic-proxy returns real replies):

# 1. build (packages + web) and start a real-LLM server on :3333
npm run build:all
WAGGLE_TRUST_LOCALHOST=1 node --env-file=.env \
  node_modules/tsx/dist/cli.mjs packages/server/src/local/start.ts   # NO --skip-litellm
# 2. capture (reuses the running :3333 server)
npx playwright test tests/vision/capture.spec.ts

2. Judge — `scripts/vision-judge-workflow.mjs` (run via the Workflow tool)

(Lives under scripts/ — Workflow scripts use top-level return/await + injected globals, so they're not standard ES modules and scripts/** is ESLint-ignored.) One independent vision-judge subagent per screenshot (each Reads the PNG — that is the vision step) grades the 5-dimension rubric; the reducer applies the objective floor and an agent writes artifacts/vision-report.md.

// assemble the manifest from the capture sidecars, then:
Workflow({
  scriptPath: "scripts/vision-judge-workflow.mjs",
  args: { captures: [ { surface, png, expectation, consoleErrors } /* … */ ] }
})

The manifest is the artifacts/*.json sidecars merged with their PNG paths (one {surface, png, expectation, consoleErrors} per capture). args may be a JSON object or string — the script accepts both.

Rubric (per surface)

renders_correctly · no_error_state · flow_completes · theme_legible (vision-graded) + no_console_errors (objective, from the capture driver). FAIL if any vision dimension fails at confidence ≥ 0.7 or a real console error is present; WARN at 0.4–0.7 (human spot-check, never auto-blocks).

Status

Capture layer: structurally verified (playwright --list → 42 tests).
Judge workflow + reducer + report: proven end-to-end against real Waggle screenshots (6 agents, accurate verdicts, report written).
First-run finding: the Visual-Regression — Dark Mode baselines under tests/visual/baselines/ are 404 error pages, not Waggle UI (verified) — the pixel-diff visual suite has been comparing against garbage. Re-baseline once the app serves correctly. The one genuine UI tested (settings-light) graded PASS at 0.95.
Pending: the full live capture→judge run (local sidecar is blocked by a tsx/esbuild version skew on this Windows box — runs in CI Linux / a clean env).

Caveats

tests/vision/ is not wired into any CI gate (it's outside the e2e/visual lanes), so it's inert until invoked explicitly.
artifacts/ is gitignored (regenerated per run).

3.8 KiB Raw Blame History Unescape Escape