This commit is contained in:
Oleg Maslov
@@ -0,0 +1,504 @@
|
||||
# E2E Persona Test Matrix — 3 Tier × 3 Proficiency × Persona Mapping
|
||||
|
||||
**Date**: 2026-04-25 (autored 2026-04-24 late evening)
|
||||
**Status**: Test scripts ready, čekaju executable app + test accounts
|
||||
**Authored by**: claude-opus-4-7 (PM Cowork)
|
||||
**Execution method**: PM (claude-opus-4-7) sa Claude in Chrome computer use, observational testing kao stvarni user; friction log generates JSON per scenario; Marko reviewa rezultate
|
||||
**Scope per Marko brief 2026-04-24**: "ne samo naplatne tiere nego i tri nivoa usera - starter, pro, professional - power user"
|
||||
|
||||
## §0 Architecture
|
||||
|
||||
### 3 monetization tiers (LOCKED per `project_locked_decisions`)
|
||||
- **FREE** ($0/forever) — 5 workspaces, 22 personas, 60+ tools, persistent .mind, harvest, encrypted vault, wiki compiler, self-evolution
|
||||
- **PRO** ($19/mo) — Free + unlimited workspaces, all embedding providers, skills marketplace, custom skills, compliance audit reports, priority support
|
||||
- **TEAMS** ($49/seat/mo) — Pro + shared team memory, WaggleDance coordination, team skill library, admin governance, S3 team storage, audit trail compliance
|
||||
|
||||
### 3 user proficiency levels (per Marko brief)
|
||||
- **STARTER** — first-time user, never used AI agent platform, expects guided experience, limited technical context, learns by doing
|
||||
- **PRO** — regular user, knows AI agent basics (used Claude Code / Cursor / GPT API), comfortable with concepts but new to Waggle paradigm
|
||||
- **PROFESSIONAL** (power user) — advanced workflows, multiple agents simultaneously, custom skills, MCP server integration, governance + audit needs
|
||||
|
||||
### 3 × 3 = 9 archetype matrix
|
||||
|
||||
| | STARTER | PRO | PROFESSIONAL |
|
||||
|---|---|---|---|
|
||||
| **FREE** | A1: Curious newbie | A2: Existing AI user testing alternatives | A3: Power user evaluating before buying |
|
||||
| **PRO** | A4: Onboarded paying user (first month) | A5: Settled paying user (3+ months) | A6: Solo professional sa heavy workflows |
|
||||
| **TEAMS** | A7: New team member onboarded by admin | A8: Active team contributor | A9: Team admin sa governance ownership |
|
||||
|
||||
### Persona overlay (13 bee personas iz DS spec)
|
||||
|
||||
Per archetype mapping, izaberem 1-2 reprezentativne persona za testing realism:
|
||||
- A1 (Free Starter): **bee-confused** (overwhelmed first-timer)
|
||||
- A2 (Free Pro): **bee-researcher** (academic on free tier)
|
||||
- A3 (Free Professional): **bee-architect** (systems thinker, evaluating)
|
||||
- A4 (Pro Starter): **bee-builder** (developer building first project)
|
||||
- A5 (Pro Pro): **bee-writer** (content creator, regular use)
|
||||
- A6 (Pro Professional): **bee-orchestrator** (multi-agent coordinator)
|
||||
- A7 (Teams Starter): **bee-marketer** (joined team, learning)
|
||||
- A8 (Teams Pro): **bee-analyst** (active team contributor)
|
||||
- A9 (Teams Professional): **bee-team** (team admin/lead)
|
||||
|
||||
Plus dva edge case persone:
|
||||
- **bee-hunter** (sales/BD, used cross-archetype za commercial scenarios)
|
||||
- **bee-celebrating** (success state — does notification flow work?)
|
||||
- **bee-sleeping** (idle state — what happens when user disengages?)
|
||||
|
||||
---
|
||||
|
||||
## §1 Pre-test setup checklist
|
||||
|
||||
Pre svakog scenario-a:
|
||||
|
||||
1. **Browser**: incognito Chromium (no cached state, no localStorage pollution)
|
||||
2. **Device emulation**: standard desktop 1440×900, ne mobile (Tauri app je desktop-first)
|
||||
3. **Network throttling**: none initially, simulate Fast 3G u stress scenarijima
|
||||
4. **Test account credentials** (Marko provides): per-tier test accounts seeded sa appropriate persona profile
|
||||
5. **Seed data**:
|
||||
- FREE accounts: empty workspace
|
||||
- PRO accounts: 3-5 sample memories, 2 agents idle
|
||||
- TEAMS accounts: shared workspace sa 5 members, 10 sample memories, audit log entries
|
||||
6. **Time-of-day**: skip A/B variants for now, all tests at same time-of-day to remove temporal variance
|
||||
7. **Snapshots**: pre-test screenshot, post-test screenshot, friction-event screenshots (captured by computer-use mid-flow)
|
||||
8. **Friction log**: open `friction-log-{archetype}-{persona}.json` template at start, populate during
|
||||
|
||||
### Friction log JSON schema
|
||||
|
||||
```json
|
||||
{
|
||||
"archetype": "A4-Pro-Starter",
|
||||
"persona": "bee-builder",
|
||||
"session_id": "uuid",
|
||||
"started_at": "2026-04-25T10:00:00Z",
|
||||
"ended_at": "2026-04-25T10:32:00Z",
|
||||
"duration_seconds": 1920,
|
||||
"scenario_completed": true,
|
||||
"events": [
|
||||
{
|
||||
"step": 1,
|
||||
"action": "click signup button",
|
||||
"expected": "redirect to signup form",
|
||||
"observed": "redirect to signup form",
|
||||
"friction_score": 0,
|
||||
"friction_note": null,
|
||||
"screenshot": "01-signup-clicked.png",
|
||||
"timestamp_seconds": 5
|
||||
},
|
||||
{
|
||||
"step": 2,
|
||||
"action": "complete email + password",
|
||||
"expected": "submit, redirect to onboarding wizard",
|
||||
"observed": "submit, but error 'password too weak' surprised user",
|
||||
"friction_score": 2,
|
||||
"friction_note": "User attempted 8-char password. App requires 12+ but error message didn't say that until after submit. Pre-validate during typing.",
|
||||
"screenshot": "02-password-error.png",
|
||||
"timestamp_seconds": 65
|
||||
}
|
||||
],
|
||||
"summary": {
|
||||
"completion_rate": "5/6 sub-tasks completed",
|
||||
"avg_friction": 1.4,
|
||||
"highest_friction_step": 2,
|
||||
"deal_breakers": [],
|
||||
"delight_moments": [
|
||||
"Onboarding step 3 (persona selection) — smooth, well-designed grid"
|
||||
],
|
||||
"improvement_suggestions": [
|
||||
"Pre-validate password during typing",
|
||||
"Add 'show password' toggle (currently hidden)"
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
friction_score scale: 0 (smooth) / 1 (slight pause, no impact) / 2 (noticeable hesitation) / 3 (re-try required) / 4 (user almost abandoned) / 5 (complete blocker, scenario fail)
|
||||
|
||||
---
|
||||
|
||||
## §2 Coverage area inventory
|
||||
|
||||
Per scenario, svi profili pokrivaju subset:
|
||||
|
||||
| Coverage area | Description |
|
||||
|---|---|
|
||||
| **CA-1**: Signup + onboarding (8-step) | Welcome → WhyWaggle → Persona → ApiKey → Template → ModelTier → Import → Tier → Ready |
|
||||
| **CA-2**: First memory creation | Manual entry sa scope + tag + content; verify save + appears u Memory app |
|
||||
| **CA-3**: Harvest from chat | Connect provider, harvest existing chat session into memory |
|
||||
| **CA-4**: Memory search | Search by name + filter by scope/tag + date range |
|
||||
| **CA-5**: Graph viewport | Open Graph app, navigate force-directed canvas, click node, see drawer |
|
||||
| **CA-6**: Agent spawn | Spawn agent via dock → app → "+ New Agent" or ⌘K, assign task, monitor status |
|
||||
| **CA-7**: Multi-window | Open Memory + Graph + Cockpit simultaneously, drag/resize, z-order interactions |
|
||||
| **CA-8**: ⌘K palette | Trigger palette in different contexts (desktop, Memory, Graph, Agents), execute commands |
|
||||
| **CA-9**: Provenance audit | Open Provenance app, filter by event type, replay event state, export CSV |
|
||||
| **CA-10**: Light/dark toggle | Settings → Appearance → toggle Auto/Light/Dark, verify smooth transition + token swap |
|
||||
| **CA-11**: Tier upgrade flow | Click upgrade CTA, complete Stripe checkout (test card), verify tier change + new features unlocked |
|
||||
| **CA-12**: Tier downgrade flow | Cancel subscription, verify graceful degradation (data preserved, features locked) |
|
||||
| **CA-13**: Settings configuration | Preferences, keyboard shortcuts, providers, policy |
|
||||
| **CA-14**: Notifications | Trigger toast (success + error + policy), open NotificationInbox, mark read, filter |
|
||||
| **CA-15**: ⌘? shortcuts modal | Open keyboard shortcuts registry, search filter, navigate, learn |
|
||||
| **CA-16**: Error handling | Network drop, invalid API key, parse error mid-flow — graceful recovery |
|
||||
| **CA-17**: Workspace management | Create new workspace, switch workspaces, archive |
|
||||
| **CA-18**: Skills marketplace (Pro+) | Browse marketplace, install skill, configure |
|
||||
| **CA-19**: Team coordination (Teams) | Invite member, share workspace, audit member actions |
|
||||
| **CA-20**: Custom skill creation (Pro+) | Build custom skill, test, publish to team library |
|
||||
|
||||
---
|
||||
|
||||
## §3 Per-archetype test scripts
|
||||
|
||||
### A1 — FREE Starter (bee-confused)
|
||||
|
||||
**Profile**: First-time AI agent platform user. Has heard about Waggle from a friend. Privacy-conscious. Tech savvy enough to install desktop apps but new to AI agent paradigms. Goal: try it free, see if it makes sense.
|
||||
|
||||
**Pre-test state**: Fresh download, no account, no .mind files.
|
||||
|
||||
**Scenario duration target**: 30-45 min
|
||||
|
||||
**Coverage**: CA-1, CA-2, CA-7, CA-8, CA-15, CA-16
|
||||
|
||||
**Step-by-step script**:
|
||||
|
||||
1. **Land on waggle-os.ai** — observable: hero banner, "Free for individuals" subtext, Download CTAs visible
|
||||
- Friction probe: does CTA copy resonate? Is "no credit card" trust signal clear?
|
||||
2. **Click "Download for Windows"** — observable: redirect to GitHub releases latest
|
||||
- Friction probe: does GitHub UI feel scary to non-developer? Does .exe vs .msi vs portable confuse?
|
||||
3. **Install + launch** — observable: BootScreen, OnboardingWizard appears
|
||||
4. **Onboarding step 1 (Welcome)** — read welcome copy, click "Begin →"
|
||||
- Friction probe: does "AI agents that remember" make sense without technical context?
|
||||
5. **Step 2 (WhyWaggle)** — read why-now narrative
|
||||
- Friction probe: too long? boring? appropriate depth?
|
||||
6. **Step 3 (Persona)** — select "bee-confused" persona ("New to AI? Start here.")
|
||||
- Friction probe: does 13-grid overwhelm? Are persona descriptions clear?
|
||||
7. **Step 4 (ApiKey)** — IMPORTANT — STARTER may not have any API key
|
||||
- Friction probe: does "Skip — use local Ollama" option exist? If not, ABANDON RISK
|
||||
- Expected: graceful path for users without API keys (local model fallback)
|
||||
8. **Step 5 (Template)** — select "personal notes" template
|
||||
9. **Step 6 (ModelTier)** — select default (local Ollama / Llama 3 if no API key)
|
||||
10. **Step 7 (Import)** — skip (no existing data)
|
||||
11. **Step 8 (Tier)** — select Free tier
|
||||
12. **Step 9 (Ready)** — click "Start using Waggle"
|
||||
13. **Land on desktop** — observable: BootScreen complete, desktop with dock visible, Cockpit auto-opened
|
||||
- Friction probe: does empty desktop feel inviting or intimidating?
|
||||
14. **First memory creation (CA-2)** — open Memory app from dock, click "+ New memory"
|
||||
- Type: name = "My first thought", scope = "personal", content = "Testing Waggle to see how this works."
|
||||
- Save, verify appears u list
|
||||
15. **Multi-window test (CA-7)** — open Graph app, observe empty graph (no nodes yet)
|
||||
- Friction probe: does empty state explain "Add memories to populate graph"?
|
||||
16. **⌘K palette (CA-8)** — press Cmd-K (or Ctrl-K on Win), see palette
|
||||
- Try "search memories" → find created memory
|
||||
- Friction probe: does ⌘K feel discoverable? Hint visible somewhere u UI?
|
||||
17. **⌘? shortcuts (CA-15)** — press Cmd-? to see shortcuts registry
|
||||
- Friction probe: does power-user feature gate intimidate Starter?
|
||||
18. **Error simulation (CA-16)** — disconnect network, try search
|
||||
- Expected: graceful "you're offline, search using local cache" message
|
||||
19. **End test** — close all windows, observe state preservation
|
||||
|
||||
**Success criteria**:
|
||||
- Onboarding completed without abandoning (8/8 steps)
|
||||
- First memory created and searchable
|
||||
- User did not require external help (no Discord/email)
|
||||
- Net friction score average ≤ 2.0
|
||||
- 0 deal-breakers (friction_score ≥ 5)
|
||||
|
||||
---
|
||||
|
||||
### A2 — FREE Pro (bee-researcher)
|
||||
|
||||
**Profile**: PhD candidate, uses Claude/GPT daily, knows about RAG, vector DBs. Has API keys for multiple providers. Wants to evaluate Waggle as alternative to NotebookLM / Mem0. Will switch if it's better.
|
||||
|
||||
**Coverage**: CA-1 (faster), CA-3, CA-4, CA-5, CA-9, CA-13
|
||||
|
||||
**Step-by-step script**:
|
||||
|
||||
1. Skip — same Hero + download as A1, but completes onboarding ~10x faster
|
||||
2. **Onboarding (CA-1, abbreviated)** — provides Anthropic + OpenAI + Together API keys, selects "researcher" persona, "academic-research" template
|
||||
3. **Harvest existing chat (CA-3)** — connect Anthropic API key, harvest last 50 conversations
|
||||
- Friction probe: does harvest UI explain what gets imported? Privacy implication clear?
|
||||
4. **Memory search (CA-4)** — after harvest, search "elasticity" or domain-specific term
|
||||
- Friction probe: does search return semantic matches or only keyword? Is ranking sensible?
|
||||
5. **Graph viewport (CA-5)** — open Graph app, see imported memories as nodes
|
||||
- Click a node, see drawer sa Properties / Neighbors / Bitemporal
|
||||
- Friction probe: bitemporal interface — does Researcher persona understand "VALID vs RECORDED"? Tooltip / explainer needed?
|
||||
6. **Provenance audit (CA-9)** — open Provenance app, see harvest events
|
||||
- Filter by source = "anthropic", inspect single event
|
||||
- Friction probe: does provenance UI feel valuable to academic (citation use case) or overkill?
|
||||
7. **Settings (CA-13)** — Preferences, configure default model = Claude Sonnet 4
|
||||
- Friction probe: does provider routing UI confuse? Is "cost meter" visible?
|
||||
|
||||
**Success criteria**:
|
||||
- Harvest succeeds (50/50 chats imported, memories created)
|
||||
- Search returns relevant results (not just keyword match)
|
||||
- Graph visualization meaningful (clusters, edges represent something)
|
||||
- User remains on Free tier after test (no upsell pressure resented)
|
||||
- User comments "I'd recommend this to colleagues" (qualitative)
|
||||
|
||||
---
|
||||
|
||||
### A3 — FREE Professional / Power User (bee-architect)
|
||||
|
||||
**Profile**: Senior systems architect, runs local LLMs, builds MCP servers, evaluates tooling for adoption. Goal: stress-test Waggle's architecture, see if it's production-grade.
|
||||
|
||||
**Coverage**: CA-1 (skipped — direct config), CA-3, CA-5, CA-7, CA-9, CA-13, CA-14
|
||||
|
||||
**Step-by-step script**:
|
||||
|
||||
1. Skip onboarding via "advanced setup" path (if exists)
|
||||
2. **MCP server inspection** — verify Waggle's MCP server endpoint exposed locally (default port?), test from Claude Code
|
||||
- Friction probe: is MCP endpoint discoverable without docs?
|
||||
3. **Custom provider** — add custom OpenAI-compatible endpoint (e.g., local vLLM)
|
||||
- Friction probe: provider configuration sufficiently flexible?
|
||||
4. **Heavy harvest** — import 5,000+ memory items via batch script (.mind file format)
|
||||
- Friction probe: large import progress indicator, error recovery?
|
||||
5. **Graph stress** — open Graph app sa 5,000 nodes, pan/zoom performance
|
||||
- Friction probe: rendering FPS, search latency u large graph?
|
||||
6. **Multi-window stress (CA-7)** — open all 23 apps simultaneously
|
||||
- Friction probe: does compositor handle? Memory leak?
|
||||
7. **Audit provenance (CA-9)** — query 100k events, export CSV
|
||||
- Friction probe: query latency, CSV size limit, EU AI Act audit triggers visible?
|
||||
8. **Notification flood (CA-14)** — trigger 50 simultaneous notifications
|
||||
- Friction probe: NotificationInbox aggregation? Toast queue management?
|
||||
9. **Resource monitoring** — check Cockpit u stress conditions: memory usage, CPU, network
|
||||
- Friction probe: visible OOM risk warnings? Cost meter accurate?
|
||||
|
||||
**Success criteria**:
|
||||
- MCP server discovery + connection working
|
||||
- 5,000-node graph remains usable (≥30 FPS pan)
|
||||
- Notification system doesn't break under flood
|
||||
- No data loss on stress operations
|
||||
- Cockpit metrics accurate
|
||||
- User adoption decision: "I'll try this in my team" (qualitative)
|
||||
|
||||
---
|
||||
|
||||
### A4 — PRO Starter (bee-builder)
|
||||
|
||||
**Profile**: Junior developer, hired into team using Waggle, paid Pro tier issued. First week, learning the tool. Goal: become productive without feeling overwhelmed.
|
||||
|
||||
**Coverage**: CA-1 (with API keys provided by team), CA-2, CA-4, CA-6, CA-8, CA-11
|
||||
|
||||
**Step-by-step script**:
|
||||
|
||||
1. **Login sa pre-existing Pro account** — observable: tier badge "PRO" visible u Settings or Cockpit
|
||||
2. **Quick onboarding (CA-1)** — accept defaults set by admin (template, model tier)
|
||||
3. **First memory (CA-2)** — same as A1 but slightly more complex (project context)
|
||||
4. **Memory search (CA-4)** — search project terms
|
||||
5. **Spawn agent (CA-6)** — open Agents app, click "+ Spawn agent" → "Researcher" template
|
||||
- Assign task: "Read README and summarize project structure"
|
||||
- Monitor status: idle → running → done
|
||||
- Friction probe: spawn UX — does Starter understand parameter knobs?
|
||||
6. **⌘K palette context (CA-8)** — try ⌘K when Memory focused vs Agents focused
|
||||
- Friction probe: does context-switching feel natural?
|
||||
7. **Tier upgrade hint** — observe upsell hints (skills marketplace teaser, custom skill creation gate)
|
||||
- Friction probe: are upsell prompts honest or pushy?
|
||||
|
||||
**Success criteria**:
|
||||
- First agent task completes successfully
|
||||
- Pro features (skills marketplace) discoverable but not pushy
|
||||
- User self-rates productivity gain "above average" or higher
|
||||
|
||||
---
|
||||
|
||||
### A5 — PRO Pro (bee-writer)
|
||||
|
||||
**Profile**: Content creator, 3+ months on Pro tier, daily user. Has personal workflows established. Goal: efficient task execution, minor optimization.
|
||||
|
||||
**Coverage**: CA-2, CA-4, CA-6, CA-7, CA-9, CA-18 (skills marketplace), CA-13
|
||||
|
||||
**Step-by-step script**:
|
||||
|
||||
1. **Existing workspace** — open with established memories, agents
|
||||
2. **Daily workflow** — search existing memory, edit, save
|
||||
3. **Spawn agent for routine task** — "Draft tomorrow's newsletter from this week's notes"
|
||||
4. **Multi-window** — Memory + Chat + Agents simultaneously
|
||||
5. **Skills marketplace (CA-18)** — browse, install "newsletter-formatter" skill
|
||||
- Friction probe: install UX, configuration prompts, immediate availability?
|
||||
6. **Provenance check (CA-9)** — verify last week's auto-generated newsletter has proper citations
|
||||
7. **Settings tweaks (CA-13)** — change keyboard shortcut for "spawn newsletter agent"
|
||||
|
||||
**Success criteria**:
|
||||
- All routine tasks complete < 50% time vs without Waggle (subjective comparison)
|
||||
- Skill install + first use < 2 min
|
||||
- Custom shortcut configuration works first try
|
||||
|
||||
---
|
||||
|
||||
### A6 — PRO Professional / Power User (bee-orchestrator)
|
||||
|
||||
**Profile**: Solo professional sa heavy parallel workflows. Runs 5+ agents simultaneously. Custom skills built. Goal: scale operations without context switching cost.
|
||||
|
||||
**Coverage**: CA-6 (parallel), CA-7 (heavy multi-window), CA-19 N/A solo, CA-20 (custom skills), CA-13
|
||||
|
||||
**Step-by-step script**:
|
||||
|
||||
1. **Parallel agent orchestration (CA-6)** — spawn 5 agents simultaneously, different tasks
|
||||
- Friction probe: dock indicator clarity, status overlap, message routing?
|
||||
2. **Custom skill creation (CA-20)** — build "competitor-tracker" skill (scrape + summarize + memory store)
|
||||
- Friction probe: skill DSL learning curve? Test environment? Publish workflow?
|
||||
3. **Multi-window heavy (CA-7)** — Cockpit + Memory + Graph + Agents + Chat + Provenance + Files all open
|
||||
- Friction probe: window management cognitive load? Snap-zone effectiveness?
|
||||
4. **Workspace switching (CA-17)** — 5 workspaces, switch quickly
|
||||
5. **Audit (CA-9)** — review week's agent activity, identify cost optimization
|
||||
|
||||
**Success criteria**:
|
||||
- 5 parallel agents complete tasks without collision
|
||||
- Custom skill published + works
|
||||
- Multi-window paradigm scales (no FPS drop, no z-order confusion)
|
||||
- Cost meter informs efficient model routing decisions
|
||||
|
||||
---
|
||||
|
||||
### A7 — TEAMS Starter (bee-marketer)
|
||||
|
||||
**Profile**: New team member added by admin, first day. Doesn't know Waggle. Has shared workspace access via team license.
|
||||
|
||||
**Coverage**: CA-1 (team-onboarded), CA-2, CA-4, CA-19 (member side)
|
||||
|
||||
**Step-by-step script**:
|
||||
|
||||
1. **Email invite link** — click, land on team workspace
|
||||
2. **Auto-onboarding (CA-1, team variant)** — provider keys inherited from team, persona selection only
|
||||
3. **Shared workspace tour (CA-19 member side)** — see existing team memories, agents (read-only initially)
|
||||
4. **Add first memory (CA-2)** — contribute personal note to shared scope
|
||||
- Friction probe: does sharing model (private vs team) feel clear?
|
||||
5. **Search team memory (CA-4)** — find colleague's memory, see attribution
|
||||
|
||||
**Success criteria**:
|
||||
- Team workspace access immediate (no admin waiting)
|
||||
- Shared vs private boundary clear
|
||||
- New member feels productive within first hour
|
||||
|
||||
---
|
||||
|
||||
### A8 — TEAMS Pro (bee-analyst)
|
||||
|
||||
**Profile**: Active team contributor, daily Waggle user, 6+ months. Has personal scope + contributes to team scope.
|
||||
|
||||
**Coverage**: CA-4 (cross-scope), CA-6, CA-9 (team audit), CA-19, CA-14
|
||||
|
||||
**Step-by-step script**:
|
||||
|
||||
1. **Cross-scope search (CA-4)** — search across personal + team scopes simultaneously
|
||||
2. **Spawn agent on team data (CA-6)** — agent reads team memory, produces analysis
|
||||
3. **Team audit (CA-9 + CA-19)** — see all team agent runs this week, costs, impact
|
||||
4. **Notifications (CA-14)** — receive notification when colleague's agent enriches shared memory
|
||||
|
||||
**Success criteria**:
|
||||
- Cross-scope queries fast + intuitive
|
||||
- Team audit visibility appropriate (not invasive but transparent)
|
||||
- Notification routing makes sense
|
||||
|
||||
---
|
||||
|
||||
### A9 — TEAMS Professional / Admin (bee-team)
|
||||
|
||||
**Profile**: Team admin, owns governance + billing. Manages 10-50 seats. Compliance-conscious (GDPR, EU AI Act).
|
||||
|
||||
**Coverage**: CA-9 (full audit), CA-12 (downgrade scenario), CA-13 (admin governance), CA-19 (admin side), CA-11 (seat add/remove)
|
||||
|
||||
**Step-by-step script**:
|
||||
|
||||
1. **Admin governance panel (CA-13)** — open Policy app, define team policies
|
||||
- "All agents must use models with EU data residency"
|
||||
- "Audit triggers fire on every external memory share"
|
||||
- Friction probe: policy DSL learning curve? Built-in templates?
|
||||
2. **Add/remove seats (CA-11)** — invite 3 new members, then remove 1 (graceful downgrade CA-12)
|
||||
3. **Audit trail review (CA-9)** — last 30 days, all team activity, export for compliance officer
|
||||
- Friction probe: GDPR data subject access request workflow?
|
||||
4. **Billing review** — see usage breakdown per member, per project, per provider
|
||||
5. **Compliance trigger drill** — simulate EU AI Act Article 13 audit request, verify reproducibility
|
||||
|
||||
**Success criteria**:
|
||||
- Policy enforcement working (test by attempting violation)
|
||||
- Audit trail meets compliance officer review (subjective)
|
||||
- Billing breakdown accurate vs Stripe receipts
|
||||
- Seat management smooth
|
||||
|
||||
---
|
||||
|
||||
## §4 Cross-cutting test scenarios
|
||||
|
||||
Beyond per-archetype, run these cross-cutting flows once:
|
||||
|
||||
### CC-1 — Full upgrade journey (Free → Pro → Teams)
|
||||
Single user account. Start Free, hit Pro feature gate, upgrade. Use Pro for a week. Hit Teams feature gate (collaboration), upgrade. Verify data persistence + feature unlock at each step.
|
||||
|
||||
### CC-2 — Light/dark mode toggle (CA-10) across all apps
|
||||
Fresh user, default Auto mode, toggle Dark, toggle Light, observe transition. Open every app sa toggle in different states. Verify no theming regression.
|
||||
|
||||
### CC-3 — Network resilience (CA-16)
|
||||
Mid-session, simulate: brief offline (5s), prolonged offline (5min), provider API down (Anthropic 503), invalid API key. Verify graceful UI states + automatic recovery.
|
||||
|
||||
### CC-4 — Multi-window paradigm stress (CA-7)
|
||||
Open all 23 apps, drag/resize/snap, observe focus state, z-order, animation FPS. Ensure no compositor stutter.
|
||||
|
||||
### CC-5 — Onboarding abandonment recovery
|
||||
Start onboarding, exit at step 3. Re-launch app. Verify resume from step 3, not restart.
|
||||
|
||||
### CC-6 — Tier downgrade graceful (CA-12)
|
||||
Pro → Free downgrade. Verify: Pro features locked, Free features remain, data preserved, no surprise data loss.
|
||||
|
||||
### CC-7 — Keyboard shortcut discoverability (CA-15)
|
||||
First-time user attempts to find keyboard shortcuts. ⌘? must be discoverable somehow (menubar Help item, footer hint, etc.).
|
||||
|
||||
### CC-8 — Provenance replay (CA-9)
|
||||
Trigger an event (memory edit), wait 1 hour, replay event state. Verify exact reproduction.
|
||||
|
||||
### CC-9 — Cost meter accuracy
|
||||
Run a complex multi-agent task, compare Cockpit cost reading to Stripe billing event. Should match.
|
||||
|
||||
### CC-10 — Persona switch mid-flow
|
||||
Switch persona from "Researcher" to "Engineer" via Settings. Verify dock layout, default agents, preferences update appropriately.
|
||||
|
||||
---
|
||||
|
||||
## §5 Execution sequencing
|
||||
|
||||
**Day 1** (post-build): A1, A4, A7 — Starter tier across 3 monetization
|
||||
**Day 2**: A2, A5, A8 — Pro proficiency across 3 monetization
|
||||
**Day 3**: A3, A6, A9 — Professional / Power user across 3 monetization
|
||||
**Day 4**: CC-1 through CC-5 — first 5 cross-cutting
|
||||
**Day 5**: CC-6 through CC-10 — second 5 cross-cutting + remediation pass
|
||||
|
||||
Total: ~25 scenarios × 30-60 min average = ~15-25h E2E testing wall-clock + report generation.
|
||||
|
||||
PM (claude-opus-4-7) executes via Claude in Chrome computer-use, generates friction-log JSON per scenario, aggregates into single test report sa:
|
||||
- Per-archetype completion rates
|
||||
- Average friction score
|
||||
- Top 10 deal-breakers (P0)
|
||||
- Top 20 high-friction items (P1)
|
||||
- Top 30 medium-friction items (P2)
|
||||
- Delight moments (positive feedback for marketing)
|
||||
- Improvement recommendations sorted by RICE score (Reach × Impact × Confidence / Effort)
|
||||
|
||||
---
|
||||
|
||||
## §6 Pre-execution prerequisites — Marko side
|
||||
|
||||
Before PM can start:
|
||||
|
||||
1. **App accessible**: dev server running locally OR staging deployed URL OR Tauri build distributed
|
||||
- Decide deployment target — recommend staging URL on Vercel preview deploy for ease of access
|
||||
2. **Test accounts**: 9 accounts seeded sa appropriate persona + tier + data
|
||||
- Account creation script u repo? Seed data scripts?
|
||||
3. **Test card credentials**: Stripe test card 4242 4242 4242 4242 (or environment-specific)
|
||||
4. **Webhook stubs**: Provenance audit replay needs working backend; ensure replay endpoint live
|
||||
5. **Reset-between-tests procedure**: how to clean state between archetypes (separate accounts? wipe localStorage? incognito each session?)
|
||||
|
||||
If any of these aren't ready, PM will identify u test results and flag back.
|
||||
|
||||
---
|
||||
|
||||
## §7 Output deliverables (post-execution)
|
||||
|
||||
PM produces:
|
||||
- `briefs/e2e-persona-tests/results/2026-04-XX-friction-log-A1-confused.json` (per scenario)
|
||||
- `briefs/e2e-persona-tests/results/2026-04-XX-friction-log-aggregate.md` — synthesis
|
||||
- `briefs/e2e-persona-tests/results/2026-04-XX-improvement-roadmap.md` — RICE-prioritized fixes for CC-1 implementation sprint
|
||||
|
||||
---
|
||||
|
||||
## §8 Authorized by
|
||||
|
||||
PM Marko Marković, 2026-04-24 evening, scope expansion ratified ("ne samo naplatne tiere nego i tri nivoa usera - starter, pro, professional - power user, sve treba da spremiš i smisliš na osnovu repoa").
|
||||
|
||||
PM (claude-opus-4-7) authored matrix overnight 2026-04-24/25, čeka Marka ujutru za review of prerequisite checklist (§6) and prerequisites readiness ratification before E2E execution begins.
|
||||
Reference in New Issue
Block a user