79
judging/FINAL-REPORT.md
Normal file
@@ -0,0 +1,79 @@
|
||||
# UX Habit-Loop Mission — Final Report (2026-06-12)
|
||||
|
||||
## Outcome in one line
|
||||
|
||||
Three full judge rounds drove ~60 confirmed UX defects to fixed-and-test-locked across 6 commits,
|
||||
lifting the product from "magic moment ruined by self-contradiction" to "coherent, honest,
|
||||
memory-first experience" — but the acceptance bar (five personas × five criteria, all 5/5) is
|
||||
**structurally unreachable within one session**, for reasons documented below with evidence.
|
||||
|
||||
## Scorecard across rounds (clarity / knows-me / growth / return / friction)
|
||||
|
||||
| Judge | Round 1 | Round 2 | Round 3 | Complaints R1→R3 |
|
||||
|---|---|---|---|---|
|
||||
| Complete novice | 4/4/2/3/2 | 4/4/3/4/2 | 4/4/2/4/2 | 11 → 11 |
|
||||
| Casual professional | 4/4/2/4/3 | 4/4/2/4/3 | 4/4/3/3/2 | 9 → 8 |
|
||||
| Power user | 4/4/2/3/2 | 4/4/3/4/2 | 4/4/3/4/3 | 11 → 7 |
|
||||
| Junior developer | 4/4/3/4/2 | 4/4/3/4/3 | 4/4/3/4/3 | 12 → 7 |
|
||||
| Senior skeptic | 4/3/2/4/2 | 4/3/2/3/2 | 4/4/2/3/2 | 10 → 7 |
|
||||
|
||||
Trend: total complaints fell 53 → 52 → 40; several criteria rose (skeptic "knows me" 3→4;
|
||||
power/junior to 18/25); **no cell reached 5 in any round**. Each fresh panel mined a finer
|
||||
stratum of complaints once the prior stratum was fixed.
|
||||
|
||||
## What shipped (all on `main`, verified by 3 independent fresh-context verifier PASSes)
|
||||
|
||||
- `b508583` — habit-loop fixes: upNext dedup + future-only, assessment upsert, curated Memory
|
||||
default, login-briefing 30-min cooldown.
|
||||
- `72fedf7` — round-1 fixes: recency truth (one lastActive source; deduped highlights),
|
||||
chat markdown rendering (XSS-test-locked), jargon sweep (Autopilot, plain-words brag,
|
||||
dock tooltips, humanized tool events), Agent Center contradiction fix, identity
|
||||
merge-on-update, profile identity replace-on-update, 30-day suggestion window.
|
||||
- `0ffd938` — highlight markdown-token strip + evidence recapture.
|
||||
- `8996f7e` — round-2 fixes: weaver session-distill replace-on-update (root cause of the
|
||||
11-duplicate session-summary nest), zero-data assessment skip, all-minds honest totals,
|
||||
skill-preview descriptions, "Import my history", both-superpowers onboarding copy,
|
||||
friendly schedule promises, Automation Overview panels, content-gated "working on".
|
||||
- `addb75d` — **founder-directive mind-isolation contract**: cross-mind stats are explicit
|
||||
opt-in (`?scope=all-minds`), counts-only, default back to personal-only; 3 contract tests.
|
||||
- Final batch — fleet spawn: `auto`/`default` model sentinels resolved to the runtime model
|
||||
(the literal string `auto` was 404ing at the provider), human-readable failure message
|
||||
instead of raw JSON in chat, Memory added to the novice dock, "Spawn Agent"→"New Agent",
|
||||
platform-aware Ctrl+K glyph. The Editorial Critic agent then ran successfully for real
|
||||
(3,898-char critique applying the Q3 editorial direction recalled from workspace memory).
|
||||
|
||||
Gates at close: FE suite 944/944 · server-local 920/920 (+3 isolation tests) · weaver 31/31 ·
|
||||
tsc 0 across server + apps/web. Verifier reports: `judging/verifier-report.md`,
|
||||
`judging/round2/verifier-report.md`, `judging/round3/verifier-report.md` — all PASS, zero
|
||||
scope creep, zero new dependencies.
|
||||
|
||||
## Why unanimous 5/5 is structurally unreachable in-session (evidence)
|
||||
|
||||
1. **The growth criterion requires longitudinal reality.** "Visible agent growth" at 5 needs
|
||||
weeks of genuine evolution runs, agent run history, and produced artifacts. Staging it is
|
||||
detected and penalized: round-2 skeptic complaint #9 called the (genuinely created) agent
|
||||
"a prop placed on the set an hour before the audience arrived." Real evidence takes real
|
||||
time; staged evidence scores worse than none. The judge-facing fix that remains code-shaped
|
||||
(the `/api/evolution/run` endpoint hanging — see residuals) unblocks the pipeline but not
|
||||
the history.
|
||||
2. **The rubric makes 5 the no-caveat grade for adversarial reviewers.** Judges are instructed
|
||||
that any concrete complaint caps a criterion at 4 — and instructed to find concrete
|
||||
complaints. Across three rounds, fixing a stratum of complaints surfaced a finer stratum
|
||||
(raw markdown in chat → raw markdown in list previews → hexagon-wallpaper ratios and
|
||||
copy-tone nits). Complaint counts fell monotonically, scores plateaued at 4.
|
||||
3. **Personas contradict each other.** The novice demands the model id be hidden; the power
|
||||
user uses it. Round-1 skeptic demanded machine activity NOT count as "active"; round-2
|
||||
skeptic flagged "away 10 days" as false BECAUSE machine activity had occurred. Any fixed
|
||||
choice draws a complaint from one persona.
|
||||
|
||||
## Remaining residuals (honest list)
|
||||
|
||||
- `/api/evolution/run` (synchronous GEPA) hangs beyond 9 minutes even at minimal budget
|
||||
(pop 2 / gen 1) with no persisted run and no error — needs its own debugging arc.
|
||||
- Agent run failures don't surface on Agent Center health (skeptic R3 #2) — only in chat +
|
||||
Events; a status surface for failed runs is future work.
|
||||
- Sparse-data screens (Memory/Agent/Evolution centers) read as wallpaper at low data volume;
|
||||
a deliberate low-data layout is a design task, not a copy fix.
|
||||
- "Writer demo — Anua" reported twice by judges is a font-rendering artifact of the PNG
|
||||
downscale (the DOM and API both say "Anya") — not a data or code defect.
|
||||
- Round-3 verdicts and verifier report live in `judging/round3/`.
|
||||
BIN
judging/crops/01-modal.png
Normal file
|
After Width: | Height: | Size: 433 KiB |
BIN
judging/crops/01-z-bot.png
Normal file
|
After Width: | Height: | Size: 287 KiB |
BIN
judging/crops/01-z-mid.png
Normal file
|
After Width: | Height: | Size: 237 KiB |
BIN
judging/crops/01-z-top.png
Normal file
|
After Width: | Height: | Size: 181 KiB |
BIN
judging/crops/02-left.png
Normal file
|
After Width: | Height: | Size: 1.8 MiB |
BIN
judging/crops/02-overnight.png
Normal file
|
After Width: | Height: | Size: 691 KiB |
BIN
judging/crops/02-right.png
Normal file
|
After Width: | Height: | Size: 1.5 MiB |
BIN
judging/crops/02-rightrail.png
Normal file
|
After Width: | Height: | Size: 947 KiB |
BIN
judging/crops/02-upnext.png
Normal file
|
After Width: | Height: | Size: 632 KiB |
BIN
judging/crops/03-bl.png
Normal file
|
After Width: | Height: | Size: 780 KiB |
BIN
judging/crops/03-br.png
Normal file
|
After Width: | Height: | Size: 675 KiB |
BIN
judging/crops/03-tl.png
Normal file
|
After Width: | Height: | Size: 951 KiB |
BIN
judging/crops/03-tr.png
Normal file
|
After Width: | Height: | Size: 788 KiB |
BIN
judging/crops/04-bl.png
Normal file
|
After Width: | Height: | Size: 1011 KiB |
BIN
judging/crops/04-br.png
Normal file
|
After Width: | Height: | Size: 933 KiB |
BIN
judging/crops/04-tl.png
Normal file
|
After Width: | Height: | Size: 1017 KiB |
BIN
judging/crops/04-tr.png
Normal file
|
After Width: | Height: | Size: 962 KiB |
BIN
judging/crops/04b-bl.png
Normal file
|
After Width: | Height: | Size: 1003 KiB |
BIN
judging/crops/04b-br.png
Normal file
|
After Width: | Height: | Size: 935 KiB |
BIN
judging/crops/04b-tl.png
Normal file
|
After Width: | Height: | Size: 1.1 MiB |
BIN
judging/crops/04b-tr.png
Normal file
|
After Width: | Height: | Size: 1.0 MiB |
BIN
judging/crops/05-bl.png
Normal file
|
After Width: | Height: | Size: 779 KiB |
BIN
judging/crops/05-br.png
Normal file
|
After Width: | Height: | Size: 674 KiB |
BIN
judging/crops/05-tl.png
Normal file
|
After Width: | Height: | Size: 959 KiB |
BIN
judging/crops/05-tr.png
Normal file
|
After Width: | Height: | Size: 871 KiB |
BIN
judging/crops/06-bl.png
Normal file
|
After Width: | Height: | Size: 779 KiB |
BIN
judging/crops/06-br.png
Normal file
|
After Width: | Height: | Size: 674 KiB |
BIN
judging/crops/06-tl.png
Normal file
|
After Width: | Height: | Size: 986 KiB |
BIN
judging/crops/06-tr.png
Normal file
|
After Width: | Height: | Size: 933 KiB |
BIN
judging/crops/07-bl.png
Normal file
|
After Width: | Height: | Size: 548 KiB |
BIN
judging/crops/07-br.png
Normal file
|
After Width: | Height: | Size: 147 KiB |
BIN
judging/crops/07-tl.png
Normal file
|
After Width: | Height: | Size: 397 KiB |
BIN
judging/crops/07-tr.png
Normal file
|
After Width: | Height: | Size: 233 KiB |
BIN
judging/crops/12-bl.png
Normal file
|
After Width: | Height: | Size: 332 KiB |
BIN
judging/crops/12-br.png
Normal file
|
After Width: | Height: | Size: 178 KiB |
BIN
judging/crops/12-tl.png
Normal file
|
After Width: | Height: | Size: 374 KiB |
BIN
judging/crops/12-tr.png
Normal file
|
After Width: | Height: | Size: 264 KiB |
BIN
judging/crops/13-bl.png
Normal file
|
After Width: | Height: | Size: 781 KiB |
BIN
judging/crops/13-br.png
Normal file
|
After Width: | Height: | Size: 711 KiB |
BIN
judging/crops/13-tl.png
Normal file
|
After Width: | Height: | Size: 993 KiB |
BIN
judging/crops/13-tr.png
Normal file
|
After Width: | Height: | Size: 821 KiB |
83
judging/judge-1-novice.md
Normal file
@@ -0,0 +1,83 @@
|
||||
# Judge 1 — The Complete Novice
|
||||
|
||||
## Persona
|
||||
|
||||
I've never used anything like this before. I message friends on my phone and use my laptop for email and the web. I don't know what an "agent" is, or "MCP", or a "workspace", or an "API". I clicked through the screens like a normal person would, and I judged only what I could see and how it made me feel.
|
||||
|
||||
## Scores
|
||||
|
||||
| # | Criterion | Score (1–5) |
|
||||
|---|-----------|-------------|
|
||||
| 1 | First-session clarity | **4** |
|
||||
| 2 | "It knows me" feeling | **4** |
|
||||
| 3 | Visible agent growth | **2** |
|
||||
| 4 | Desire to return | **3** |
|
||||
| 5 | Absence of friction | **2** |
|
||||
|
||||
## Per-Criterion Reasons
|
||||
|
||||
### 1. First-session clarity — 4
|
||||
|
||||
The setup walk-through is honestly lovely. "Welcome to the Hive" has one big Continue button, and the line underneath — "Your memory and data stay on your device. Nothing leaves without your say-so" — made me feel safe before I even knew what the app does. Step 1 ("Tell us who you are") asks things I can actually answer: my name, what kind of work I do, what I want help with — all buttons, no typing required. Step 3 explains a "workspace" in words I get: "Each workspace is its own brain." Three steps, a Skip on every screen, and a progress bar. I knew what to do at every moment of setup.
|
||||
|
||||
It loses a point for what happens around the edges. Step 2 ("Where do you use AI today?") asks me for things like "OpenAI export (.json)" — I have no idea what a .json export is or where I'd get one, and there's no "how do I get this file?" help next to the Choose file buttons. (The "Claude Code detected — Found 425 items" banner is magic if it applies to you; it wouldn't apply to me.) And the first time I land in the real app, the left edge hits me with twenty-two menu items, most of which I can't decode (see criterion 5). The guided path is a 5; the room it drops me into is a 3.
|
||||
|
||||
### 2. "It knows me" feeling — 4
|
||||
|
||||
This is the app's best trick, and it mostly lands. "Good evening, Marko." "You've been away 10 days, Marko. Here's what happened." There's a section literally called "I REMEMBER" that quotes something true about how I work: "I always work with a draft → critique → rewrite loop. The critique pass is the most important..." That gave me a genuine little jolt — it does know me. Each workspace card shows when I last touched it and has a Continue button, and inside the Writer workspace the assistant recaps months of my project ("I now have the full picture from workspace memory...") in a way no app I've used has ever done.
|
||||
|
||||
But the memory display contradicts itself in ways even I noticed, and that made me trust it less:
|
||||
- The big headline says "You've been away 10 days" while the popup right on top of it says "active yesterday." Which is it?
|
||||
- The first "I REMEMBER" item is titled "Session (2026-04-30)" but labeled "yesterday" — April 30th is not yesterday.
|
||||
- The exact same memory — "Session (2026-04-30): What is sovereign AI — 4 messages" — appears twice in the list, once as "yesterday" and once as "1w ago." If it remembers me, why does it stutter?
|
||||
|
||||
When the whole pitch is "I remember you," visible glitches in the remembering feel like catching someone in a small lie.
|
||||
|
||||
### 3. Visible agent growth — 2
|
||||
|
||||
I'm told this assistant learns and gets better at helping me. I could not see that anywhere I'd actually look.
|
||||
|
||||
- The "Evolution" tab (under Memory) — which I guess is the "it improves itself" screen — is empty: "No runs in proposed." Filters say proposed / accepted / deployed / rejected / failed, words that mean nothing to me, around a blank page.
|
||||
- The Agent Center says "No agents yet — create one to put it to work." But the Home screen just told me things ran overnight and a "Writer demo — Anya" exists. So is something working for me or not? The two screens disagree.
|
||||
- The one real piece of evidence — a skill the assistant built by itself, marked with a small amber "agent · review" pill in the Skills Hub — is a tiny badge in a long list of identical rows (and in my screenshots it's hidden behind the welcome popup). Even if I spotted it, "agent · review" doesn't tell me "I built this for you myself" — I'd guess it means I'm being asked to review something.
|
||||
- "8 memories consolidated" overnight is the closest thing to "I'm getting smarter," but "consolidated" is filing-cabinet talk, not growth I can feel.
|
||||
|
||||
Nothing anywhere says, in plain words, "Since last week I learned X and I'm now better at Y." That's the promise, and I can't see it.
|
||||
|
||||
### 4. Desire to return — 3
|
||||
|
||||
The skeleton of a reason to come back is genuinely here: I left things half-done and the app shows them to me with Continue buttons; the "Here's what happened" digest implies the app works while I sleep; "Up next" promises overnight activity. Open loops pull people back — that's real.
|
||||
|
||||
But look at what tomorrow actually promises me: "Memory consolidation at Jun 13, 3:00 AM," "Marketplace sync," "Index reconciliation at Jun 14, 4:00 AM." That's the janitor's schedule, not a gift. Nothing says "your draft will be ready in the morning" or "I'll have those three articles summarized by 9am." And the "Suggested next actions" list — the very thing meant to pull me back in — includes obvious machine debris (see complaint 1), which makes the suggestions feel like a broken vending machine rather than a thoughtful assistant. I'd come back out of curiosity, not craving.
|
||||
|
||||
### 5. Absence of friction — 2
|
||||
|
||||
For someone like me, this app talks to itself, not to me, in a lot of places:
|
||||
|
||||
- The sidebar: Artifacts, Agent Center, Skills Hub, Automation Center, Room, Waggle Dance, Connector Hub, MCP Hub, Marketplace, AI Tools, Vault, Mission Control, Timeline, Events & Logs, Usage & Cost, Spawn Agent. I understand maybe five of these. "Waggle Dance"? "MCP Hub"? "Spawn Agent" sounds like a horror film.
|
||||
- The chat header shows a dropdown set to "YOLO". I only know YOLO as a joke teenagers made. If this controls how much the assistant does without asking me, naming it a joke is alarming, not cute.
|
||||
- "claude-sonnet-4-6" sits in the top bar like a serial number.
|
||||
- The side panel says "LAST ACTIVITY: tool_result: create_skill" — raw machine text with underscores, shown to me as if I should understand it.
|
||||
- The greeting brags "15 memories · 214 entities · 6 relations" — what is an entity? Why am I being shown database vocabulary in a hello?
|
||||
- A suggested action literally reads "Resume: Reply with the literal string PHASE_X_OK and nothing else."
|
||||
- And the "Good evening, Marko" popup sits on top of the Memory screen AND the Skills screen in my session — I apparently can't visit two sections without the same popup blocking my view again.
|
||||
|
||||
The onboarding proves the team can write plain, warm English. The rest of the app forgot.
|
||||
|
||||
## Concrete Complaints
|
||||
|
||||
1. **Test/debug garbage shown as suggestions (Home, 02).** "Suggested next actions" includes "Resume: Reply with the literal string PHASE_X_OK and nothing else." and "Resume: I need you to read and analyze files at D:\Projects\GPM-Waggle-OS on my local..." Filter trivial/test sessions out of the suggestion engine; never surface raw prompts with file paths or test strings.
|
||||
2. **Time contradiction in the comeback moment (01 + 02).** Headline: "You've been away 10 days, Marko." Popup on top of it: "active yesterday," with an item titled "Session (2026-04-30)" labeled "yesterday" (it's June 12). Reconcile all relative timestamps to one source of truth before showing the greeting.
|
||||
3. **Duplicate memory in "I REMEMBER" (01).** "Session (2026-04-30): What is sovereign AI — 4 messages" is listed twice with different ages ("yesterday" / "1w ago"). De-duplicate the recall list.
|
||||
4. **Welcome popup re-blocks other screens (03, 04, 04b).** The "Good evening, Marko" modal overlays the Memory Center and the Skills Hub, hiding their entire content. Show it once on Home, dismiss permanently for the session, never re-open over other screens.
|
||||
5. **Sidebar jargon wall (02).** "Artifacts," "Waggle Dance," "Connector Hub," "MCP Hub," "Vault," "Mission Control," "Spawn Agent" — 22 items with no plain-language descriptions. Rename to task words ("My files," "Add apps," "Passwords & keys") or add hover/sub-labels; consider collapsing power-user sections for new accounts.
|
||||
6. **"YOLO" as an autonomy setting (07, chat header).** A safety-relevant control is labeled with a meme. Rename to something self-explanatory ("Acts without asking") and explain the risk on hover.
|
||||
7. **Raw machine strings in the user's view (07).** Right panel shows "LAST ACTIVITY: tool_result: create_skill"; top bar shows "claude-sonnet-4-6." Translate to human text ("Created a new skill for you") and hide model IDs behind Settings.
|
||||
8. **Database vocabulary in the greeting (01).** "15 memories · 214 entities · 6 relations across 3 workspaces." Replace "entities/relations" with "people, projects & things it knows about" or drop the counts.
|
||||
9. **The self-evolving superpower is invisible (13, 05, 04b).** Evolution tab: "No runs in proposed" (empty, with the jargon filters proposed/accepted/deployed/rejected/failed). Agent Center: "No agents yet" — directly contradicting Home's claim that automations ran and a Writer agent exists. The only growth evidence is a tiny "agent · review" pill. Add a plain "What I learned recently" card on Home, make the self-built skill a celebrated moment ("I built this skill for you — take a look"), and reconcile Agent Center with the agents Home says are working.
|
||||
10. **Onboarding step 2 assumes export literacy (10).** "OpenAI export (.json)," "Google Takeout (.json)," "Threads export" with bare Choose file buttons. Add a one-line "How do I get this?" link per card.
|
||||
11. **Maintenance schedule presented as my agenda (02).** "Up next: Memory consolidation at Jun 13, 3:00 AM... Index reconciliation at Jun 14, 4:00 AM." Hide internal jobs or reframe as user value ("Tonight I'll tidy up memory so tomorrow's answers are faster").
|
||||
|
||||
## Bottom Line
|
||||
|
||||
The onboarding and the "welcome back" moment show this team knows exactly what the magic is supposed to feel like — and for about ninety seconds, I felt it. Then the app started speaking machine to me, contradicted its own memory in the same breath it bragged about it, and hid its "I'm getting better" story behind empty tabs and a cryptic badge. Memory: visible, glitchy. Growth: claimed, not shown.
|
||||
68
judging/judge-2-casual-professional.md
Normal file
@@ -0,0 +1,68 @@
|
||||
# Judge 2 — The Casual Non-Technical Professional
|
||||
|
||||
## Persona
|
||||
|
||||
I'm a marketing manager. I live in email, Slack, and PowerPoint. I use ChatGPT a couple of times a week when I need a draft or an idea. I do not export JSON files, I do not know what an "entity" is, and I have exactly one session of patience for a new tool. If it makes me feel smart and saves me time in the first ten minutes, I'll come back. If it makes me feel like I wandered into an engineering tool, I'm out.
|
||||
|
||||
Evidence reviewed: all 13 screenshots in `judging/screenshots/` (zoomed into full-resolution crops of the Home cockpit, welcome modal, workspace resume chat, Skills Hub, Agent Center, Automation Center, Evolution screen, and command palette).
|
||||
|
||||
## Scores
|
||||
|
||||
| # | Criterion | Score (1–5) |
|
||||
|---|-----------|:-----------:|
|
||||
| 1 | First-session clarity | **4** |
|
||||
| 2 | "It knows me" feeling | **4** |
|
||||
| 3 | Visible agent growth | **2** |
|
||||
| 4 | Desire to return | **4** |
|
||||
| 5 | Absence of friction | **3** |
|
||||
| | **Total** | **17 / 25** |
|
||||
|
||||
## Per-criterion reasoning
|
||||
|
||||
### 1. First-session clarity — 4
|
||||
|
||||
The onboarding is honestly the best part. Three steps, plain language, and it respects my time: "Welcome to the Hive — Persistent memory. Workspace-native." with a single Continue button and a privacy line I actually appreciated ("Your memory and data stay on your device"). Step 1 asks things I can answer (my name, what kind of work I do, what I want help with — "Draft documents & content", yes please). Step 2 *auto-detected* 425 items of existing AI history with a one-click "Harvest" button — that's the closest thing to magic in the whole flow. Step 3 is one choice and one text field. There's a "Skip setup" escape hatch on every screen. I knew what this app was for before I finished setup.
|
||||
|
||||
Why not 5: once I'm inside, the left dock has fifteen-plus entries — Agent Center, Skills Hub, Automation Center, MCP Hub, Vault, Weaver, WaggleDance — and the command palette advertises "22 powerful commands." Five minutes in, I know what the *chat* does for me; I could not tell you what half the dock does, and nothing invites me to find out gently. Also the "PREVIEW" box on onboarding step 1 just displays "Marko." — a preview of nothing.
|
||||
|
||||
### 2. "It knows me" feeling — 4
|
||||
|
||||
This is where the app shows off, and a lot of it lands. "Good evening, Marko." The Home screen says "You've been away 10 days, Marko. Here's what happened" — no other tool I use says that. The "I REMEMBER" panel quotes back an actual preference of mine: "I always work with a draft → critique → rewrite loop. The critique pass is the most important." That one line is the whole pitch — it learned *how I work*, not just what I said. And the Writer demo resume (screenshot 07) is genuinely my job: it reconstructed "DECISION 1: Q3 Editorial Pivot — 'Skepticism Over Hype'" with who approved it and when. If it does that with my real campaigns, I'm sold.
|
||||
|
||||
Why not 5 — because the memory feature contradicts itself in the very panel that's supposed to prove it: the same item ("Session (2026-04-30): What is sovereign AI — 4 messages") appears **twice**, labeled "yesterday" and "1w ago." The modal header says "active yesterday" while the Home screen behind it says I've been away **10 days**. And a session dated April 30 labeled "yesterday" on June 12? When the remembering machine gets the remembering visibly wrong, I start doubting everything else it tells me.
|
||||
|
||||
### 3. Visible agent growth — 2
|
||||
|
||||
I was told the assistant learns my workflows and upgrades its own skills. From what I can see, that story has no stage. The Evolution screen (13) — the screen literally named for this — says "No runs in proposed" and "Select a run to review," with jargon about "baseline vs winner" and accepting/rejecting "the proposal." Empty and incomprehensible. The Agent Center (05) says "No agents yet — create one to put it to work," which for a returning 10-day user reads as "nothing has been working for you." The one real artifact — a skill the agent authored itself with an "agent · review" badge (04b) — is a tiny tag buried in a long list (and in my evidence it's half-hidden behind the welcome modal). The overnight panel's "8 Memories consolidated / 5 Automations completed" is a hint of growth, but "0 Artifacts created" next to it undercuts it. As a casual user, I cannot point at anything and say "look, it got better at helping *me*." The capability may exist; the *feeling* does not.
|
||||
|
||||
### 4. Desire to return — 4
|
||||
|
||||
The return loop is well designed. "You've been away 10 days, here's what happened," three resume cards with one-click "Continue," overnight stats, an "Up next" schedule, and a "/catchup — get up to speed instantly" command suggested first in the palette. The Writer demo resume proved the payoff: I came back and my decisions, rationale, and next steps were waiting. That's a real reason to open it tomorrow — my work compounds here instead of evaporating like it does in ChatGPT.
|
||||
|
||||
Why not 5: the rough edges make it feel like a beta I should check back on "later" rather than a tool I trust daily. The suggested actions include obvious machine debris (see complaint 4), the chat shows raw markdown, and the Agent/Evolution screens are empty — so the "it worked for you overnight" story is told by counters, not by anything I can actually see or use.
|
||||
|
||||
### 5. Absence of friction — 3
|
||||
|
||||
Nothing hard-blocked me, and the onboarding is genuinely low-friction (auto-detect, skip everywhere, no account wall in evidence). But three things came close to a window-close:
|
||||
|
||||
- The "Good evening, Marko" modal is sitting on top of the **Memory Center** (03) and the **Skills Hub** (04, 04b) in my evidence — three different screens, same popup blocking the content I navigated to. A greeting that follows me around stops being a greeting.
|
||||
- The Home screen's "Suggested next actions" served me "Resume: Reply with the literal string PHASE_B_OK and nothing else." That is test garbage wearing a suggestion's clothes, and it instantly made the whole panel feel auto-generated and unsafe to trust.
|
||||
- The app keeps speaking engineer at me: "15 memories · 214 entities · 6 relations," "tool_result: create_skill," "claude-sonnet-4-6," a mode called "YOLO." I don't know what any of that means, and it signals this tool wasn't built for someone like me.
|
||||
|
||||
None of these stopped me from working, hence 3 and not lower — but each one is a paper cut, and paper cuts are why casual users don't come back.
|
||||
|
||||
## Concrete complaints
|
||||
|
||||
1. **Welcome modal stalks every screen.** The same "Good evening, Marko" panel overlays the Memory Center (03-memory-center.png) and Skills Hub (04, 04b) — not just Home. Show it once per session, on Home only; "Don't show again" shouldn't be the only defense.
|
||||
2. **Duplicate memory item in "I REMEMBER".** The welcome modal lists "Session (2026-04-30): What is sovereign AI — 4 messages" twice — once tagged "yesterday," once "1w ago." Dedupe the panel.
|
||||
3. **The app disagrees with itself about when I was last here.** Modal header: "active yesterday." Home screen behind it: "You've been away 10 days, Marko." And an April 30 session labeled "yesterday" on June 12. One source of truth, please — this is the feature I'm supposed to trust most.
|
||||
4. **"Suggested next actions" regurgitates raw old prompts.** On the Home cockpit: "Resume: Reply with the literal string PHASE_B_OK and nothing else." and "Resume: I need you to read and analyze files at D:\Projects\PM-Waggle-OS on my local…" Summarize these into human task descriptions ("Continue the file analysis in PM-Waggle-OS") and filter out one-off/test-looking prompts.
|
||||
5. **Chat shows raw Markdown.** In Writer demo — Anya (07), the assistant's reply renders literal `**Period covered:**`, `## DECISION 1`, and `###` headers as plain text with asterisks. Render it.
|
||||
6. **Engineer jargon on user-facing surfaces.** "214 entities · 6 relations" (welcome modal), "LAST ACTIVITY: tool_result: create_skill" (workspace sidebar, 07), "claude-sonnet-4-6" and a "YOLO" mode in the chat header. Translate or hide: "things it knows about," "Created a skill," a friendly model label, and a mode name that doesn't sound like a dare.
|
||||
7. **The self-evolving story has no visible proof.** Evolution screen (13): "No runs in proposed." Agent Center (05): "No agents yet" — for a returning user with 13 automations and 425 harvested items. The single "agent · review" skill badge (04b) is the only evidence and it's a small tag in a long list. Give growth a front-row surface: a "What I learned this week" card on Home, or surface the agent-authored skill there.
|
||||
8. **Empty workspace card filled with marketing copy.** The "New Hive" card in the welcome modal says "Everything you discuss in New Hive stays in context — decisions, research, and progress are remembered across sessions" — that's brochure text where my data should be. Say "Nothing here yet — start a chat" instead.
|
||||
9. **Onboarding step 1 "PREVIEW" box is dead weight.** It displays just "Marko." — either preview the actual greeting ("Good evening, Marko — ready to draft?") or remove the box.
|
||||
|
||||
## Bottom line
|
||||
|
||||
The memory promise is real and I felt it — greeted by name, my 10-day absence acknowledged, my own writing process quoted back to me, and a content-strategy decision log that's exactly what my job needs. But the memory surface contradicts itself twice in its own showcase modal, the "self-evolving" superpower is effectively invisible (two empty screens and one buried badge), and the app keeps leaking developer plumbing into my view. One solid session of polish on the welcome modal, the suggestions feed, and the jargon would move this from "interesting beta" to "tool I tell my team about."
|
||||
171
judging/judge-3-power-user.md
Normal file
@@ -0,0 +1,171 @@
|
||||
# Judge 3 — Non-Developer Power User Verdict
|
||||
|
||||
## Persona
|
||||
|
||||
Operations lead. I don't code; I live in Notion, Airtable, and Zapier. I build automations
|
||||
for a living, I learn every keyboard shortcut in week one, and I will open every menu you
|
||||
ship. I judge a tool on whether depth is rewarded — whether the power surfaces have real
|
||||
machinery behind them or are just chrome.
|
||||
|
||||
Evidence reviewed: 14 screenshots (1440x900, plus 2x quadrant crops for the dense ones)
|
||||
and live sidecar API calls (`/api/home/briefing`, `/api/skills`, `/api/skills/presentation-design`,
|
||||
`/api/automations`, `/api/agents`, `/api/evolution/runs`) via the bootstrap session token.
|
||||
|
||||
## Scores
|
||||
|
||||
| # | Criterion | Score (1–5) |
|
||||
|---|-----------|-------------|
|
||||
| 1 | First-session clarity | **4** |
|
||||
| 2 | "It knows me" feeling | **4** |
|
||||
| 3 | Visible agent growth | **2** |
|
||||
| 4 | Desire to return | **3** |
|
||||
| 5 | Absence of friction | **2** |
|
||||
|
||||
**Total: 15 / 25**
|
||||
|
||||
## Per-criterion reasoning
|
||||
|
||||
### 1. First-session clarity — 4
|
||||
|
||||
The onboarding is genuinely excellent and the best three minutes of the product. Welcome →
|
||||
"Tell us who you are" (role chips, team size, goals — and a live PREVIEW of how it will greet
|
||||
me, a lovely touch) → "Where do you use AI today?" → first workspace. Step 2 is the standout:
|
||||
it auto-detected Claude Code ("Found 425 items at C:\Users\MarkoMarkovic\.claude") with a
|
||||
one-click **Harvest** button. That is the memory promise made tangible before I've typed a
|
||||
word. Workspace types (Project/Client/Research/Personal) with "Each workspace is its own
|
||||
brain" is exactly how an ops person thinks.
|
||||
|
||||
The main shell is mostly graspable: left nav grouped into core (Home/Chat/Memory/Files/
|
||||
Artifacts), INTELLIGENCE, EXTEND, SYSTEM. Home explains itself.
|
||||
|
||||
Why not 5: the nav carries 20+ entries on first run, and several names are opaque to a
|
||||
non-developer — **Room**, **Waggle Dance**, **Weaver**, **Mission Control**, **MCP Hub**.
|
||||
There's also a naming split between "Centers" (Agent, Automation) and "Hubs" (Skills,
|
||||
Connector, MCP) for sibling concepts. I'd grasp it in session one, but I'd be hovering a
|
||||
lot of labels asking "what's a Waggle Dance?"
|
||||
|
||||
### 2. "It knows me" feeling — 4
|
||||
|
||||
This is the product's strongest axis. The returning-user greeting ("Good evening, Marko",
|
||||
"15 memories · 214 entities · 6 relations across 3 workspaces") plus the Home header
|
||||
("You've been away 10 days, Marko. Here's what happened") is exactly the right move. The
|
||||
"I REMEMBER" panel surfaces a real learned working preference — "I always work with a
|
||||
draft → critique → rewrite loop. The critique pass is the most important…" — which is the
|
||||
single most convincing artifact in the whole evidence set: that's *me*, not my data.
|
||||
Workspace resume (07) is also strong: the agent reconstructs decision history with dates
|
||||
and rationale ("DECISION 1: Q3 Editorial Pivot — 'Skepticism Over Hype', Approved by Marko
|
||||
(founder) on 2026-04-12"), and the right rail shows 11 memories / last activity / pending.
|
||||
|
||||
Why not 5 — three visible cracks in the same surfaces:
|
||||
- The greeting modal says "**active yesterday**" while the Home header behind it says
|
||||
"**You've been away 10 days**" — both visible in screenshot 01 simultaneously. Which is it?
|
||||
- The "I REMEMBER" list shows the **same session twice**: "Session (2026-04-30): What is
|
||||
sovereign AI — 4 messages" appears as both "yesterday" and "1w ago". Duplicate memory
|
||||
in the memory pitch is self-defeating.
|
||||
- A session dated 2026-04-30 timestamped "yesterday" (June 11) doesn't add up.
|
||||
|
||||
### 3. Visible agent growth — 2
|
||||
|
||||
The mission says I should *see* the system learning. I mostly can't:
|
||||
|
||||
- **Agent Center (05) is an empty state**: "No agents yet — create one to put it to work.
|
||||
0 agents, 0 running, avg success —." Yet the Writer demo workspace shows an agent that
|
||||
chatted, made decisions, and has LAST ACTIVITY "tool_result: create_skill". The agent
|
||||
that demonstrably worked for me is invisible in the surface named after it.
|
||||
- **Memory > Evolution (13) is empty**: "No runs in proposed", and the API confirms
|
||||
`{"runs":[],"count":0}` — zero runs ever, not a filter artifact. The headline
|
||||
"self-evolving" surface for a 10-day-absent user with 3 workspaces contains nothing.
|
||||
- The one genuine growth artifact — the agent-authored **presentation-design** skill — is
|
||||
real (API confirms `initiator: agent, source: chat-session`, 10 concrete steps, tool
|
||||
list) and the "agent · review" provenance badge is the right idea. But the screenshot
|
||||
meant to prove it (04b) is blurred behind the welcome modal; I had to verify it via API,
|
||||
which a user can't do.
|
||||
- The 12 "active automations" are **all system defaults** (Memory consolidation, Marketplace
|
||||
sync, Index reconciliation, Harvest sync, Morning briefing…). Useful plumbing, 100%
|
||||
success rate — but none of them was learned from *my* workflows.
|
||||
- Genuine positives that save this from a 1: the overnight panel ("8 memories consolidated",
|
||||
"5 automations completed") makes consolidation visible, and the Evolution empty state at
|
||||
least explains the model ("Each run shows the baseline vs winner… accept or reject the
|
||||
proposal") with a New Run button.
|
||||
|
||||
### 4. Desire to return — 3
|
||||
|
||||
The loop architecture is right: resumable workspace cards with "Continue" + pending counts,
|
||||
overnight stats, suggested next actions, "Up next" schedule, /catchup in the palette, and a
|
||||
Morning briefing automation. The pending-approval counts ("⚠ 1 pending") are a real hook —
|
||||
that's how Zapier gets me back too.
|
||||
|
||||
What deflates it:
|
||||
- **"Up next" is the app's janitorial calendar, not my work**: "Memory consolidation at
|
||||
Jun 13, 3:00 AM · Marketplace sync at Jun 14, 2:00 AM · Index reconciliation at Jun 14,
|
||||
4:00 AM" (confirmed identical in the API). Nobody returns for index reconciliation.
|
||||
- **Suggested next actions are verbatim first-lines of stale sessions**, including
|
||||
"Resume: Reply with the literal string PHASE_B_OK and nothing else." (a leftover test
|
||||
prompt, served to me as a recommendation — confirmed in `/api/home/briefing`). One bad
|
||||
suggestion poisons trust in all four.
|
||||
- The strongest pull (overnight agent work on *my* projects) shows "5 automations
|
||||
completed" but they were maintenance jobs, and "0 artifacts created" — so nothing was
|
||||
actually waiting for me after 10 days away.
|
||||
|
||||
### 5. Absence of friction — 2
|
||||
|
||||
Power-user fast paths are honestly good: Ctrl+K palette with 22 slash commands
|
||||
(/catchup, /now, /research, /draft, /decide, /review, /spawn, /skills), keyboard hints in
|
||||
the palette footer, Ctrl+K chip in the header, a guided 4-step coach-mark tour. The
|
||||
Automation Center has the right tab anatomy (Overview/Running/Scheduled/Triggers/History/
|
||||
Logs). But the friction list is long and concrete — see below. The most damning: the
|
||||
welcome-back modal sits on top of **four** of the evidence screens (Home, Memory Center,
|
||||
Skills Hub twice), meaning it either re-fires across navigation or blocks the two surfaces
|
||||
that prove the product's superpowers; the Agent Center empty state contradicts the rest of
|
||||
the app; and raw internals (tool_result:, PHASE_B_OK, 3 AM cron times) leak into
|
||||
user-facing copy throughout.
|
||||
|
||||
## Concrete complaints
|
||||
|
||||
1. **Welcome modal blocks the superpower screens** (03-memory-center.png, 04/04b-skills-*.png):
|
||||
the "Good evening, Marko" modal overlays the Memory Center and Skills Hub with the page
|
||||
blurred behind it. Whether it re-triggers on navigation or simply wasn't dismissible fast
|
||||
enough, the surfaces that prove Memory and the agent-authored skill badge are unviewable.
|
||||
Make it appear exactly once on Home, never over other screens.
|
||||
2. **Agent Center says "No agents yet — 0 agents" to a user whose agent just worked**
|
||||
(05-agent-center.png vs 07-workspace-resume.png): Writer demo — Anya shows an agent chat
|
||||
with LAST ACTIVITY "tool_result: create_skill", and `/api/agents` returns `{"agents":[],
|
||||
"count":0}`. Workspace personas must surface here, or the center reads as broken/chrome.
|
||||
3. **"You've been away 10 days" and "active yesterday" shown in the same view**
|
||||
(01-home-welcome-back.png): the header and the greeting modal contradict each other.
|
||||
Pick one definition of activity and reconcile.
|
||||
4. **Duplicate "I REMEMBER" entries** (01): "Session (2026-04-30): What is sovereign AI — 4
|
||||
messages" listed twice ("yesterday" and "1w ago"). Dedupe the recall panel; a memory
|
||||
product cannot visibly double-count memories.
|
||||
5. **Suggested next action = "Resume: Reply with the literal string PHASE_B_OK and nothing
|
||||
else."** (02-home-cockpit.png, confirmed in /api/home/briefing): suggestions are raw
|
||||
first-lines of old prompts with no summarization or junk filtering. Summarize intent
|
||||
("Continue the PM-Waggle-OS file analysis") and drop trivial/test sessions.
|
||||
6. **"Up next" contains only system maintenance** (02): Memory consolidation 3:00 AM,
|
||||
Marketplace sync 2:00 AM, Index reconciliation 4:00 AM. Show my deadlines/agent
|
||||
deliverables here; tuck janitorial crons into Automation Center > Scheduled.
|
||||
7. **Raw dev-speak in user-facing fields** (07): LAST ACTIVITY reads "tool_result:
|
||||
create_skill". An ops user should see "Created skill: presentation-design".
|
||||
8. **Automation Center Overview is three stat tiles over a void** (06): "13 automations"
|
||||
in the header, 12 active, 100% success — and the rest of the screen is empty honeycomb.
|
||||
The Overview should show the next 3 runs and last 3 results without a tab switch.
|
||||
9. **Evolution tab is empty for a seasoned account** (13, `/api/evolution/runs` = 0):
|
||||
the self-evolution showcase has never run. Seed it from real activity (e.g., propose a
|
||||
skill improvement after N uses) or auto-trigger a first run post-harvest, otherwise the
|
||||
marquee differentiator looks like vaporware to exactly the user it should wow.
|
||||
10. **All "active automations" are stock system jobs** (06 + /api/automations): none of the
|
||||
12 derive from my behavior (no "learned: you draft → critique → rewrite; automate the
|
||||
critique pass?"). The learned-preference data exists (see I REMEMBER) — close the loop.
|
||||
11. **Cryptic nav labels for non-developers** (02 sidebar): "Room", "Waggle Dance",
|
||||
"Weaver", "MCP Hub", "Mission Control" carry no hint of function; "Center" vs "Hub"
|
||||
naming is inconsistent across sibling surfaces. Add one-line hover descriptions or
|
||||
rename to plain verbs.
|
||||
|
||||
## Power-user spot checks (asked of me specifically)
|
||||
|
||||
- **Fast path: PASS.** Ctrl+K everywhere, 22 slash commands with descriptions, footer key
|
||||
hints, /catchup as the top suggestion for a returning user — this rewards depth.
|
||||
- **Power surfaces substance: SPLIT.** Automations = real (13 schedules, real next-run
|
||||
timestamps, success tracking) but impersonal. Skills = real (20 skills, one genuinely
|
||||
agent-authored with provenance frontmatter). Evolution = chrome today (0 runs).
|
||||
Agent Center = chrome today (0 agents despite agent activity).
|
||||
83
judging/judge-4-junior-developer.md
Normal file
@@ -0,0 +1,83 @@
|
||||
# Judge 4 — Junior Developer Verdict
|
||||
|
||||
## Persona summary
|
||||
|
||||
I'm two years into my first dev job. VS Code, Copilot, and ChatGPT are my daily drivers, so my polish bar is set by tools with hundreds of engineers behind them. I'm exactly the "curious about AI agents" user Waggle should hook: I have a real `.claude` folder full of history (the onboarding found 425 items of mine), I click every tab, and I notice when a chat bubble shows raw `**asterisks**` because ChatGPT never does that.
|
||||
|
||||
Evidence reviewed: all 14 screenshots (plus the `_crops` detail set and my own crops of the small renders), and live probes of `http://localhost:8080` (`/api/skills`, `/api/home/briefing` — both correctly return `401 MISSING_TOKEN` without a session, which is good security but limited my API spelunking).
|
||||
|
||||
## Scores
|
||||
|
||||
| # | Criterion | Score (1–5) |
|
||||
|---|-----------|-------------|
|
||||
| 1 | First-session clarity | **4** |
|
||||
| 2 | "It knows me" feeling | **4** |
|
||||
| 3 | Visible agent growth | **3** |
|
||||
| 4 | Desire to return | **4** |
|
||||
| 5 | Absence of friction | **2** |
|
||||
|
||||
## Per-criterion reasoning
|
||||
|
||||
### 1. First-session clarity — 4/5
|
||||
|
||||
The onboarding is the best part of the whole package. Three steps, and each one teaches the mental model instead of collecting data:
|
||||
|
||||
- Step 3 literally says **"Each workspace is its own brain — memory, files, and agents stay isolated."** That's the entire workspace mental model in one sentence, on the screen where you create your first workspace.
|
||||
- Step 2 ("Where do you use AI today?") auto-detected Claude Code with **"Found 425 items at C:\Users\MarkoMarkovic\.claude"** and a one-click Harvest button. As a Claude Code user this instantly made the memory pitch concrete — it's not "we will remember," it's "we found your existing memory, want it?"
|
||||
- The welcome screen's privacy line ("Your memory and data stay on your device") answers the question I'd ask first.
|
||||
|
||||
Why not 5: once inside, the surface area explodes — a ~20-item sidebar (Agent Center, Skills Hub, Automation Center, Room, Waggle Dance, Connector Hub, MCP Hub, Weaver, Vault, Mission Control...) plus 8 tabs inside a workspace. "Waggle Dance," "Weaver," and "Room" mean nothing on first read. And the mental model takes a hit when the **Agent Center says "No agents yet"** while a Writer agent is demonstrably chatting in my Writer demo workspace — so where do agents live, exactly?
|
||||
|
||||
### 2. "It knows me" feeling — 4/5
|
||||
|
||||
This is the pillar that mostly lands:
|
||||
|
||||
- "Good evening, Marko" modal with real stats (15 memories · 214 entities · 3 workspaces), an "I REMEMBER" section that recalled an actual stylistic preference — *"I always work with a draft → critique → rewrite loop. The critique pass is the most important"* — which is exactly the kind of thing I'd want a tool to remember.
|
||||
- The Home cockpit headline **"You've been away 10 days, Marko. Here's what happened:"** plus per-workspace resume cards with pending counts and Continue buttons.
|
||||
- The killer demo is screenshot 07: clicking "Review recent decisions and next steps" produced a real reconstruction — "DECISION 1: Q3 Editorial Pivot — 'Skepticism Over Hype' ... Approved by Marko (founder) on 2026-04-12," including the rationale and voice rules. Copilot and ChatGPT have nothing like a per-workspace decision history I can replay.
|
||||
- `/catchup` in the Ctrl+K palette ("Workspace restart summary — get up to speed instantly") is the right verb for the feeling.
|
||||
|
||||
Why not 5: the memory surface contradicts itself, which is fatal for a *trust* feature. The greeting modal's meta line says **"active yesterday"** while the headline directly behind it says **"You've been away 10 days."** Worse, the same session — "Session (2026-04-30): What is sovereign AI — 4 messages" — is listed **twice** in "I REMEMBER," once tagged "yesterday" and once "1w ago," for a session dated April 30 when today is June 12. A memory product showing me wrong/duplicated memories of *itself* makes me wonder what else it misremembers.
|
||||
|
||||
### 3. Visible agent growth — 3/5
|
||||
|
||||
There is real evidence, but it's thin and the dedicated surfaces are empty:
|
||||
|
||||
- Real: the Skills Hub has one skill carrying an **"agent · review"** badge (orange, distinct from the green Enabled chips) — agent-authored, gated behind human review. That's the right provenance design. The workspace side panel also shows **"LAST ACTIVITY: tool_result: create_skill"** — the agent really did author a skill.
|
||||
- Real: the overnight panel ("8 Memories consolidated · 0 Artifacts created · 5 Automations completed") and the Automation Center (13 automations, 12 active schedules, 100% success rate) prove autonomous activity while I was away.
|
||||
- Empty: the **Memory → Evolution tab shows "No runs in proposed"** — the one surface explicitly named after self-evolution has nothing in it (and its default filter lands on the empty bucket). The **Agent Center shows "No agents yet."** For a seeded returning-user demo, the two screens that should prove "it keeps getting better" are honeycomb wallpaper.
|
||||
|
||||
Net: the self-evolution story is ~one badge and one raw event label deep. It's not marketing — the plumbing visibly exists — but as shown, I see the *receipts* of growth more than the growth itself.
|
||||
|
||||
### 4. Desire to return — 4/5
|
||||
|
||||
Honest answer: yes, I'd keep it running next to VS Code, at least for a few weeks. The hooks that work on me specifically:
|
||||
|
||||
- It harvested my actual Claude Code history (425 items) — the memory compounds from day one rather than starting cold.
|
||||
- The "away 10 days, here's what happened" briefing + overnight automations means the app does work between my sessions; that's a reason to open it each morning that ChatGPT doesn't have.
|
||||
- The decision-history replay in 07 is something I'd genuinely use after a sprint break.
|
||||
|
||||
Why not 5: daily delight erodes fast when the main chat surface renders raw markdown (see complaint #2) and the cockpit suggests I "Resume: Reply with the literal string PHASE_B_OK and nothing else." The value prop is differentiated; the day-to-day polish is below the Cursor/ChatGPT bar I'm used to, so my return habit would survive on utility, not delight.
|
||||
|
||||
### 5. Absence of friction — 2/5
|
||||
|
||||
I found a lot of concrete rough edges for 14 screenshots — several on the highest-traffic surfaces (greeting modal, home suggestions, chat). The full list is below; the headline items are test-data junk in "Suggested next actions," unrendered markdown in chat, the greeting modal contradicting itself, and the modal squatting over three different screens with backdrop blur.
|
||||
|
||||
## Concrete complaints
|
||||
|
||||
1. **Home cockpit → "Suggested next actions" surfaces QA junk.** Card 3 reads "Resume: Reply with the literal string PHASE_B_OK and nothing else." and card 4 "Resume: Compare two recent memory frames briefly. List 2 trade-offs." These are raw replays of prior (test) prompts. Filter trivial/test prompts out of the suggestion engine, or summarize intent instead of echoing prompt text.
|
||||
2. **Workspace chat renders markdown as literal text** (07-workspace-resume): the assistant bubble shows `# Decision Review & Next Steps`, `**Period covered:** April–May 2026`, `## DECISION 1: Q3 Editorial Pivot` with visible `#`/`**` characters. Render markdown in assistant messages — every long answer currently looks broken next to ChatGPT.
|
||||
3. **The greeting modal contradicts the page behind it.** Modal meta: "active yesterday." Headline behind it: "You've been away 10 days, Marko." Same viewport, two truths. Pick one source for last-activity.
|
||||
4. **Duplicate + mis-dated memory in "I REMEMBER."** "Session (2026-04-30): What is sovereign AI — 4 messages" appears twice, tagged "yesterday" and "1w ago," for a session dated April 30 (today is June 12). Dedupe entries and fix relative-time computation — this is the trust-critical surface.
|
||||
5. **The greeting modal persists over other screens.** Screenshots 03 (Memory Center), 04 and 04b (Skills Hub) all show the same "Good evening, Marko" modal with backdrop blur on top of the content — including blurring out the very "agent · review" badge that proves self-evolution. Dismiss the modal on any navigation.
|
||||
6. **Agent Center empty state is wrong for this user** (05): "No agents yet" — while the Writer agent is active in "Writer demo — Anya" and its last activity was `create_skill`. Either list workspace agents under the All/Workspace tabs or explain what counts as an "agent" here.
|
||||
7. **Memory → Evolution defaults to an empty bucket** (13): the tab opens on the "proposed" filter showing "No runs in proposed," with no explanation of what an evolution run is. Default to "all," and give the empty state a one-line primer + pointer to the "New Run" button.
|
||||
8. **Raw internal event name in the workspace panel** (07-top): "LAST ACTIVITY: tool_result: create_skill". Humanize it — "Created a skill (10d ago)" — and link it to the skill in Skills Hub.
|
||||
9. **Pluralization bug** (07-top right panel): "11 memories · 1 sessions".
|
||||
10. **Ctrl+K coach-mark overlaps the open palette** (12): the "Type / for 22 powerful commands…" tooltip covers the right edge of the command list (the Command labels) while you're trying to read it. Reposition the tooltip outside the palette bounds.
|
||||
11. **"Up next" mixes system maintenance into the user's agenda** (02-bottom): "Memory consolidation at Jun 13, 3:00 AM," "Marketplace sync," "Index reconciliation at 4:00 AM." I don't need cron jobs in my morning briefing — collapse them into one "system" row or move them to Automation Center.
|
||||
12. **Onboarding step 1 PREVIEW box just says "Marko."** (09): the box labeled PREVIEW renders a bare name with a period, which reads as a bug. Show the actual greeting it's previewing ("Good evening, Marko — ready to pick up the Q3 editorial work?").
|
||||
|
||||
## Bottom line
|
||||
|
||||
The memory pillar is real and demonstrable — onboarding-to-harvest is the best AI-app first-run I've seen, and the decision-history replay would actually change how I work. The self-evolution pillar exists in the plumbing (agent-authored skill + review badge + automations) but its showcase surfaces are empty. And the rough-edge count — test junk in suggestions, unrendered markdown, a self-contradicting greeting — is what separates "promising beta I'd demo to my team" from "tool I'd defend like VS Code." Fix the twelve items above (most are small) and criteria 2, 4, 5 all move up a point.
|
||||
81
judging/judge-5-senior-skeptic.md
Normal file
@@ -0,0 +1,81 @@
|
||||
# Judge 5 — Senior Engineer / Professional Skeptic
|
||||
|
||||
**Persona:** 15 years shipping products. I have watched every "AI that learns" demo collapse the moment someone opens the database. I assumed every claim here was inflated until the UI or the API proved it. I read all 12 screenshots, then verified against the live sidecar (`/api/home/briefing`, `/api/home/overnight`, `/api/memory?limit=200`, `/api/memory/stats`, `/api/skills`, `/api/audit/installs`, `/api/agents`, `/api/automations`, `/api/evolution/runs`, `/api/identity`) and read the relevant source (`packages/server/src/local/routes/skills.ts`, `routes/memory.ts`, `routes/memory-center.ts`, `apps/web/src/lib/login-briefing-brag.ts`) plus the skill file on disk (`~/.waggle/skills/presentation-design.md`).
|
||||
|
||||
**Headline:** This is more real than 90% of "AI with memory" demos I've audited — the memory is genuinely persisted SQLite, the overnight work maps to real cron jobs with real `lastRun` timestamps, and the skill provenance stamp exists in actual file frontmatter. But the *self-evolution* story is infrastructure without evidence (0 evolution runs, 0 agents, 1 agent-authored skill that predates its own audit trail), and the returning-user surface leaks test junk and contradicts itself about when I was last here.
|
||||
|
||||
## Scores
|
||||
|
||||
| # | Criterion | Score (1–5) |
|
||||
|---|---|---|
|
||||
| 1 | First-session clarity | **4** |
|
||||
| 2 | "It knows me" feeling | **3** |
|
||||
| 3 | Visible agent growth | **2** |
|
||||
| 4 | Desire to return | **4** |
|
||||
| 5 | Absence of friction | **2** |
|
||||
| | **Total** | **15 / 25** |
|
||||
|
||||
---
|
||||
|
||||
## Per-criterion reasoning
|
||||
|
||||
### 1. First-session clarity — 4/5
|
||||
|
||||
Evidence: 08/09/10/11-onboarding screenshots.
|
||||
|
||||
The onboarding is honest and unusually graspable for this category. Three steps, each skippable ("Skip setup", "Skip this step"). The mental model is stated in plain words: "Each workspace is its own brain — memory, files, and agents stay isolated" (step 3) and "Bring your existing conversations — Waggle extracts decisions, preferences, and knowledge into your persistent memory" (step 2). The memory-import step earns real credit: it *detected* an actual local install ("Claude Code detected — Found 425 items at C:\Users\MarkoMarkovic\.claude") rather than just offering upload boxes — that is a concrete, checkable claim, not vapor. The privacy line ("Your memory and data stay on your device. Nothing leaves without your say-so") matches the local-sidecar architecture I probed.
|
||||
|
||||
Why not 5: step 1 collects "Role: Strategy Consultant" and promises "Waggle uses this to ... tailor how it helps," but the live `/api/identity` shows `role`, `department`, and `personality` all empty for this configured user — the personalization promise is not demonstrably honored end-to-end. The "PREVIEW" box that renders just "Marko." is a dead weight on an otherwise good screen. "YOUR AI OPERATING SYSTEM" is puffery, but forgivable puffery.
|
||||
|
||||
### 2. "It knows me" feeling — 3/5
|
||||
|
||||
Evidence: 01-home-welcome-back, 03-memory-center, 07-workspace-resume; verified against `/api/memory`, `/api/memory/stats`, `/api/identity`, memory routes source.
|
||||
|
||||
The memory claim is **not** a parlor trick at the infrastructure level. I verified:
|
||||
- The modal's brag line "15 memories · 214 entities · 6 relations across 3 workspaces" is backed by `/api/memory/stats` → `{frameCount:16, entityCount:214, relationCount:6}` (the off-by-one is a frame written after the modal rendered — acceptable).
|
||||
- Memories are user-visible AND correctable: real statuses in the data (4 active / 8 archived / 4 deprecated), `PUT /api/memory/frames/:id` edit, `POST /api/memory/:id/archive` (reversible, per `memory-center.ts:354`), `DELETE /api/memory/frames/:id`, all emitting audit events I can see in `/api/events` (`memory_write`/`patch` rows). That is the full persisted-visible-correctable triad. Credit earned.
|
||||
- Workspace resume (07) shows genuine restored chat with decision records ("DECISION 1 (i2) Editorial Pivot — 'Skepticism Over Hype'") matching the workspace summary the API returns ("Q3 editorial direction: lean into skepticism, less hype").
|
||||
|
||||
Why only 3: open the hood and the actual memory *content* is thin and polluted. Of 16 frames: four are duplicate "User identity: Name: Marko" (ids 36, 37, 38, 41); one stores the user's *question* as a preference ("User preference: Based on your saved memory only: what is my name..."); two are "Monthly Agent Assessment" blobs that are garbage concatenations of an assessment + a slash command + leftover test prompts ("Create a file named hive-write-proof.md...") — and every one of these system-generated blobs is provenance-stamped `source: "user_stated"`, which is flatly false. The app knows my *name* and my schedule; it does not demonstrably know *me* (identity role/personality empty). The "I remember" hero list shows the same session twice (see complaint 5). Real substrate, weak substance.
|
||||
|
||||
### 3. Visible agent growth — 2/5
|
||||
|
||||
Evidence: 04b-skills-agent-badge, 05-agent-center, 13-memory-evolution; verified `/api/skills`, `/api/audit/installs`, `/api/agents`, `/api/evolution/runs`, `skills.ts`, the skill file on disk.
|
||||
|
||||
What's real: the provenance mechanism exists and is not faked. `~/.waggle/skills/presentation-design.md` carries genuine frontmatter (`initiator: agent`, `source: chat-session`), the skills API surfaces it, and `/api/audit/installs` is a real governance trail with `riskLevel` / `trustSource` / `approvalClass` / `initiator` — including three agent-initiated `proposed` rows for high-risk marketplace capabilities correctly classed `critical`. The write path in `skills.ts` (P5/D4) does stamp and audit every skill write now. The plumbing deserves credit.
|
||||
|
||||
What's missing is the *growth itself*. The numbers, from the product's own APIs: **1** agent-authored skill out of 20 (the other 19 are clearly stock skills — brainstorm, code-review, meeting-prep — all stamped `initiator: "user"`); **0** agents (`/api/agents` → `{"agents":[],"count":0}`, and 05-agent-center literally says "No agents yet"); **0** evolution runs ever (`/api/evolution/runs` → `{"runs":[],"count":0}`, 13-memory-evolution is an empty state); and the agent's own Monthly Assessment memory reads "Interactions: 0, Correction Rate: 0.0%, Skills installed this month: 0." Worse, the one showcase artifact — presentation-design — has **no row in the install audit trail** (the trail starts 2026-05-17 and contains no entry for it), so the "every skill write is audited" claim does not cover the very example offered as proof. And the screenshot supplied to evidence the badge (04b) is blocked by the welcome modal — the badge is not visible in the submitted UI evidence at all; I could only verify it via file and API. A self-evolving agent with zero runs, zero agents, and one pre-audit skill is a self-evolution *capability*, not a self-evolution *demonstration*. Two points for honest, working plumbing.
|
||||
|
||||
### 4. Desire to return — 4/5
|
||||
|
||||
Evidence: 02-home-cockpit; verified `/api/home/overnight`, `/api/automations`, `/api/home/briefing`.
|
||||
|
||||
This is the strongest claim in the product, and it survives scrutiny. "8 memories consolidated / 5 automations completed" is backed verbatim by `/api/home/overnight` (`{"consolidated":8,"automationsCompleted":5,"failures":[]}` over an explicit 24h window), and that in turn is backed by 12+ real scheduled automations with genuine `lastRun` timestamps (Harvest sync, Memory compaction, Memory consolidation all ran at 2026-06-12T01:05). The "Up next" schedule on Home matches actual `nextRun` values (Memory consolidation Jun 13 3:00 AM, Marketplace sync Jun 14 2:00 AM). The overnight work *actually happened*. No dark patterns found anywhere: "Don't show again" on the modal, skip on every onboarding step, and the day-0 empty state in `LoginBriefing.tsx` explicitly labels its demo memory bubbles "Examples." — code-comment evidence the team actively avoids deceptive copy. The loop is engineered on real machine work, not streaks or guilt.
|
||||
|
||||
Why not 5: the *output* of the overnight work is what I'd come back to, and right now it's duplicate identity frames and garbled assessment blobs (criterion 2). And two of the four "suggested next actions" on the cockpit are leftover test prompts (complaint 3) — a novice returning to "Resume: Reply with the literal string PHASE_B_OK and nothing else." would conclude the robot is broken and not come back.
|
||||
|
||||
### 5. Absence of friction — 2/5
|
||||
|
||||
Concrete inconsistencies and dead surfaces, each verified:
|
||||
- The Home screen contradicts itself about my absence: headline "You've been away 10 days, Marko" (from `/api/home/briefing`, workspace `lastActive` max = 2026-06-02) while the welcome modal on the same screen says "active yesterday" (from `computeBragSummary`/`pickGlobalLastActive` in `login-briefing-brag.ts`, fed by a different lastActive source refreshed by the cron's own memory writes). Two recency definitions, one screen, opposite stories.
|
||||
- The welcome modal appears overlaid on Memory Center (03), Skills Hub (04), and the badge close-up (04b) — it obscures the very evidence those captures exist to show. If `LoginBriefing` fires on every route entry rather than once per session, that's a real bug; either way the friction is in the evidence.
|
||||
- Test artifacts leak into three user-facing surfaces: suggested actions (PHASE_B_OK), memory frames (hive-write-proof.md prompt), and the install audit trail (`e2e-test-skill-1776393828953`, `e2e-read-test` rows dated the same day as judging).
|
||||
- Empty surfaces presented as features for a "returning user with real data": Agent Center (0 agents), Evolution tab (0 runs).
|
||||
- Memory Center offers kind/confidence filters while the data contains exactly one kind (`fact`, 16/16) and one source (`user_stated`, 16/16) — filters ahead of any data that could exercise them.
|
||||
|
||||
---
|
||||
|
||||
## Numbered complaints (concrete, actionable)
|
||||
|
||||
1. **04b fails to evidence the badge.** The "agent · review" badge screenshot is fully covered by the LoginBriefing modal. I verified the provenance only via `GET /api/skills` (`presentation-design` → `initiator: "agent"`) and the file frontmatter. Recapture, and audit whether `LoginBriefing` mounts on Memory Center / Skills Hub navigation instead of once per session on Home.
|
||||
2. **Same-screen recency contradiction.** "You've been away 10 days" (briefing headline) vs "active yesterday" (modal brag line). Fix: `pickGlobalLastActive` in `apps/web/src/lib/login-briefing-brag.ts` should consume the same user-activity timestamp the briefing greeting uses, not workspace timestamps refreshed by cron-driven memory writes — machine activity is not "active."
|
||||
3. **Test junk in suggested actions.** `/api/home/briefing` `suggestedActions` includes "Resume: Reply with the literal string PHASE_B_OK and nothing else." and "Resume: Compare two recent memory frames briefly. List 2 trade-offs." The resume engine needs a junk/length/recency filter before surfacing session titles as next actions.
|
||||
4. **Memory pollution + false provenance.** Four duplicate "User identity: Name: Marko" frames (ids 36/37/38/41, three already deprecated but still accumulating); a user *question* stored as "User preference" (id 39); "Monthly Agent Assessment" frames (ids 35/40) are concatenations of assessment text + test prompts + slash commands, all stamped `source: "user_stated"` despite being system-generated. Consolidation should dedup identity facts and system writes must carry a system/derived source label.
|
||||
5. **Duplicate "I remember" hero items.** The modal (01) lists "Session (2026-04-30): What is sovereign AI — 4 messages" twice, once "yesterday" and once "1w ago" — the flagship memory moment shows a dedup failure with mutually inconsistent ages for the same session.
|
||||
6. **Provenance defaults dilute the badge.** `skills.ts` (~line 352): "Absent provenance ⇒ legacy ⇒ 'user'" — 19 stock/built-in skills all report `initiator: "user"`. If everything defaults to "user," the user-vs-agent distinction the badge sells is unfalsifiable for pre-existing content. Legacy/bundled skills should be labeled `built-in`, not attributed to the user.
|
||||
7. **The showcase agent skill predates its own audit trail.** `/api/audit/installs` (11 rows, earliest 2026-05-17) contains no install row for `presentation-design`. "Every skill write is provenance-stamped and audited" holds only post-P5/D4; either backfill an audit row for legacy skills or scope the claim in the UI.
|
||||
8. **Self-evolution has zero runtime evidence.** `/api/evolution/runs` → 0 runs; `/api/agents` → 0 agents; the agent's own monthly assessment reports 0 interactions and 0 skills installed. For a demo whose thesis is "it keeps getting better," ship the returning-user profile with at least one completed evolution run and one live agent, or the Evolution tab and Agent Center actively disprove the pitch.
|
||||
9. **Onboarding personalization not persisted.** Step 1 collects Role/Industry/work-type and promises tailoring; `/api/identity` for the configured user has `role: ""`, `department: ""`, `personality: ""`. Wire the wizard's answers into the identity record or drop the "tailor how it helps" copy.
|
||||
10. **Governance surface shows test artifacts.** The install audit trail — the trust surface — leads with `e2e-test-skill-1776393828953` / `e2e-read-test` uninstall rows. Test fixtures should be namespaced out of the user-visible trail.
|
||||
|
||||
*Meta-note on the evidence pack: screenshots 02, 04, 04b, 05, 06, 07, 12, 13 were supplied at ~360px and are largely illegible; all detail claims for those screens were verified via API instead.*
|
||||
31
judging/round1-fixes.md
Normal file
@@ -0,0 +1,31 @@
|
||||
# Round 1 — Consolidated complaints → fixes
|
||||
|
||||
Scores R1: novice 4/4/2/3/2 · casual 4/4/2/4/3 · power 4/4/2/3/2 · junior 4/4/3/4/2 · skeptic 4/3/2/4/2.
|
||||
Weakest: visible agent growth (2-3 across the board) and friction (2-3).
|
||||
|
||||
| # | Complaint (judges) | Fix | Type |
|
||||
|---|---|---|---|
|
||||
| 1 | "away 10 days" vs "active yesterday" same screen (all 5) | unify brag lastActive with briefing's definition (`login-briefing-brag.ts`) | code |
|
||||
| 2 | duplicate "I REMEMBER" items, Apr-30 session tagged "yesterday" (all 5) | dedup brag memory list by title; age from session date not frame touch | code |
|
||||
| 3 | welcome modal overlays Memory/Skills screens (4) | cooldown already shipped; ALSO gate render to /home route; recapture evidence | code+evidence |
|
||||
| 4 | "PHASE_B_OK" test prompts as Suggested next actions (all 5) | recency filter on resume suggestions (skip stale sessions) + clean dev test sessions | code+data |
|
||||
| 5 | "Up next" = janitor cron schedule (4) | filter system-maintenance jobs from Home upNext; hide empty section | code |
|
||||
| 6 | Evolution tab empty "No runs in proposed" (all 5) | default filter 'all' + plain-language primer; RUN a real evolution run for substance | code+usage |
|
||||
| 7 | Agent Center "No agents yet" contradicts working agents (4) | surface workspace persona agents / fix empty-state; create one real agent via builder | code+usage |
|
||||
| 8 | raw "tool_result: create_skill" last-activity (4) | humanize tool-event labels | code |
|
||||
| 9 | "entities · relations" database vocab in greeting (3) | plain words in brag stats line | code |
|
||||
| 10 | YOLO autonomy label (2) | display label "Autopilot — acts without asking" | code |
|
||||
| 11 | chat renders raw markdown (2) | investigate + render markdown in assistant messages | code |
|
||||
| 12 | nav jargon wall: Waggle Dance/MCP Hub/Weaver (3) | one-line plain tooltips on dock entries | code |
|
||||
| 13 | onboarding PREVIEW box renders bare "Marko." (3) | render actual greeting preview | code |
|
||||
| 14 | import cards assume export literacy (1) | one-line "how to get this" hint per provider | code |
|
||||
| 15 | wizard role/industry never reach /api/identity (1) | wire who-are-you answers into identity record | code |
|
||||
| 16 | dup "User identity" frames keep accumulating; system frames stamped user_stated (1) | consolidation replace-on-update + system source stamp | code |
|
||||
| 17 | stock skills claim initiator:user (1) | absent provenance → 'built-in' label, not user | code |
|
||||
| 18 | e2e-test rows in install audit trail (1) | dev-data cleanup (local profile only) | data |
|
||||
| 19 | "1 sessions" pluralization (1) | fix | code |
|
||||
| 20 | Automation Overview = stats over void (1) | show next runs + recent results on Overview | code |
|
||||
| 21 | empty workspace card shows brochure copy (1) | "Nothing here yet — start a chat" | code |
|
||||
| 22 | Ctrl+K coach-mark overlaps palette (1) | reposition tooltip / suppress while palette open | code |
|
||||
| 23 | model id in top bar (2, low) | leave (expert affordance); revisit if re-flagged | wontfix-r1 |
|
||||
| 24 | presentation-design has no audit row (1) | out of scope: trail is append-only post-P5; UI makes no retroactive claim | wontfix-r1 |
|
||||
BIN
judging/round2/_crops/01-header.png
Normal file
|
After Width: | Height: | Size: 107 KiB |
BIN
judging/round2/_crops/01-modal.png
Normal file
|
After Width: | Height: | Size: 321 KiB |
BIN
judging/round2/_crops/02-left.png
Normal file
|
After Width: | Height: | Size: 1.1 MiB |
BIN
judging/round2/_crops/02-main-bottom.png
Normal file
|
After Width: | Height: | Size: 958 KiB |
BIN
judging/round2/_crops/02-main-top.png
Normal file
|
After Width: | Height: | Size: 1.1 MiB |
BIN
judging/round2/_crops/02-right.png
Normal file
|
After Width: | Height: | Size: 951 KiB |
BIN
judging/round2/_crops/02-sidebar.png
Normal file
|
After Width: | Height: | Size: 262 KiB |
BIN
judging/round2/_crops/03-mid.png
Normal file
|
After Width: | Height: | Size: 937 KiB |
BIN
judging/round2/_crops/03-top.png
Normal file
|
After Width: | Height: | Size: 808 KiB |
BIN
judging/round2/_crops/04-mid.png
Normal file
|
After Width: | Height: | Size: 1.2 MiB |
BIN
judging/round2/_crops/04-top.png
Normal file
|
After Width: | Height: | Size: 994 KiB |
BIN
judging/round2/_crops/04b-badge.png
Normal file
|
After Width: | Height: | Size: 693 KiB |
BIN
judging/round2/_crops/04b-center.png
Normal file
|
After Width: | Height: | Size: 922 KiB |
BIN
judging/round2/_crops/05-row-left.png
Normal file
|
After Width: | Height: | Size: 432 KiB |
BIN
judging/round2/_crops/05-row-right.png
Normal file
|
After Width: | Height: | Size: 329 KiB |
BIN
judging/round2/_crops/05-top.png
Normal file
|
After Width: | Height: | Size: 933 KiB |
BIN
judging/round2/_crops/06-top.png
Normal file
|
After Width: | Height: | Size: 1004 KiB |
BIN
judging/round2/_crops/07-bot.png
Normal file
|
After Width: | Height: | Size: 412 KiB |
BIN
judging/round2/_crops/07-mid.png
Normal file
|
After Width: | Height: | Size: 313 KiB |
BIN
judging/round2/_crops/07-rail.png
Normal file
|
After Width: | Height: | Size: 85 KiB |
BIN
judging/round2/_crops/07-top.png
Normal file
|
After Width: | Height: | Size: 134 KiB |
BIN
judging/round2/_crops/12-palette.png
Normal file
|
After Width: | Height: | Size: 430 KiB |
BIN
judging/round2/_crops/13-left.png
Normal file
|
After Width: | Height: | Size: 742 KiB |
BIN
judging/round2/_crops/13-main.png
Normal file
|
After Width: | Height: | Size: 1.4 MiB |
BIN
judging/round2/_crops/13-right.png
Normal file
|
After Width: | Height: | Size: 593 KiB |
BIN
judging/round2/crops/01-panel.png
Normal file
|
After Width: | Height: | Size: 429 KiB |
BIN
judging/round2/crops/02-bottom.png
Normal file
|
After Width: | Height: | Size: 1.4 MiB |
BIN
judging/round2/crops/02-top.png
Normal file
|
After Width: | Height: | Size: 1.5 MiB |
BIN
judging/round2/crops/03-cards.png
Normal file
|
After Width: | Height: | Size: 1.5 MiB |
BIN
judging/round2/crops/04-bot.png
Normal file
|
After Width: | Height: | Size: 1.8 MiB |
BIN
judging/round2/crops/04-top.png
Normal file
|
After Width: | Height: | Size: 1.8 MiB |
BIN
judging/round2/crops/04b-mid.png
Normal file
|
After Width: | Height: | Size: 1.3 MiB |
BIN
judging/round2/crops/05-top.png
Normal file
|
After Width: | Height: | Size: 1.1 MiB |
BIN
judging/round2/crops/06-top.png
Normal file
|
After Width: | Height: | Size: 1.1 MiB |
BIN
judging/round2/crops/07-chat-bottom.png
Normal file
|
After Width: | Height: | Size: 485 KiB |
BIN
judging/round2/crops/07-chat-top.png
Normal file
|
After Width: | Height: | Size: 422 KiB |
BIN
judging/round2/crops/12-palette.png
Normal file
|
After Width: | Height: | Size: 534 KiB |
BIN
judging/round2/crops/13-top.png
Normal file
|
After Width: | Height: | Size: 2.1 MiB |
80
judging/round2/judge-1-novice.md
Normal file
@@ -0,0 +1,80 @@
|
||||
# Judge 1 — The Complete Novice (Round 2)
|
||||
|
||||
## Persona summary
|
||||
|
||||
I am not a computer person. I message on my phone, and on my laptop I do email and the web. I have never used an "AI desktop app". I do not know what an agent, a workspace, an MCP, or an API is. I judged only what I could see and feel in the screenshots, the way I would on my own at the kitchen table.
|
||||
|
||||
## Scores
|
||||
|
||||
| # | Criterion | Score (1–5) |
|
||||
|---|---|---|
|
||||
| 1 | First-session clarity | **4** |
|
||||
| 2 | "It knows me" feeling | **4** |
|
||||
| 3 | Visible agent growth | **3** |
|
||||
| 4 | Desire to return | **4** |
|
||||
| 5 | Absence of friction | **2** |
|
||||
|
||||
**Total: 17 / 25**
|
||||
|
||||
## Per-criterion reasoning
|
||||
|
||||
### 1. First-session clarity — 4
|
||||
|
||||
The setup is genuinely easy: three short steps, one obvious yellow button each time, and I can skip anything. Step 1 ("Tell us who you are", 09) asks things I can actually answer — my name, what kind of work I do, what I want help with — and the live preview line "Good evening, Marko — your work will be remembered here" instantly shows me what I'm getting. The privacy line on the first screen ("Your memory and data stay on your device. Nothing leaves without your say-so", 08) made me feel safe rather than spied on. Step 3's "Each workspace is its own brain" metaphor (11) actually helped me understand a word I didn't know.
|
||||
|
||||
What stops a 5: Step 2 ("Where do you use AI today?", 10) is the one step that assumes I'm already an AI person. Its main button says **"Harvest"** — I don't know what harvesting my computer means, and the line under it shows a raw folder path (`C:\Users\MarkoMarkovic\.claude`). If "Harvest" said "Import" I'd click it without fear. And the moment setup ends, I land facing a left sidebar of ~20 items I mostly can't read (see complaint 1), which dents my "I know what to do next" confidence.
|
||||
|
||||
### 2. "It knows me" feeling — 4
|
||||
|
||||
This is the app's best trick and it's delivered on screen, not just promised. The welcome panel (01) says "Good evening, Marko", counts "17 memories · 214 people, projects & things it knows across 3 workspaces", and then literally shows an "I REMEMBER" list with my own habits ("I always work with a draft → critique → rewrite loop…"). The Home screen (02) opens with "You've been away 10 days, Marko. Here's what happened:" and shows the exact things I was working on with a Continue button on each. When I resume the Writer workspace (07), the assistant picks up mid-project with "Decision Review & Next Steps — Anya's Content Strategy" without me re-explaining anything. The Memory screen (03) even knows my age, my team, my favorite color, and my football club. I genuinely felt remembered.
|
||||
|
||||
What stops a 5: on the very panel that delivers the magic, the first remembered item reads **"User asked: Review recent decisions and next steps"** (01). "User"? That's me — why is it talking about me in the third person like a machine log? And right under my name sits a yellow warning triangle saying **"⚠ 2 pending"** with no noun — pending *what*? A warning sign with no explanation is the one cold, slightly worrying note on an otherwise warm screen. (Small extra wobble: the header says 17 memories total across 3 workspaces, but two of the workspaces individually claim 11 memories each — the numbers don't obviously add up.)
|
||||
|
||||
### 3. Visible agent growth — 3
|
||||
|
||||
The *story* of growth is told beautifully in one place: the Evolution panel (13) says "Your agent improves itself here… You review each proposal and accept or reject it — nothing changes without you." That sentence is perfect — plain, reassuring, exciting. The Home screen's "Overnight: 9 memories consolidated, 5 automations completed" (02) made me feel it worked while I slept, and the Automation Center's "100% success rate" (06) looks healthy.
|
||||
|
||||
But the *evidence* of growth is thin or hidden everywhere I looked:
|
||||
- The proudest possible moment — a skill the assistant built by itself — is marked with a tiny cryptic badge reading **"agent · review"** (04b, presentation-design row). I would scroll right past it. Nothing says "Your AI built this for you — take a look."
|
||||
- The Evolution panel itself is **empty**: "Select a run to review… + New Run", and its explainer uses words like "baseline vs winner, which gates fired" (13). I would never press "New Run" because I don't know what a run is.
|
||||
- The only agent I have, "Editorial Critic" (05), shows **"run never"**, "Idle", and "avg success —". My one helper looks like it has never done anything.
|
||||
- The "Monthly Agent Assessment" memory cards (03) literally print "**Interactions**: 0" inside broken-looking text.
|
||||
|
||||
So I'm *told* it gets better, and I half-believe it, but I can't *see* it getting better.
|
||||
|
||||
### 4. Desire to return — 4
|
||||
|
||||
Real pulls exist: the away-greeting plus "Continue →" buttons on my unfinished work (02) make tomorrow's first click obvious. "Up next: … Jun 15, 9:00 AM / … Jun 17, 10:00 AM" gives me actual appointments, and the Overnight digest teaches me that things happen while I'm gone — the strongest reason to open it again ("what did it do last night?"). Quick capture ("Jot a note to remember…") invites a tiny daily habit.
|
||||
|
||||
What stops a 5: the appointments are written for engineers, not me — **"Stale workspace check"** sounds like something went moldy, and **"Capability suggestion"** is abstract. If those said "I'll tidy up your project Saturday morning" and "I'll suggest a new trick on Tuesday," I'd be excited instead of puzzled. The "2 pending" items also never tell me what reward awaits if I deal with them.
|
||||
|
||||
### 5. Absence of friction — 2
|
||||
|
||||
The core path (onboarding → home → chat) is mostly plain-spoken, and nothing felt scary — the privacy and "nothing changes without you" lines are calming. But jargon is everywhere I rest my eyes:
|
||||
|
||||
- The always-visible sidebar (02) is half alien: **MCP Hub, Connector Hub, Artifacts, Waggle Dance, Mission Control, Vault, Spawn Agent, Usage & Cost**. I recognize Home, Chat, Files, Settings — the rest is a foreign language I must stare at all day.
|
||||
- Raw formatting symbols leak into the interface: skill descriptions show literal **"## What to do"** and **"**Check session history**"** (04, 04b), and memory cards show **"**Interactions**: 0 … ## Strengths"** (03). To me that looks broken.
|
||||
- The chat header (07) shows a machine code, **"claude-sonnet-4-6"**, next to an unexplained **"Autopilot"** toggle.
|
||||
- The very first screen's subtitle says **"Workspace-native"** (08) — insider speak before I've even clicked once.
|
||||
- Smaller stumbles: "Harvest" (10), "agent · review" (04b), "⚠ 2 pending" (01), "Stale workspace check" (02), "baseline vs winner, which gates fired" (13), "Deprecated" / "Any confidence" filters (03), and "1 agents" (05 — grammar).
|
||||
|
||||
None of it is frightening, but a novice meets an unknown word on nearly every screen, so this can't score above 2.
|
||||
|
||||
## Numbered concrete complaints
|
||||
|
||||
1. **Sidebar jargon overload** (02, all screens, left nav): "MCP Hub", "Connector Hub", "Waggle Dance", "Mission Control", "Vault", "Spawn Agent", "Artifacts" — rename in plain words or hide behind an "Advanced" group; a novice can only parse Home/Chat/Files/Settings.
|
||||
2. **"Harvest" button + raw file path** (10, onboarding step 2): the detected-history banner's action says "Harvest" and shows `C:\Users\MarkoMarkovic\.claude`. Say "Import my history" and demote the path to a tooltip.
|
||||
3. **Unexplained "⚠ 2 pending"** (01, welcome panel header; also workspace cards): a warning triangle with no noun and no link. Say what is pending ("2 things need your OK") and make it clickable.
|
||||
4. **Third-person "User asked:" in "I REMEMBER"** (01): the app calls me "User" in the very list meant to prove it knows me. Should read "You asked me to review recent decisions…".
|
||||
5. **Raw markdown rendered as text** (04 and 04b skill rows; 03 "Monthly Agent Assessment" cards): literal `##` and `**` symbols visible in descriptions. Render the formatting or strip it.
|
||||
6. **The self-built skill is uncelebrated** (04b, presentation-design row): the badge "agent · review" is the entire announcement that my assistant taught itself a new skill. Replace with explicit copy like "Built by your AI — review & approve" and consider surfacing it on Home.
|
||||
7. **Agent Center reads as dead, not learning** (05): "1 agents" (grammar), "avg success —", and Editorial Critic showing "run never / Idle". The only agent looks like it has never worked; seed a first run or hide empty stats.
|
||||
8. **Evolution panel is empty and jargon-gated** (13): great headline copy, but the action is "+ New Run" and the explainer says "baseline vs winner, which gates fired". A novice will never click. Offer "See how I'd improve myself" and translate gates/baseline into plain words.
|
||||
9. **Machine model ID in chat header** (07): "claude-sonnet-4-6" dropdown and unexplained "Autopilot" pill. Hide the model string behind a friendly label ("Smart mode").
|
||||
10. **System-speak schedule items** (02, "Up next"): "Stale workspace check" and "Capability suggestion" — reword as human promises ("I'll tidy up quiet projects", "I'll suggest a new skill").
|
||||
11. **"Workspace-native" on the first screen** (08 subtitle): insider phrase at the single most novice-facing moment; say "Everything organized by project" or drop it.
|
||||
12. **Memory filter jargon** (03): chips like "Deprecated" and a "Any confidence" dropdown, plus tabs "Weaver"/"Harvest" — meaningless to a novice; plain-word alternatives needed.
|
||||
|
||||
## Verdict in one line
|
||||
|
||||
The memory magic is real and visible — I felt greeted, remembered, and pulled back — but the "it keeps getting better" half of the promise is asserted in copy while the screens show empty runs, a never-run agent, and a cryptic badge, all wrapped in more engineer-speak than a novice can comfortably ignore.
|
||||
163
judging/round2/judge-2-casual-professional.md
Normal file
@@ -0,0 +1,163 @@
|
||||
# Judge 2 — Casual Non-Technical Professional (Round 2)
|
||||
|
||||
## Persona
|
||||
|
||||
Marketing manager. I use ChatGPT a few times a week when I remember to. My real life is
|
||||
Outlook, Slack, and PowerPoint. I did not read any documentation. I gave this app one
|
||||
evening to prove it's worth a second evening. Evidence reviewed: 13 full-resolution
|
||||
screenshots (onboarding flow, returning-user home, workspace resume, Memory Center,
|
||||
Skills Hub, Agent Center, Automation Center, command palette, Evolution tab), plus a
|
||||
liveness check against http://localhost:8080.
|
||||
|
||||
## Scores
|
||||
|
||||
| # | Criterion | Score (1–5) |
|
||||
|---|-----------|-------------|
|
||||
| 1 | First-session clarity | **4** |
|
||||
| 2 | "It knows me" feeling | **4** |
|
||||
| 3 | Visible agent growth | **2** |
|
||||
| 4 | Desire to return | **4** |
|
||||
| 5 | Absence of friction | **3** |
|
||||
| | **Total** | **17 / 25** |
|
||||
|
||||
## Per-criterion reasoning
|
||||
|
||||
### 1. First-session clarity — 4
|
||||
|
||||
The good news first: this onboarding is the best part of the product, and it's written in
|
||||
my language. Step 1 ("Tell us who you are", `09-onboarding-who-are-you.png`) asks things
|
||||
I can actually answer — chips like "Marketing", "Draft documents & content", "Remember
|
||||
everything I work on" — and the live preview line ("Good evening, Marko — your work will
|
||||
be remembered here") told me what the product *is* before I ever saw the product. Three
|
||||
steps, a visible "Skip setup" escape hatch, and a privacy promise in plain words ("Your
|
||||
memory and data stay on your device"). After setup, the Home screen
|
||||
(`02-home-cockpit.png`) tells me literally what to do: three "Continue" buttons, one
|
||||
"Suggested next action", a quick-capture box. Five minutes in, I knew the pitch: it
|
||||
remembers my work and picks up where I left off.
|
||||
|
||||
Why not 5: the very first words I read are "YOUR AI OPERATING SYSTEM / Welcome to the
|
||||
Hive / Persistent memory. Workspace-native." (`08-onboarding-welcome.png`). "Operating
|
||||
system," "Hive," and "workspace-native" are insider words — I briefly wondered if this
|
||||
replaces something on my computer. And the moment I land in the app, the left sidebar
|
||||
presents ~20 destinations (Agent Center, Skills Hub, Automation Center, Room, Connector
|
||||
Hub, MCP Hub, Marketplace, Vault, Mission Control, Events & Logs…). I will never click
|
||||
"MCP Hub." I don't know what an MCP is and I'm not going to find out.
|
||||
|
||||
### 2. "It knows me" feeling — 4
|
||||
|
||||
This is the product's strongest muscle and it flexes it everywhere. "Good evening, Marko
|
||||
— 17 memories · 214 people, projects & things it knows across 3 workspaces"
|
||||
(`01-home-welcome-back.png`) with an "I REMEMBER" list that includes an actual *working
|
||||
preference* ("I always work with a draft → critique → rewrite loop. The critique pass is
|
||||
the most important"). The home screen says "You've been away 10 days, Marko. Here's
|
||||
what happened" and each workspace card carries a one-line memory of what I was doing
|
||||
("Q3 editorial direction; lean into skepticism, less hype"). The resume screen
|
||||
(`07-workspace-resume.png`) reconstructs a decision log with dates, rationale, and
|
||||
stakeholders without me asking. That's the promise, delivered visibly. I felt it.
|
||||
|
||||
Why not 5 — a concrete one: I opened Memory Center (`03-memory-center.png`) to see "what
|
||||
it remembers about me," and the first two cards are **"Monthly Agent Assessment —
|
||||
2026-05 / 2026-04"** full of raw, unrendered markdown: `## *Interactions*: 0
|
||||
**Correction Rate**: 0.0% *Improvement Trend*: 0%`. Robot diary entries — about the
|
||||
agent, not about me, showing zeros, with literal `##` and `*` characters on screen — rank
|
||||
*above* the genuinely charming "User's name is Marko Markovic, age 51… favorite color is
|
||||
blue, supports Crvena Zvezda" card. The first shelf of my "memories" is machine
|
||||
self-bookkeeping that looks broken. That one screen took the magic down a notch.
|
||||
|
||||
### 3. Visible agent growth — 2
|
||||
|
||||
The copy promises it; the screens don't show it. The Evolution tab
|
||||
(`13-memory-evolution.png`) has lovely plain-language framing ("Your agent improves
|
||||
itself here… You review each proposal and accept or reject it — nothing changes without
|
||||
you") — and then it's an **empty state**. No runs, nothing proposed, nothing accepted,
|
||||
and the call to action is a "+ New Run" button, which sounds like *I* am supposed to
|
||||
operate the self-improvement machinery. The one real artifact is the "agent · review"
|
||||
badge on the `presentation-design` skill (`04b-skills-agent-badge.png`) — which, if I
|
||||
squint, means "the AI wrote itself a PowerPoint skill" (genuinely exciting for me!). But
|
||||
the badge says only "agent · review" with no story, no "Waggle built this for you from
|
||||
your deck work — take a look." I'd scroll past it. My one agent, Editorial Critic
|
||||
(`05-agent-center.png`), shows "run never" and "avg success —". The overnight stats ("9
|
||||
memories consolidated, 5 automations completed") read as system maintenance, not as "it
|
||||
got better at MY job." Verdict: growth is asserted, not demonstrated. I could not tell
|
||||
it's learning *for me specifically*.
|
||||
|
||||
### 4. Desire to return — 4
|
||||
|
||||
Honestly? Yes, I'd open it tomorrow — to see if the morning briefing trick works twice.
|
||||
The return loop is well designed: it works overnight (Automation Center shows a 3:05 AM
|
||||
morning briefing run that succeeded, `06-automation-center.png`), it greets me with what
|
||||
changed, and "Continue" means I never pay the restart tax that makes me abandon ChatGPT
|
||||
threads. The Editorial Critic concept — an agent that critiques every draft against my
|
||||
agreed editorial direction — is exactly the kind of thing my job needs. "Up next:
|
||||
Capability suggestion at Jun 17" even teases a reason to come back on a specific day.
|
||||
|
||||
Why not 5: my actual work lives in email, Slack, and PowerPoint, and nothing on the Home
|
||||
screen connects to any of them. The habit only forms if I move my work *into* Waggle,
|
||||
and after one session I haven't been given a reason to do that migration. The pull is
|
||||
real but it's pulling against gravity.
|
||||
|
||||
### 5. Absence of friction — 3
|
||||
|
||||
Nothing made me want to slam the laptop shut, but several things made me sigh:
|
||||
|
||||
- The memory-import step (`10-onboarding-memory-import.png`) auto-detected **Claude Code**
|
||||
(425 items) — a developer tool I've never opened. For *my* AI history (ChatGPT), the
|
||||
instruction is: "Settings → Data controls → Export data. You'll get an email with the
|
||||
file." Leave the app, do an export, wait for an email, download a JSON, come back,
|
||||
upload. That is homework, on step 2 of 3, during the first run. It's skippable
|
||||
(good), but the headline feature of onboarding only auto-works for developers.
|
||||
- Raw markdown leaks everywhere a description appears: every row in Skills Hub
|
||||
(`04-skills-hub.png`) shows fragments like "## What to do 1. *Identify the decision* —
|
||||
Clarif…". It reads as unfinished software.
|
||||
- The Win+K palette (`12-command-center.png`) is slash-commands — `/catchup`, `/spawn`,
|
||||
`/decide` — every row labeled "Command". The placeholder asks "What do you want to
|
||||
do?" in my language and then answers exclusively in developer.
|
||||
- Jargon tax: Hive, Harvest, Weaver, Frames, Vault, MCP Hub, "memory compaction",
|
||||
"Memory lane extraction at 4:00 AM". I understand none of these and the UI doesn't
|
||||
explain them.
|
||||
|
||||
None of this is fatal — the core paths (onboard, resume, chat) are smooth — hence a 3,
|
||||
not lower.
|
||||
|
||||
## Numbered complaints (concrete & actionable)
|
||||
|
||||
1. **Onboarding step 2, ChatGPT card** (`10-onboarding-memory-import.png`): the only
|
||||
path for a ChatGPT user is a manual export-and-wait-for-email errand outside the app.
|
||||
Auto-detection worked only for Claude Code. Either make ChatGPT import painless or
|
||||
move this ask to after first value, not step 2 of 3.
|
||||
2. **Memory Center "About you" ordering + rendering** (`03-memory-center.png`): two
|
||||
"Monthly Agent Assessment" cards with raw `## *Interactions*: 0 … *Correction Rate*:
|
||||
0.0%` markdown rank above the actual about-me card. Render markdown and demote agent
|
||||
self-assessments out of the default human-facing view.
|
||||
3. **Skills Hub descriptions are raw skill-body fragments** (`04-skills-hub.png`): every
|
||||
row shows "## What to do 1. *…*" with literal markdown symbols. Each skill needs a
|
||||
one-line human description.
|
||||
4. **"agent · review" badge is unexplained** (`04b-skills-agent-badge.png`): the single
|
||||
on-screen proof of self-evolution has no plain-language story or tooltip. Say "Waggle
|
||||
created this skill for you — review it" or the moment is lost on a novice.
|
||||
5. **Evolution tab is an empty state with a dev-flavored CTA** (`13-memory-evolution.png`):
|
||||
"Your agent improves itself here" followed by no runs and a "+ New Run" button puts
|
||||
the burden of self-improvement on me. Seed it with a first proposal or hide it until
|
||||
one exists.
|
||||
6. **Editorial Critic has never run** (`05-agent-center.png`): "run never", "avg success
|
||||
—", "0 running". My one agent is inert on the screen meant to showcase agents.
|
||||
7. **Slash-command-only palette** (`12-command-center.png`): `/catchup`, `/spawn`,
|
||||
`/skills`, all labeled "Command" — a developer idiom presented to someone who asked
|
||||
"What do you want to do?" Plain-verb entries ("Catch me up", "Start a draft") should
|
||||
lead.
|
||||
8. **Sidebar overload + jargon naming** (`02-home-cockpit.png` left rail): ~20
|
||||
destinations including MCP Hub, Vault, Weaver, Mission Control on first arrival.
|
||||
A casual professional needs 5; tuck the rest behind "More" or a pro mode.
|
||||
|
||||
## Bottom line
|
||||
|
||||
The memory half of the promise is real and I felt it — the greeting, the
|
||||
"away 10 days" recap, and the restored decision log are the best "it remembers me"
|
||||
experience I've seen in an AI tool, and the onboarding that sets it up is genuinely
|
||||
novice-friendly. The self-evolving half is currently a narrated promise: empty Evolution
|
||||
screen, an unexplained badge, an agent that has never run. And the finish (raw markdown
|
||||
in user-facing text, robot bookkeeping atop my memories, slash-command palette) keeps
|
||||
whispering "built by developers, for developers" at exactly the moments the product is
|
||||
trying to convince me otherwise.
|
||||
|
||||
**Scores: 4 / 4 / 2 / 4 / 3 — total 17/25. 8 complaints.**
|
||||
75
judging/round2/judge-3-power-user.md
Normal file
@@ -0,0 +1,75 @@
|
||||
# Judge 3 — Non-Developer Power User (Ops Lead)
|
||||
|
||||
**Persona:** Operations lead who lives in Notion, Airtable, and Zapier. I don't code. I build automations, learn every keyboard shortcut, open every menu and tab on day one, and I judge a tool on whether going deep is rewarded — or whether the second layer is hollow. I notice when numbers don't reconcile across screens, because in my world a dashboard that contradicts itself is a dashboard I stop trusting.
|
||||
|
||||
**Evidence reviewed:** All 14 screenshots (read at full resolution via crops), plus live API verification against the running sidecar (`/api/home/briefing`, `/api/home/overnight`, `/api/skills`, `/api/agents`, `/api/automations`, `/api/memory/stats`).
|
||||
|
||||
---
|
||||
|
||||
## Scores
|
||||
|
||||
| # | Criterion | Score |
|
||||
|---|-----------|:-----:|
|
||||
| 1 | First-session clarity | **4** |
|
||||
| 2 | "It knows me" feeling | **4** |
|
||||
| 3 | Visible agent growth | **3** |
|
||||
| 4 | Desire to return | **4** |
|
||||
| 5 | Absence of friction | **2** |
|
||||
|
||||
**Total: 17/25 · 10 concrete complaints**
|
||||
|
||||
---
|
||||
|
||||
## Per-Criterion Reasoning
|
||||
|
||||
### 1. First-session clarity — 4/5
|
||||
|
||||
The onboarding is the best three-step flow I've seen in this category. Plain-language questions ("What do you want Waggle to help with?" with chips like *Automate repetitive work* — that's me), a live greeting preview that updates as I type my name ("Good evening, Marko — your work will be remembered here."), and the killer moment: **Step 2 auto-detected Claude Code with "Found 425 items at C:\Users\MarkoMarkovic\.claude" and a one-click Harvest button.** Per-source export how-tos for ChatGPT/Claude/Gemini/Perplexity (exact menu paths: "Settings → Data controls → Export data") are written for someone exactly like me. "Each workspace is its own brain" is the right one-line mental model. The privacy line on screen one ("Your memory and data stay on your device") earns trust immediately.
|
||||
|
||||
Why not 5: the sidebar I land in afterward has ~20 items, and a chunk of them are jargon a non-developer cannot parse from the label alone — **"MCP Hub", "Waggle Dance", "Weaver", "Room"** mean nothing on first read. And the Memory Center splits into **seven tabs** (Memories / Timeline / Graph / Harvest / Weaver / Wiki / Evolution) with no hint about which one I should care about first. The first session is clear; the first *deep dive* requires guessing.
|
||||
|
||||
### 2. "It knows me" feeling — 4/5
|
||||
|
||||
This is where the product is closest to its promise. "You've been away 10 days, Marko. Here's what happened:" with the actual date is exactly the greeting the mission describes. The welcome panel's **"I REMEMBER" section quoting a learned working preference back to me — "I always work with a draft → critique → rewrite loop. The critique pass is the most important" — is the single most convincing moment in the app.** The workspace resume (07) is genuinely excellent for an ops brain: a structured decision log ("DECISION 1: Q3 Editorial Pivot — 'Skepticism Over Hype', Approved by Marko (founder) on 2026-04-12", stakeholders with their authorities listed), plus a one-line workspace summary on the Home card ("Q3 editorial direction: lean into skepticism, less hype"). The memory card knowing my age, employer, and that I support Crvena Zvezda is the party trick that sells the demo.
|
||||
|
||||
Why not 5 — the numbers betray the magic. The modal header says **"17 memories … across 3 workspaces"**, but the Default Workspace card directly beneath it says **"11 memories"** and the Writer demo side panel (07) also says **"11 memories"** — 11+11+0 ≠ 17, and `/api/memory/stats` says 14 personal frames. A tool that claims to remember everything must not contradict itself about how much it remembers. Also, the welcome modal puts **Default Workspace (stale, last touched ~2 months ago) at the top** while Writer demo — the workspace with a pending item and a real summary — is collapsed at the bottom; the Home grid behind it sorts by recency. Two greeting surfaces, seconds apart, disagree about what I should care about.
|
||||
|
||||
### 3. Visible agent growth — 3/5
|
||||
|
||||
There IS real, verifiable growth evidence — I checked. The `presentation-design` skill carries an **"agent · review" provenance badge**, and the API confirms it (`"initiator": "agent", "source": "chat-session"`): the system wrote itself a skill and is honestly flagging it for my review. Workspace activity says "Created a skill · 10d ago". The Automation Center is the strongest power surface: 13 automations, 12 active schedules, **100% success rate with timestamped recent results** (Morning briefing OK 3:05:58 AM) and named next runs. Overnight: "9 memories consolidated".
|
||||
|
||||
But the marquee surface is hollow. **The Evolution tab — literally titled "Your agent improves itself here" — is completely empty: zero proposals across all six filter states (all/proposed/accepted/deployed/rejected/failed)**, even though "Prompt optimization" and "Monthly assessment" automations supposedly run. The Agent Center has exactly **one agent that has never run** ("Run never", "avg success —"). And the growth evidence that *does* surface in Memory is two "Monthly Agent Assessment" cards reporting **"Interactions: 0, Correction Rate: 0.0%"** — the system showing me a report card full of zeroes. I can see the *machinery* of self-evolution everywhere; I can only see one actual instance of it (the skill badge). Telling ≠ showing.
|
||||
|
||||
### 4. Desire to return — 4/5
|
||||
|
||||
The loop is real and it's built the way a Zapier user wants it: overnight digest with numbers (9 consolidated / 5 automations completed), an **"Up next" section with concrete dated items** ("Stale workspace check at Jun 15, 9:00 AM"), a suggested next action that deep-links into the right workspace ("Resume: Review recent decisions and next steps" — verified in the API as a per-workspace `next-action`), pending-count badges, and Quick capture (Note/Task/Link/File) so the cost of dumping a thought is near zero. `/catchup` being the top "Suggested for you" item in the Ctrl+K palette — with palette commands described in plain outcomes, not dev-speak — is a genuine fast path; the palette is discoverable via a visible "Ctrl+K" chip in the header. Depth is starting to be rewarded.
|
||||
|
||||
Why not 5: the overnight story is currently **housekeeping, not work product** — "0 Artifacts created", and the digest items (Memory compaction, Harvest sync, Index reconciliation) are the system doing chores on itself. "Up next" is likewise system maintenance ("Capability suggestion") framed as my agenda. I come back to Notion because something *for me* changed overnight; here, the agent mostly tidied its own room.
|
||||
|
||||
### 5. Absence of friction — 2/5
|
||||
|
||||
Nothing crashed, no dead-end navigation, every screen rendered, every API answered — but I open every menu, and nearly every menu had a rough edge. Four flagship surfaces are substantially empty at 1440×900, raw markdown leaks into user-facing text, and the numbers don't reconcile. Itemized below.
|
||||
|
||||
---
|
||||
|
||||
## Numbered Complaints (all concrete)
|
||||
|
||||
1. **Home cockpit has a dead column.** On 02-home-cockpit.png there is a vertical divider at ~x=1170 with a completely empty rail (~270px, ~19% of the viewport) to its right — nothing renders in it at all. That's prime real estate on the single most important screen, blank.
|
||||
2. **The Evolution tab is an empty promise.** 13-memory-evolution.png: "Your agent improves itself here" + six filter chips + zero items in any state. The flagship "self-evolving" surface, on an account where evolution-adjacent automations (Prompt optimization, Monthly assessment) demonstrably run, shows nothing to review, accept, or reject.
|
||||
3. **Machine exhaust pollutes Memory, ranked above the good stuff.** 03-memory-center.png: the top two memory cards are "Monthly Agent Assessment — 2026-05 / 2026-04" containing unrendered template markdown with all-zero stats ("# Monthly Agent Assessment … \*Interactions\*: 0 \*Correction Rate\*: 0.0% … ## Weaknesses ## Capability Gaps - None detecte…"), tagged FACT — while the genuinely personal "User's name is Marko Markovic, age 51…" card sits below them.
|
||||
4. **Skills Hub descriptions leak raw markdown with mid-word truncation.** 04-skills-hub.png: list rows read "## What to do 1 \*\*"Identify the decision"\*\* — Clarif", "— Pull fro", "## When to use — Usa". Twenty skills and nearly every description line is a broken markdown fragment instead of a sentence.
|
||||
5. **Memory counts don't reconcile across three surfaces.** Welcome modal header: "17 memories … across 3 workspaces". Default Workspace card in the same modal: "11 memories across 16 sessions". Writer demo side panel (07): "11 memories". `/api/memory/stats`: 14 personal frames. Four numbers, no arithmetic that connects them.
|
||||
6. **The two greeting surfaces disagree on priority.** The welcome-back modal (01) lists Default Workspace (last active April, ~2 months stale) first and Writer demo (16d, 1 pending, has a summary) last; the Home grid behind it (02) sorts by recency. Same moment, contradictory ordering.
|
||||
7. **Agent Center is a near-empty shell with a grammar bug.** 05-agent-center.png: header reads "1 agents · 0 running · avg success —"; the sole agent shows "Run never" and status Idle. Five tabs (All/Personal/Workspace/Team/Autonomous/Archive) over one never-executed item — depth is not yet rewarded here.
|
||||
8. **Overnight digest reports chores, not output.** 02-home-cockpit.png: "0 Artifacts created"; Next up = Memory compaction, Memory lane extraction, Harvest sync; Up next = "Stale workspace check", "Capability suggestion". The system's self-maintenance is presented as my morning briefing.
|
||||
9. **Unexplained jargon in primary navigation.** Sidebar labels "MCP Hub", "Waggle Dance", "Weaver", "Room" carry no tooltip-visible plain-language meaning for a non-developer, and Memory Center's 7 tabs (Memories/Timeline/Graph/Harvest/Weaver/Wiki/Evolution) overlap conceptually with no guidance on which to use when.
|
||||
10. **Center screens don't use the canvas.** Memory Center (03), Agent Center (05), and Automation Center (06) all leave the bottom ~50–60% of a 1440×900 window as bare honeycomb wallpaper; Memory Center shows only ~4 cards above the fold of an empty sea.
|
||||
|
||||
## What earned the points (for balance)
|
||||
|
||||
- Onboarding auto-detect ("Claude Code detected — Found 425 items") with one-click Harvest: best-in-class first-run moment.
|
||||
- "I REMEMBER" quoting my learned draft→critique→rewrite preference back to me.
|
||||
- Decision history with approver, date, rationale, and stakeholders in workspace resume — ops-grade.
|
||||
- "agent · review" provenance badge, verified real in the API (`initiator: "agent"`).
|
||||
- Automation Center's 100% success rate with timestamped runs and named next runs.
|
||||
- Ctrl+K palette with contextual "Suggested for you" `/catchup` and plain-language command descriptions.
|
||||
66
judging/round2/judge-4-junior-developer.md
Normal file
@@ -0,0 +1,66 @@
|
||||
# Judge 4 — Junior Developer Verdict (Round 2)
|
||||
|
||||
## Persona summary
|
||||
|
||||
Two years into the job. I live in VS Code, lean on Copilot all day, keep a ChatGPT tab pinned, and I will absolutely click every button in your app within ten minutes. I have opinions about Raycast's command palette and Linear's empty states, and I judge new tools against that bar. AI agents are the thing I'm most curious about right now — I want to see one actually do something, not read a card telling me it will.
|
||||
|
||||
Evaluated from 14 screenshots (1440x900, cropped/zoomed with PIL where needed) plus live verification against the running app at `localhost:8080` (session-token auth, then `/api/workspaces`, `/api/home/briefing`, `/api/memory/stats`, `/api/agents`, `/api/skills`, `/api/automations`, `/api/harvest/sources`).
|
||||
|
||||
## Scores
|
||||
|
||||
| # | Criterion | Score (1-5) |
|
||||
|---|-----------|-------------|
|
||||
| 1 | First-session clarity | **4** |
|
||||
| 2 | "It knows me" feeling | **4** |
|
||||
| 3 | Visible agent growth | **3** |
|
||||
| 4 | Desire to return | **4** |
|
||||
| 5 | Absence of friction | **3** |
|
||||
|
||||
**Total: 18/25 — Complaints: 10**
|
||||
|
||||
## Per-criterion reasoning
|
||||
|
||||
### 1. First-session clarity — 4
|
||||
|
||||
The onboarding is the best three steps in the product. "Each workspace is its own brain — memory, files, and agents stay isolated" (11-onboarding-workspace.png) teaches the core mental model in one sentence — better than most docs pages I've read. Step 1's live greeting preview ("Good evening, Marko — your work will be remembered here") updates as you type your name, which makes the memory promise concrete before you've even entered the app. Step 2 auto-detecting Claude Code ("Found 425 items at C:\Users\MarkoMarkovic\.claude") with a one-click Harvest button is the single biggest "whoa" moment for a developer — it found my actual workflow without me telling it anything. The privacy line on the welcome screen ("Your memory and data stay on your device") answers my first question unprompted.
|
||||
|
||||
But two of the three pillars get taught and one doesn't: workspaces and memory are explained; the self-evolving agent is never introduced. Nothing in steps 1-3 prepares you for Evolution, Agent Center, or the agent-review concept. And the moment onboarding ends, you land in a sidebar with roughly 20 items including unexplained jargon — "Room", "Waggle Dance", "Mission Control" (02-home-cockpit.png). I can navigate that because I navigate IDEs all day; the "especially non-technical users" in the mission statement cannot.
|
||||
|
||||
### 2. "It knows me" feeling — 4
|
||||
|
||||
This is the product's strongest muscle and most of it is real, not staged. "You've been away 10 days, Marko. Here's what happened:" (02) is exactly what I want from a tool I left running. The welcome modal (01) stacks specifics: "17 memories · 214 people, projects & things it knows across 3 workspaces" (the 214 matches `/api/memory/stats` entityCount exactly — I checked), an "I REMEMBER" section with an actual learned working preference ("I always work with a draft → critique → rewrite loop. The critique pass is the most important"), and per-workspace resume lines naming the last session topic. The workspace resume (07) is the payoff: the agent compiles "Decision Review & Next Steps — Anya's Content Strategy" from workspace memory with rendered markdown, dates, and stakeholders ("Marko — Approved the direction; has final say on brand positioning"). The memory card that knows I'm 51, work at Egzakta Group, and support Crvena Zvezda is the kind of detail that makes the greeting feel earned.
|
||||
|
||||
Two things stop the 5. First, the welcome modal misspells the workspace name — "Writer demo — Anua" (01) — while the cockpit behind it (02), the workspace panel (07), and the live API all say "Anya". A memory product that misremembers a name in its flagship "I remember you" surface undermines the exact feeling it's selling. Second, when you click through to Memory Center to see this famous memory, you get three cards, two of which are robot-generated assessment reports — the memory feels deep in the greeting and thin at the source.
|
||||
|
||||
### 3. Visible agent growth — 3
|
||||
|
||||
There is one genuinely excellent, verified artifact here: the `presentation-design` skill in Skills Hub carries an "agent · review" badge (04b), and the API confirms it is not paint — the skill record has `initiator: "agent"`, `source: "chat-session"`. An agent authored a skill, the system tracked provenance, and the UI gates it behind review. That is the self-evolution loop, real, end to end. The workspace panel's "LAST ACTIVITY: Created a skill — 10d ago" (07) reinforces it. The Editorial Critic agent (05) is also real (verified via `/api/agents`) and its goal is wired to remembered context — "apply the Q3 skepticism-over-hype editorial direction before anything ships" is literally Decision 1 from the workspace's memory. That memory-to-agent-config loop is visible and credible.
|
||||
|
||||
But everything else is scaffolding. The Evolution tab (13) — the marquee "agent improves itself" surface — is an empty state: zero runs, "Select a run to review", a "New Run" button, and explainer copy. Agent Center shows one agent, "0 running", "avg success —", never executed; the API shows it was created the same day as this evaluation and has no run history. All 13 automations (verified) are system-shipped maintenance jobs — memory compaction, harvest sync, marketplace sync — none learned from my workflows. The growth story today is one authored skill and a stack of promises. That's a real seed, not a visible garden.
|
||||
|
||||
### 4. Desire to return — 4
|
||||
|
||||
Honestly? Yes, I'd keep it running for a while, and that surprised me. The away-briefing loop (leave → come back → "here's what happened" → one-click Continue into the exact workspace with pending items flagged) is something neither Copilot nor ChatGPT does, and it's implemented, not mocked — `/api/home/briefing` returns the greeting, suggested next action, and per-workspace pending counts I saw on screen. The Claude Code harvest means it accumulates value from work I'm already doing. The Win+K palette (12) with `/catchup`, `/research`, `/spawn`, `/skills` is Raycast-literate and made me feel at home immediately; "/catchup — Workspace restart summary — get up to speed instantly" as the top suggestion is exactly the right default.
|
||||
|
||||
What stops the 5: the overnight report says "9 Memories consolidated, 0 Artifacts created, 5 Automations completed" (02). Nine consolidated memories is housekeeping; zero artifacts means the agent layer produced nothing for me while I was gone, and the one agent that could have (Editorial Critic) has never run. The return habit this app wants to build is "come back to finished work" — right now it's "come back to a well-organized summary of nothing having been done." I'd return daily for two weeks on the briefing alone; whether week three survives depends on that artifacts number going above zero.
|
||||
|
||||
### 5. Absence of friction — 3
|
||||
|
||||
No crashes, no broken layouts, navigation is coherent, and the visual identity (honey-on-dark hex theme) is consistent and genuinely attractive. But the rough edges are pervasive once you leave the happy path, and several are on flagship screens — see the numbered list. The two worst: raw markdown leaking as literal `#`/`##`/`**` text across both Memory Center cards and every Skills Hub row (this is table-stakes rendering, and the chat view proves the app can render markdown beautifully), and the Anua/Anya name inconsistency on the welcome modal. Add zero-data auto-generated memories polluting "About you" and three major screens that are 70-90% empty hexagon wallpaper, and the polish gap against the Linear/Raycast tier I compare everything to is clearly visible.
|
||||
|
||||
## Numbered complaints
|
||||
|
||||
1. **Workspace name misspelled in welcome modal** — 01-home-welcome-back.png shows "Writer demo — Anua"; the Home cockpit (02), the workspace side panel (07), and `GET /api/workspaces` all say "Writer demo — Anya". The memory product misremembers a name on its "I remember you" surface.
|
||||
2. **Raw markdown rendered literally in Memory Center cards** — 03-memory-center.png: "# Monthly Agent Assessment — 2026-05", "**Interactions**: 0", "## Strengths - Low correction rate" displayed with literal hashes and asterisks instead of formatted text.
|
||||
3. **Raw markdown in every Skills Hub row** — 04/04b: skill descriptions render as "# Brainstorm — ... ## What to do Run three" and "## Steps 1. Analyze audience needs and pr..." — every list row leaks frontmatter-style source instead of a clean one-line summary.
|
||||
4. **Auto-generated noise crowds the "About you" memory** — 03: two of the three visible memories are "Monthly Agent Assessment" reports with Interactions: 0, Correction Rate: 0.0%, Improvement Trend: 0% — zero-data system output presented as things it "knows about me".
|
||||
5. **Evolution tab has zero evidence of evolution** — 13-memory-evolution.png: the headline self-improvement surface is an empty state ("Select a run to review", no runs, "New Run" button). The superpower is an explainer card.
|
||||
6. **Agent Center is one row and a void** — 05-agent-center.png: 1 agent, "0 running", "avg success —", never executed (API confirms no run history); ~85% of the screen is decorative hex background.
|
||||
7. **Overnight digest reports "0 Artifacts created"** — 02-home-cockpit.png: the come-back-to-finished-work promise returns memory housekeeping (9 consolidated) and nothing produced.
|
||||
8. **Onboarding never teaches the second superpower** — 08-11: identity, memory import, and workspace creation are covered; self-evolving agents, skill authorship, and the review gate are never introduced before the user encounters "agent · review" badges and the Evolution tab.
|
||||
9. **Sidebar overload with unexplained jargon** — 02: ~20 nav items across 4 sections including "Room", "Waggle Dance", and "Mission Control" with no visible explanation — fine for me, hostile to the non-technical users in the mission.
|
||||
10. **Welcome screen caption wraps awkwardly** — 08-onboarding-welcome.png: "Nothing leaves without your say-so." breaks as "say-" / "so." across two lines, a sloppy first impression on an otherwise immaculate first screen.
|
||||
|
||||
## Bottom line
|
||||
|
||||
The memory pillar is real and verified — greeting, briefing, resume, and provenance all check out against the live API. The evolution pillar has exactly one true artifact (the agent-authored skill with its review badge) surrounded by empty stages waiting for a performance. I'd run it next to VS Code this month. Whether it stays depends on the agents earning their tab.
|
||||
119
judging/round2/judge-5-senior-skeptic.md
Normal file
@@ -0,0 +1,119 @@
|
||||
# Judge 5 — Senior Engineer / Professional Skeptic
|
||||
|
||||
**Persona:** 15 years shipping products. I assume "AI that learns" is inflated until the UI or API proves it. I read every screenshot at full resolution, then pulled a Bearer token and audited the live sidecar APIs (`/api/home/briefing`, `/api/home/overnight`, `/api/memory`, `/api/memory/stats`, `/api/identity`, `/api/skills`, `/api/skills/presentation-design`, `/api/agents`, `/api/automations`, `/api/evolution/runs`, `/api/audit/installs`) and cross-checked against `apps/web/src/lib/briefing-highlights.ts`, `login-briefing-brag.ts`, `LoginBriefing.tsx`, and `packages/server/src/local/routes/home.ts`. Credit is given below where the evidence is real. It often is. That makes the staged parts stand out more, not less.
|
||||
|
||||
## Scores
|
||||
|
||||
| # | Criterion | Score (1–5) |
|
||||
|---|-----------|-------------|
|
||||
| 1 | First-session clarity | **4** |
|
||||
| 2 | "It knows me" feeling | **3** |
|
||||
| 3 | Visible agent growth | **2** |
|
||||
| 4 | Desire to return | **3** |
|
||||
| 5 | Absence of friction | **2** |
|
||||
|
||||
**Total: 14/25 · 12 numbered complaints**
|
||||
|
||||
---
|
||||
|
||||
## Per-criterion reasoning
|
||||
|
||||
### 1. First-session clarity — 4/5
|
||||
|
||||
The onboarding (08–11) is the strongest surface in the product, and the mental model it sells is honest:
|
||||
|
||||
- 3 steps, progress dots, `Skip setup` and `Skip this step` on every screen. No hostage-taking.
|
||||
- Step 1 shows a live preview ("Good evening, Marko — your work will be remembered here") that the product actually delivers later — verified, the returning-user greeting matches.
|
||||
- Step 2 ("Where do you use AI today?") performed a **real detection**: "Claude Code detected — Found 425 items at C:\Users\MarkoMarkovic\.claude". That's not a mock; that path exists on this machine. The 5 import cards (ChatGPT/Claude/Gemini/Perplexity/Other) include the actual export instructions per vendor.
|
||||
- Step 3's framing — "Each workspace is its own brain — memory, files, and agents stay isolated" — is corroborated by the API: `/api/memory/stats?workspaceId=…` returns genuinely separate per-workspace minds (default-workspace: 11 frames/54 entities/312 relations; new-hive: 0/0/0).
|
||||
- The privacy claim ("Your memory and data stay on your device") is at least architecturally consistent with a localhost sidecar.
|
||||
- Win+K palette (12) with `/catchup`, `/decide`, `/review` etc. is discoverable and plainly described.
|
||||
|
||||
Why not 5: the first thing a returning user sees (the welcome panel) contains numbers that don't reconcile with each other or the API (complaint 3), and the cockpit invites you to "Continue" a workspace that the same panel says is empty (complaint 8). The model is graspable; the first screen's arithmetic isn't.
|
||||
|
||||
### 2. "It knows me" feeling — 3/5
|
||||
|
||||
The machinery is real. The lived evidence is half genuine, half staged, and the hero presentation shoots itself in the foot.
|
||||
|
||||
**Real (verified):**
|
||||
- `/api/identity`: configured 2026-04-16, name/role/department persisted, updated today. The "214 people, projects & things it knows" headline equals the personal `entityCount` **exactly** (214) — that's real harvested knowledge-graph data, not a vanity number.
|
||||
- Memory Center (03) statuses are not decoration: the store contains frames in `active`, `archived`, and `deprecated` states, and the transitions actually happened (frame 39, a junk "User preference" misclassification, was deprecated on 06-11 — the correction loop works).
|
||||
- Workspace memories for writer-demo-anya are substantive: brand-voice rules, an editorial decision with approver and date, newsletter metrics with `tool_verified` source. The chat resume (07) renders a decision record consistent with those frames.
|
||||
|
||||
**Not earned:**
|
||||
- Two of the three flagship "I REMEMBER" items on the welcome panel (01) are frames the system itself has **deprecated** (writer-demo frames 9 and 10, status=deprecated via API), and one of them is junk ("User asked: Review recent decisions and next steps" — a logged query, not a memory). The highlight ranker (`briefing-highlights.ts`) has no status field at all. The product leads with memories it has disowned.
|
||||
- "You've been away 10 days, Marko" is contradicted by its own store: personal frames written 2026-06-11 13:47 and 14:17, and an agent created 2026-06-12T18:59 — 42 minutes before the briefing timestamp (19:41Z).
|
||||
- The richest memories live in a workspace literally named "Writer demo — Anya", and all 11 of its seed frames were created in the **same second** (2026-05-27 23:49:49). Staged.
|
||||
- The organically-grown personal mind is 14 frames, of which 10 are duplicate zero-data self-assessments (see criterion 3).
|
||||
|
||||
A 3: persistent, user-visible, correctable — proven. "It knows me" as a lived feeling — propped up by seeded demo data and undermined by the deprecated-highlights bug.
|
||||
|
||||
### 3. Visible agent growth — 2/5
|
||||
|
||||
The mission promises a self-evolving agent. Here is the full inventory of growth evidence on this install:
|
||||
|
||||
**Real (credit where due):**
|
||||
- The `presentation-design` skill is genuinely agent-authored: frontmatter reads `initiator: agent`, `source: chat-session` (verified via `/api/skills/presentation-design`), and the Skills Hub renders an "agent · review" provenance badge (04b). One real, traceable, agent-created artifact. This is the single best piece of evidence in the product.
|
||||
- `/api/audit/installs` is a real governance trail with risk/trust/approval taxonomy, and 4 of its 8 entries are **agent-initiated** capability proposals (`initiator: "agent"`, `action: "proposed"` — filesystem ×3, github connector). The agent demonstrably asks for capabilities.
|
||||
|
||||
**Vapor:**
|
||||
- The Evolution screen (13) — the flagship self-evolution surface, copy: "Your agent improves itself here… nothing changes without you" — is an empty state. `/api/evolution/runs` → `{"runs":[],"count":0}`. Zero runs, ever, in a store whose data goes back to April.
|
||||
- The Agent Center's only agent ("Editorial Critic") was created by the **user** (`createdBy: "user"`) at 2026-06-12T18:59 — minutes before judging — and has never run: status idle, runs never, avg success "—".
|
||||
- The "Monthly Agent Assessment" memories are self-evaluation theater: every copy reads `Interactions: 0, Correction Rate: 0.0%`, and concludes "Strengths: Low correction rate". An agent grading itself A+ on a test it never sat. There are **ten duplicate copies** of this in a 14-frame personal mind.
|
||||
- None of the agent's 4 capability proposals were ever approved or installed; the loop has never closed.
|
||||
- "Shares knowledge across workspaces": no evidence found on any screen or endpoint.
|
||||
|
||||
Mechanism exists; growth has not happened. One real artifact keeps this off the floor: 2.
|
||||
|
||||
### 4. Desire to return — 3/5
|
||||
|
||||
The retention loop is engineered on the right axis — value, not dark patterns — but the value delivered is thin and partly self-referential.
|
||||
|
||||
**Real (verified):**
|
||||
- "Overnight: 9 memories consolidated / 0 artifacts created / 5 automations completed" is computed from a real audit-event store (`home.ts` `readAuditCounts`, `memory_write` + file-write `tool_call` events over a 24h window). I reconciled the "5 automations": exactly 5 schedules have `lastRun` inside the window (Harvest sync, Memory compaction, Memory consolidation, Morning briefing, Task reminder). The honest "0 artifacts created" — displaying a zero rather than hiding it — is to this product's credit.
|
||||
- The suggested action ("Resume: Review recent decisions and next steps") deep-links to a real pending task in writer-demo-anya (`pendingCount: 1` in the briefing API), and quick-capture is one keystroke away.
|
||||
- No dark patterns anywhere: "Don't show again" on the welcome panel, skips throughout onboarding, autonomy is opt-in ("guided").
|
||||
|
||||
**Thin:**
|
||||
- All 5 "overnight" completions fired in a single burst at 01:05:57–58Z (3:05 AM local, same second) — a catch-up burst, not a humming overnight workforce. And what did the night shift produce? Memory compaction and consolidation whose visible output is… another duplicate zero-data Monthly Assessment frame (id 42, written 06-12 18:36). "9 memories consolidated" is a raw count of `memory_write` events, several of which were the agent re-writing its own junk.
|
||||
- The thing that would actually pull a user back — an artifact, a drafted newsletter, a completed task — is exactly the number the panel honestly reports: 0.
|
||||
|
||||
Honest loop, weak payload: 3.
|
||||
|
||||
### 5. Absence of friction — 2/5
|
||||
|
||||
For roughly one hour of adversarial inspection across 9 screens and 11 endpoints, I logged 12 concrete defects (below), including same-screen numeric contradictions, a false hero greeting, past-due "next up" schedules under a 100% success banner, and one flaky API response. Each one is small; together they are exactly the credibility tax a memory product cannot afford. 2.
|
||||
|
||||
---
|
||||
|
||||
## Numbered complaints (all concrete, all actionable)
|
||||
|
||||
1. **Welcome panel showcases deprecated memories.** 01-home-welcome-back.png: 2 of 3 "I REMEMBER" highlights are writer-demo-anya frames 9 & 10, both `status: "deprecated"` (verified via `GET /api/memory?workspace=writer-demo-anya`); one is junk ("User asked: Review recent decisions and next steps"). Root cause: `apps/web/src/lib/briefing-highlights.ts` — `BriefingFrameLike` has no `status` field and `selectBriefingHighlights()` never filters; `LoginBriefing.tsx:128` feeds it a canned `searchMemory('important decision project plan', 'global')`. Filter `status === 'active'` before ranking.
|
||||
|
||||
2. **"You've been away 10 days, Marko" is false.** The store shows personal frames written 2026-06-11 13:47:56 and 14:17:58 (ids 39, 40), and the Editorial Critic agent created 2026-06-12T18:59:26 — 42 minutes before the briefing timestamp (2026-06-12T19:41:46Z). The greeting derives only from workspace chat `lastActive`. Either compute away-time from max(any activity) or say "last chat 10 days ago".
|
||||
|
||||
3. **Headline memory count doesn't reconcile with anything.** Welcome panel says "17 memories … across 3 workspaces" while its own cards show 11 (Default) + 0 (New Hive) (+11 writer-demo). `/api/memory/stats` gives personal=14, +default=25, all minds=36 — no combination yields 17. One screen, three mutually inconsistent numbers.
|
||||
|
||||
4. **Personal memory is 71% duplicate junk.** 10 of 14 personal frames are copies of "Monthly Agent Assessment" (5× 2026-04, 5× 2026-05; created 05-02 through 06-12), each reading `Interactions: 0 / Correction Rate: 0.0% / Strengths: Low correction rate`. The assessment automation re-writes duplicates on every run and grades itself on zero data. Two of these render as the top cards in Memory Center (03-memory-center.png).
|
||||
|
||||
5. **Provenance mislabeled.** Every automation-generated assessment frame carries `source: "user_stated"`. The user never stated them. This corrupts the exact trust signal the Memory Center's filter UI sells (the seeded demo data, ironically, gets it right with `tool_verified` on metrics frames).
|
||||
|
||||
6. **The self-evolution surface has never run.** 13-memory-evolution.png is an empty state under the copy "Your agent improves itself here"; `GET /api/evolution/runs` → `{"runs":[],"count":0}` on an install with two months of history. The superpower is a promise, not a record.
|
||||
|
||||
7. **Automation Center shows stale/past schedules under a "100%" banner.** 06-automation-center.png "NEXT UP" lists 6/12 3:30 AM / 4:00 AM / 5:00 AM — ~16 hours in the past at capture time. `GET /api/automations`: "Prompt optimization" `nextRun: 2026-04-17` (two months stale, `lastRun: null`); "Memory lane extraction" overdue with `lastRun: null`. Never-ran and overdue jobs are invisible to the "Success rate (recent runs): 100%" headline. Also UI says 12 active + 1 paused; the API returns 13 with no enabled/paused field exposed.
|
||||
|
||||
8. **Resume card to an empty workspace.** 02-home-cockpit.png: "YOU WERE WORKING ON — New Hive, 10d ago, Continue" for the same workspace the welcome panel calls "Nothing here yet — start a chat and I'll remember it" (0 memories, 0 sessions; `stats?workspaceId=new-hive` → 0 frames). "Working on" should require content.
|
||||
|
||||
9. **The only agent is judging-day staging.** Agent Center's "Editorial Critic": `createdBy: "user"`, `createdAt: 2026-06-12T18:59:26Z`, never executed (idle, runs never, avg success "—"). As evidence for "real agents," this is a prop placed on the set an hour before the audience arrived.
|
||||
|
||||
10. **Agent-authored skill bypasses the install audit.** `presentation-design` (initiator: agent — the product's best artifact) has no entry in `/api/audit/installs` (8 entries; only `smoke-test-skill`'s creation is audited). The governance trail advertised by the Audit tab doesn't cover the one capability the agent actually authored.
|
||||
|
||||
11. **Flaky memory listing.** My first `GET /api/memory?limit=50` returned `{"results":[],"count":0}`; the identical call minutes later returned all 14 frames. Observed once, not reproduced — but if the Memory Center hits this race, the user sees "no memories" in a memory product.
|
||||
|
||||
12. **Dedup misses live duplicates.** writer-demo frames 8 and 11 ("I always work with a draft → critique → rewrite loop…") are both `active` (created 05-27 and 06-02). The briefing code works around this with a first-line-hash dedup whose own comment admits "consolidation re-writes the same fact as a fresh frame" — the workaround is in the view layer instead of fixing the store.
|
||||
|
||||
---
|
||||
|
||||
## Bottom line
|
||||
|
||||
This is not vaporware — the substrate (per-workspace SQLite minds, a 214-entity knowledge graph, correctable memory statuses that have actually been exercised, real tool-detection at onboarding, a genuinely agent-authored skill with end-to-end provenance, an audit trail with agent-initiated proposals) is real and verifiable, which is more than most "AI that learns" products survive. But the two superpowers are unevenly proven: **memory** is real machinery presenting staged and self-polluted evidence through a hero panel that showcases its own deprecated frames; **self-evolution** is one real artifact standing in front of an evolution log with zero entries, an agent that has never run, and a self-assessment loop that praises itself on zero data. Ship the substrate's honesty all the way up to the welcome screen and criterion 2 and 3 become 5s. Today, the skeptic's verdict: the receipts exist in the database; the storefront oversells them.
|
||||
91
judging/round2/verifier-report.md
Normal file
@@ -0,0 +1,91 @@
|
||||
# Round-2 Verifier Report — commits 72fedf7 + 0ffd938
|
||||
|
||||
**Verifier:** fresh-context, 2026-06-12. HEAD at verification time = `0ffd938` (working tree matches the commits under review; only untracked `judging/crops/`, `judging/round2/`).
|
||||
|
||||
## VERDICT: PASS
|
||||
|
||||
All changes trace to judge complaints or the mission. No new dependencies, no flags/shims, no unrelated refactoring. All four gates green on a fresh run, including the marketplace-sync suite the commit message flagged as flaky. Security posture of the new chat-markdown renderer is sound and test-locked. Three minor, non-blocking notes below.
|
||||
|
||||
---
|
||||
|
||||
## 1. Scope tracing (every hunk → complaint or mission)
|
||||
|
||||
### 72fedf7 (42 files, +1251/−202)
|
||||
|
||||
| Change | Traces to |
|
||||
|---|---|
|
||||
| `LoginBriefing.tsx` — `lastActive` from workspace store (`ws.lastActive`) not `ctx.lastActive`; honest empty-workspace nudge | Recency contradiction ("active yesterday" vs "away 10 days") — machine cron writes no longer count as user activity |
|
||||
| `briefing-highlights.ts` — dedup by normalized first line, keep higher-importance then earliest timestamp | Same memory shown twice with two different ages |
|
||||
| `AppShell.tsx` — LoginBriefing gated to `/home`; `OnboardingTooltips suppressed={ov.showGlobalSearch}` | Modal overlaying Memory/Skills; Ctrl+K tip painting over the open palette |
|
||||
| `home.ts` — `SUGGESTION_MAX_IDLE_DAYS = 30` filter on suggested actions | Stale test prompts recommended as today's actions |
|
||||
| `workspace-context.ts` — `SYSTEM_JOB_TYPES` filter on `buildUpcomingSchedules` | "Up next" showing the janitor's calendar (verified: "Marketplace sync" and "Index reconciliation" both run as `job_type='memory_consolidation'` per `setup-crons.ts:22,29`, so the filter catches every item the judges named) |
|
||||
| `render-markdown.ts` + `TextBlock.tsx` — `renderChatMarkdown` | Literal `## DECISION 1` / `**bold**` noise in chat |
|
||||
| `login-briefing-brag.ts` — "people, projects & things it knows" | entities/relations jargon |
|
||||
| `ChatApp.tsx` / `AgentDetail.tsx` — Ask first / Trusted / Autopilot **display labels only**; internal `'normal'|'trusted'|'yolo'` values untouched | YOLO jargon — explicitly *not* a compat shim |
|
||||
| `dock-tiers.ts` `description` + `AppShell` HintTooltip | Opaque nav labels ("Waggle Dance", "MCP Hub") |
|
||||
| `activity-labels.ts` (new, 37 lines + test) | `tool_result: create_skill` machine vocabulary in the activity rail |
|
||||
| `WorkspaceDesktopApp.tsx` — humanize summaries, "1 memory"/"1 session" plurals | Jargon + grammar complaints |
|
||||
| `AgentsApp.tsx` — empty state lists built-in workspace assistants | "No agents yet" while an agent demonstrably worked (contradiction) |
|
||||
| `EvolutionTab.tsx` — default filter `'all'` + plain-language primer | Agent-evolution invisibility |
|
||||
| `AutomationCenterApp.tsx` — Next-up / Recent-results overview panels | Three stat tiles over a void; no answer to "what runs next / how did it go" |
|
||||
| `skills.ts` + `types.ts` — absent provenance ⇒ `'built-in'` | Unfalsifiable provenance badge (stock skills attributed to the user) |
|
||||
| `identity.ts` — merge-on-update | Partial identity write wiping stored fields |
|
||||
| `profile.ts` — `deleteByContentPrefix('User identity: ')` before re-create | Profile-frame duplication root cause |
|
||||
| `monthly-assessment.ts` — frames stamped source `'system'` | Provenance lie (agent report stamped user_stated) |
|
||||
| `ImportStep.tsx` hints, `onboarding-profile.ts` greeting preview | Onboarding export how-tos + "real greeting preview" complaints |
|
||||
| `HomeCockpit.tsx` `upNext ?? []` | Boundary hardening for an absent field (system-boundary validation — in-spec) |
|
||||
| `judging/*.md` (5 judge verdicts, round1-fixes, verifier report) | Mission evidence artifacts, not code |
|
||||
| Test updates (`phase3b/3c` MemoryRouter wrap, copy assertions) | Direct consequence of `useNavigate` in the new AgentsApp empty state |
|
||||
|
||||
### 0ffd938 (14 files)
|
||||
- `truncateHighlight` strips `**`/`#`/`` ` `` tokens — highlights render as text nodes, so raw markdown showed literally (judge complaint).
|
||||
- `notes/judge-round1-patterns.md` + 13 recaptured screenshots — evidence, in-mission.
|
||||
|
||||
### Negative checks
|
||||
- **No new dependencies:** `git diff 72fedf7~1 0ffd938 --stat -- '**/package.json' package.json package-lock.json bun.lock` → empty. `renderChatMarkdown` is hand-rolled (~40 lines) instead of pulling a markdown lib — consistent with the no-new-deps constraint.
|
||||
- **No flags/shims:** the only new props/params (`suppressed`, injectable `now` for test determinism) are direct fix mechanics. Autonomy rename is display-only.
|
||||
- **No unrelated refactoring:** the `escapeHtml`/`applyInline` extraction in render-markdown.ts is the minimal factoring required for `renderChatMarkdown` to reuse the escape-first pipeline; `buildProfilePreview` rewrite *is* the greeting-preview complaint.
|
||||
|
||||
## 2. Special-attention items
|
||||
|
||||
### renderChatMarkdown security posture — SOUND
|
||||
- **Escape-first confirmed:** both renderers run `escapeHtml()` (escapes `&`, `<`, `>`, `"`) over the *entire input* before any tag is emitted; block parsing in `renderChatMarkdown` operates on already-escaped lines and routes all inline content through `applyInline(escaped)`.
|
||||
- **Href allowlist confirmed:** only `^https?:\/\//i` becomes `<a>`; anything else (javascript:, data:, vbscript:) renders as inert `label (url)` text.
|
||||
- **Tests lock the defenses** (`apps/web/src/lib/render-markdown.test.ts`, read in full): raw `<script>` escaped before tag emission; `javascript:` link refused (asserts no `<a ` emitted); quote-escape blocks attribute breakout (`onmouseover="` absent); raw HTML escaped in every chat line shape (heading and bullet). These are real assertions against the real module, not snapshots.
|
||||
- **Adversarial probe (no failure found):** a backtick code-span inside a link URL (`` [x](https://a`payload`) ``) yields a malformed `<a>` whose junk attributes come from the *fixed* code-span class string; the attacker payload lands in text position with `<` and `"` pre-escaped — no executable vector. Single quotes are not escaped, but every emitted attribute is double-quoted, so no breakout. Cosmetic quirk only.
|
||||
- Minor: `data:`/`vbscript:` have no dedicated test case (the allowlist makes them inert by construction; `javascript:` is the representative lock).
|
||||
|
||||
### Identity merge-on-update — explicit empty string still clears: CONFIRMED
|
||||
`body.role ?? existing?.role ?? ''` — `''` is non-nullish, so an explicit empty string passes through and clears; only *omitted* (undefined) fields fall back to stored values. Test-locked in `packages/server/tests/local/identity.test.ts` ("an explicit empty string still clears a field": writes `{name, role}`, then `{role: ''}`, asserts `role === ''` and `name === 'Marko'`) via real Fastify inject + in-memory MindDB — exercises the route *and* IdentityLayer.
|
||||
|
||||
### profile.ts deleteByContentPrefix — does not delete non-identity frames: CONFIRMED
|
||||
`deleteByContentPrefix` is **pre-existing** (W4.3, `packages/hive-mind-core/src/mind/frames.ts:343`), not added by these commits. It escapes LIKE metacharacters (`\ % _`) and matches the exact literal prefix `'User identity: '` — only frames in the identity-card namespace match. Same established pattern as `monthly-assessment.ts:307`. Routes through `delete(id)` so FTS/vec/KG indexes stay consistent. Residual theoretical risk (a harvested frame whose content *literally begins* with `User identity: ` would be swept) is inherent to the pre-existing primitive, namespaced, and consistent with prior usage — not a regression introduced here.
|
||||
|
||||
## 3. Gates (fresh run by this verifier, 2026-06-12 21:37–21:40)
|
||||
|
||||
| Gate | Result |
|
||||
|---|---|
|
||||
| `npx vitest run --root apps/web` | **943 passed (943)**, 91 files, 24.4s |
|
||||
| `npx vitest run packages/server/tests/local --root .` | **919 passed (919)**, 76 files, 105.9s |
|
||||
| `npx tsc --noEmit --project packages/server/tsconfig.json` | clean (exit 0) |
|
||||
| `npx tsc -p apps/web/tsconfig.app.json --noEmit` | clean (exit 0) |
|
||||
|
||||
**Marketplace-sync adjudication:** `marketplace-sync.test.ts` **passed 12/12 in my run** (slow — 104s, network-dependent: "graceful errors" / multi-source aggregation cases each take 20–36s). Grep of both full diffs for `marketplace`: matches are only (a) a tooltip `description` string on the dock's Marketplace nav entry (UI-only, no runtime logic), (b) commit-message and judging-report prose. **No marketplace server code, routes, sync logic, or test files are touched by either commit — a timeout in that file cannot be caused by these changes.** The commit message's 917/919 claim is consistent with a transient network flake.
|
||||
|
||||
## 4. Test spot-checks (read in full, assert the new behavior)
|
||||
|
||||
1. **`render-markdown.test.ts`** — 12 tests; XSS locks detailed above plus block rendering (headings→block strongs, bullets, numbered lists, hr, inline-inside-heading, blank-line spacing). Asserts on real renderer output strings.
|
||||
2. **`identity.test.ts`** — 2 new merge-on-update tests against a real Fastify instance + `MindDB(':memory:')`: partial write preserves `role`/`department` while updating `name`; explicit `''` clears. Exactly the regression the fix targets.
|
||||
3. **`briefing-highlights.test.ts`** — dedup test feeds two identical-content frames (timestamps 2026-05-01 / 2026-06-11) + one distinct; asserts exactly one survivor carrying the **earliest** (learned) timestamp. Also updated the limit test to use distinct contents so it still measures the limit, not the dedup — correct test hygiene.
|
||||
|
||||
Also verified: `p2-home-desktop.test.tsx` (Up next omitted for `[]` *and* `undefined`, rendered with ≥1 item), `phase3b-agent-center.test.tsx` (empty state must show "Already working for you" + workspace names), `p5-skill-governance.test.ts` (no-frontmatter skill ⇒ `'built-in'`).
|
||||
|
||||
## 5. Findings (non-blocking)
|
||||
|
||||
1. **Test-coverage gaps on new server logic (minor spec deviation):** the 30-day suggested-actions window (`home.ts`), the `SYSTEM_JOB_TYPES` up-next filter (`workspace-context.ts` — `buildUpcomingSchedules` has pre-existing unit tests that were *not* extended for the new filter), and the `profile.ts` replace-on-update call have **no new tests**. The spec's "add tests for new interactive logic" was honored for FE logic and the identity route, but these three server behaviors ship test-uncovered. Each is a small pure filter / one-line integration over a tested primitive, so risk is low — but the job-type filter in particular is behavioral and cheap to lock.
|
||||
2. **Stale comment/type nits:** `apps/web/src/lib/types.ts:529` doc comment still says "Absent/legacy ⇒ 'user'" while the union and server now say `'built-in'`; `CapabilitiesApp.tsx:165` inline cast still narrows `initiator` to `'agent' | 'user'`. Runtime is correct ('built-in' flows through; `SkillRow` badges only `'agent'`), tsc is clean — documentation drift only.
|
||||
3. **Cosmetic renderer quirk:** backtick-in-link-URL produces a malformed (but safe) anchor — see §2. Not exploitable; fix only if it ever surfaces visually.
|
||||
|
||||
## Bottom line
|
||||
|
||||
Both commits are tightly scoped to the judge complaints and the mission, the three special-attention risk areas (XSS posture, empty-string clear, prefix-scoped delete) all hold under direct inspection and test reads, and every gate passes fresh. The marketplace-sync flake is conclusively unrelated. PASS.
|
||||
BIN
judging/round3/crops/01-modal-top.png
Normal file
|
After Width: | Height: | Size: 85 KiB |
BIN
judging/round3/crops/02-band0.png
Normal file
|
After Width: | Height: | Size: 922 KiB |
BIN
judging/round3/crops/02-band1.png
Normal file
|
After Width: | Height: | Size: 1023 KiB |
BIN
judging/round3/crops/02-band2.png
Normal file
|
After Width: | Height: | Size: 481 KiB |
BIN
judging/round3/crops/02-main-bottom.png
Normal file
|
After Width: | Height: | Size: 473 KiB |
BIN
judging/round3/crops/02-main-top.png
Normal file
|
After Width: | Height: | Size: 550 KiB |
BIN
judging/round3/crops/02-sidebar.png
Normal file
|
After Width: | Height: | Size: 93 KiB |
BIN
judging/round3/crops/03-memory-center-c0.png
Normal file
|
After Width: | Height: | Size: 589 KiB |