oleg.m/waggle-os
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

moving
Some checks failed
Installer Smoke / installer-smoke (push) Has been cancelled
Details

Browse Source
This commit is contained in:
Oleg Maslov
2026-09-02 10:10:29 +02:00
commit 0c3e2ead3b
3841 changed files with 970576 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

218
docs/backend-map/00-MENTAL-MODEL.md Normal file
View File

@@ -0,0 +1,218 @@
# 00 · Mental Model — Waggle OS for the Frontend Rebuild
**Audience.** You are about to rebuild the Waggle OS frontend in Lovable. Before you touch a
single screen, read this. It is the *concept map* — what the system is, how its layers fit
together, and which two or three ideas everything else hangs off. The detailed wire contracts
(exact types, routes, capability flags) live in the sibling sections under
`docs/backend-map/sections/`; this document is the picture you keep in your head while you read
those. Every claim here is grounded in `CLAUDE.md`, `docs/ARCHITECTURE.md`,
`docs/WAGGLE-SYSTEM-MAP.md`, and the written subsystem sections.
---
## 1. What Waggle OS is (one paragraph)
Waggle OS is a **workspace-native AI agent operating system with persistent memory**. It looks and
behaves like a small desktop OS — a window manager, a dock, and ~18 first-party "apps" (Chat,
Memory, Files, Marketplace, Mission Control, and more) — but every app is a surface onto a stack of
AI agents that *remember*. It ships three ways from one codebase: a **Tauri 2.0 desktop binary**
(Rust shell, ~120 MB) for Windows and macOS, the same React app as a **web bundle**, and a
**Node.js Fastify sidecar** that does all the real work (agent loop, tool execution, memory,
LLM routing). The product thesis is that memory is the moat: the longer you use Waggle, the more it
knows you, and that accumulated memory is what makes the agents progressively more useful — and
expensive to leave behind. Strategically, Waggle is the demand-creation funnel for **KVARK**,
Egzakta Group's sovereign enterprise AI platform.
---
## 2. The layered mental model
Everything a user does flows through the same six layers, top to bottom. There are no hidden
services and no external dependency for core functionality — the whole stack runs locally.
```mermaid
flowchart TD
U[User] --> SHELL[Tauri Shell · Rust\nspawns sidecar · tray · global hotkey · watchdog]
SHELL -->|hosts webview| UI[React Desktop OS · apps/web\nwindow manager · dock · 18+ apps · SSE]
UI -->|HTTP + SSE + WebSocket| SIDE[Sidecar · Node.js + Fastify\n150+ routes · agent loop · tool execution]
SIDE -->|direct function calls| PKG[Packages · TypeScript libraries\nagent · core · hive-mind-core · shared · marketplace ...]
PKG --> MIND[(Per-workspace SQLite\n.mind files · memory + KG + vectors)]
PKG --> PG[(Team / Cloud Postgres\nDrizzle · users · teams · tasks · governance)]
SIDE -->|OpenAI-compatible| LLM[LLM Router · LiteLLM\nfalls back to built-in Anthropic proxy / echo]
classDef store fill:#1d2330,stroke:#a78bfa,color:#fff;
class MIND,PG store;
```
Read it as a request lifecycle:
1. **Shell (Tauri / Rust).** Owns the OS-level concerns only: spawns and supervises the sidecar
(watchdog restarts it on crash, max 5/10 min), the tray icon, the global toggle hotkey, and the
webview that loads the web app. The shell holds no business logic — for a Lovable rebuild it is
effectively invisible; you target the same web app it hosts.
2. **UI (React desktop OS — `apps/web/`).** The window manager, dock, and apps. **This is what you
are rebuilding.** It is a thin client: it renders state, streams responses, and POSTs user
intent. It never talks to a database or an LLM directly — it talks only to the sidecar over
HTTP, Server-Sent Events (streaming chat/events/notifications), and WebSocket (team presence).
3. **Sidecar (Node.js + Fastify, port 3333).** The brain stem. 150+ routes. It resolves the
workspace session, builds the system prompt, runs the agent loop, executes tools, and streams
results back as SSE. Every UI action terminates here.
4. **Packages (TypeScript libraries).** The sidecar is thin glue over a monorepo of ~27 workspace
packages. The ones you will feel through the API: `@waggle/agent` (the loop, personas, tools,
gates), `@waggle/hive-mind-core` (the memory substrate — mind + harvest), `@waggle/core`
(config, vault, telemetry, compliance), `@waggle/shared` (the wire types + tier model),
`@waggle/marketplace`, `@waggle/waggle-dance`.
5. **Dual data stores.** Two databases that never share a foreign key — see §3.
6. **LLM router (LiteLLM).** The agent loop POSTs to an OpenAI-compatible `/chat/completions`
endpoint. In practice that is LiteLLM (`litellm-config.yaml`), which routes/normalizes model
names; it falls back to a built-in Anthropic proxy, and to a deterministic echo provider when no
keys are present. The frontend never sees this — it only ever sees SSE tokens.
The single rule to internalize: **the frontend is a presentation + streaming client. All
intelligence, persistence, and secret handling lives in the sidecar and below.** If you find
yourself wanting to put logic in the UI, it almost certainly already exists behind a route.
---
## 3. Dual persistence — and why both exist
Waggle has **two** stores, deliberately. Confusing them is the most common architectural mistake.
| | Per-workspace **SQLite "mind"** | Team / Cloud **Postgres** |
|---|---|---|
| Engine | SQLite (`better-sqlite3` + `sqlite-vec`) | PostgreSQL via Drizzle ORM |
| Scope | **One workspace's private memory** | Teams, users, agents, tasks, jobs, governance |
| Holds | Memory frames, knowledge graph, embeddings, identity, awareness | Multi-user, multi-tenant relational data |
| Location | A local file per workspace (`mind_path`) | Cloud / server (`DATABASE_URL`) |
| Defined in | `packages/hive-mind-core/src/mind/` | `packages/server/src/db/schema.ts` (20 tables) |
**Why two?** They answer two different questions.
- The **mind** answers *"what does this agent know and remember here?"* It is the moat. It is
local-first, private, per-workspace, and runs synchronously with zero network. Memory is written
as **frames** (I-frame = foundational identity, P-frame = incremental update, B-frame = bridge),
searched by a **hybrid** pipeline (FTS5 keyword + sqlite-vec vector, fused by reciprocal-rank),
and grown into a knowledge graph by the Cognify pipeline. Each workspace is its own `.mind` file,
so workspace isolation is a filesystem fact, not a query filter.
- **Postgres** answers *"who are the people, teams, and tasks, and what are they allowed to do?"*
It is the shared, governed, collaborative substrate — only meaningful in TEAMS/cloud mode. It
carries no memory frames.
The **only bridge** between them is one nullable column: `users.mind_path`, a text pointer from a
cloud user row to where that user's local SQLite mind lives. There are **no cross-database foreign
keys** and no cascades — the two layers are joined only in application code. For the frontend, this
means: memory-related screens read from mind-backed routes; team/admin/governance screens read from
Postgres-backed routes; never assume one knows about the other.
---
## 4. The moat (and therefore the upgrade trigger)
This is the business model encoded into the architecture — and it dictates what the UI should and
should not nag the user about.
- **Memory + Harvest are free forever.** Harvest ingests the user's existing AI history (ChatGPT,
Claude, Claude Code, Gemini, Perplexity, and more) and memory accumulates from every
conversation. This is the lock-in moat: the longer they stay, the more Waggle knows them, the
costlier it is to leave. It is never paywalled.
- **Agents are free.** Spawning agents is enabled on *every* tier (`spawnAgents: true` everywhere).
Agents are free precisely *because they generate memory* — they feed the moat. Gating them would
be self-defeating.
- **Skills + Connectors + Team features are the upgrade trigger.** FREE blocks custom skills, caps
connectors at 5 and workspaces at 5, and limits export formats. PRO ($19/mo) unlocks unlimited
connectors/workspaces, custom skills, and full export — but stays solo. TEAMS unlocks shared
workspaces, the team skill library, governance, and cloud sync.
**Frontend consequence:** put friction on *skills, connectors, and team collaboration*, never on
*memory or agents*. Upgrade CTAs belong on locked skill/connector/team surfaces. (And there is a
distinct, more aggressive CTA reserved for the KVARK enterprise path — see §6.)
---
## 5. The agent runtime (what `POST /api/chat` actually does)
When the user sends a message, the sidecar turns it into one agent turn. Conceptually it is a
**persona-driven, memory-aware, tool-calling loop with completion gates.**
```mermaid
flowchart LR
M[User message] --> ORCH[Orchestrator\nbuild system prompt]
ORCH --> RECALL[Recall memory\nhybrid search → inject top frames]
RECALL --> PROMPT[Layered prompt\nidentity + memory + behavioral spec + persona + scaffold]
PROMPT --> LOOP{Agent loop\nup to 200 turns}
LOOP -->|tool calls| TOOLS[Execute tools\ngovernance → hooks → injection-scan → loop-guard]
TOOLS --> LOOP
LOOP -->|no tool calls| GATES{Completion gates\nverify · real writes · skill distill}
GATES -->|gate fires| LOOP
GATES -->|clean| OUT[Final answer\nstream SSE → autosave memory]
```
The pieces a frontend should understand:
- **Persona-driven.** Every turn runs under one of the built-in personas (22 in data;
general-purpose / planner / verifier / coordinator among them). The persona supplies the system
prompt slant, model preference, and — critically — its **tool allowlist/denylist**. Read-only
personas (e.g. verifier, planner) cannot be handed write tools. This is why the UI lets the user
switch personas: it changes the agent's whole posture and capability set.
- **Gated tools.** Tools are never executed blindly. Each call passes an ordered middleware chain:
governance (`blockedTools`), pre/post hooks, **injection scanning** of external/connector input,
and loop-guard (duplicate/oscillation detection). Sensitive tools hit an **approval gate** —
which surfaces in the UI as an approval card the user must accept. (YOLO/auto-approve is opt-in,
off by default.)
- **Memory recall + autosave.** Before the loop, the orchestrator recalls relevant frames and
injects them. After it, important facts/decisions/preferences are auto-saved back to the mind.
The conversation also persists to a `.jsonl` session file. This recall→answer→save cycle is the
loop that keeps the moat filling.
- **Self-evolution.** Turns are traced; recurring capability gaps, corrections, and workflow
patterns are tallied as improvement signals and surfaced to the user as suggestions (max a few at
a time). The system improves its own prompts/personas over time — a closed loop, not tier-gated.
- **Streaming contract.** The UI never blocks on a full response. It consumes SSE events —
`token`, `step`, `tool_start`/`tool_end`, `done` — and renders them incrementally. Tool cards,
approval gates, and the typing stream are all driven off this one stream.
---
## 6. The 5-tier model and the KVARK funnel
Tiers are defined canonically in `packages/shared/src/tiers.ts` and consumed by both sidecar and
UI. There are exactly five.
| Tier | Price | What it is |
|---|---|---|
| **TRIAL** | $0 / 15 days | All features unlocked; **falls back to FREE after 15 days** |
| **FREE** | $0 forever | 5 workspaces, agents, built-in skills only |
| **PRO** | $19/mo | Solo power tier: unlimited workspaces/connectors, custom skills, full export |
| **TEAMS** | $49/mo per seat | Shared workspaces, WaggleDance, governance, cloud sync, KVARK CTA active |
| **ENTERPRISE** | Consultative | **KVARK** sovereign on-prem (www.kvark.ai) — contract-billed |
Two rules the frontend must obey:
1. **Always resolve the effective tier first.** A stored `TRIAL` whose 15 days have elapsed must
be treated as `FREE`. Run the user's tier through `getEffectiveTier(tier, trialStartedAt)`
*before* reading any capability flag, or you will render features an expired trial cannot use.
In `TierCapabilities`, **`-1` means "unlimited"**, not zero.
2. **KVARK is the top of the funnel, not a sixth tier.** ENTERPRISE *is* KVARK: "everything Waggle
does, on your own infrastructure, inside your perimeter." Waggle's whole job strategically is to
create qualified demand for KVARK. The tier model exposes this as `kvarkCta` (`none`/`subtle`/
`active`) — the CTA escalates as the user climbs toward TEAMS. KVARK URLs are hardcoded in
exactly two places (`kvark-tools.ts` and the `KvarkNudge` component); the `kvark_search` /
`kvark_ask_document` tools are gated to TEAMS/ENTERPRISE. The frontend surfaces the CTA per the
tier's `kvarkCta` value — it does not invent its own enterprise pitch.
---
## TL;DR for the rebuild
- The frontend is a **streaming presentation client** over a local Fastify sidecar — no DB, no LLM,
no secrets in the UI.
- Two stores: **per-workspace SQLite mind** (memory, private, local-first) and **cloud Postgres**
(teams/governance). Bridged only by `users.mind_path`.
- **Memory + agents are free** (they fill the moat); **skills, connectors, and team features** are
what you gate and up-sell.
- Chat is a **persona-driven, gated, memory-aware loop** whose output you render as an **SSE
stream** (tokens, tool cards, approval gates).
- Five tiers; **always `getEffectiveTier` first**; **KVARK is the enterprise funnel**, surfaced via
the tier's `kvarkCta` flag.
For exact shapes and routes, continue to `docs/backend-map/sections/` (02* = data model & tiers,
03* = API surface, 05* = subsystems).

447
docs/backend-map/07-FRONTEND-REBUILD-GUIDE.md Normal file
View File

@@ -0,0 +1,447 @@
# 07 — FRONTEND REBUILD GUIDE (Lovable)
> **What this is.** The action-oriented, build-in-order playbook for rebuilding the Waggle OS web UI
> in Lovable against the **existing, unchanged Fastify sidecar**. The backend is the contract; the
> frontend is replaceable. Everything below is grounded in the backend map sections (`02c`, `03a–03g`,
> `04`) and the live `apps/web/src/` source. Field names, paths, ports, and tokens are quoted verbatim.
>
> **Companion docs (read alongside):**
> - `sections/04-feature-map.md` — full app↔endpoint matrix (the canonical screen list).
> - `sections/02c-shared-types-tiers.md` — wire types, Zod request schemas, the 5-tier capability matrix.
> - `sections/03a-api-chat-agents.md` — chat SSE event catalogue + approvals.
> - `sections/03f-api-realtime-ops.md` — the 4 SSE streams + ops endpoints.
> - `sections/03g-api-cloud-billing-kvark.md` — auth handshake, guards, Stripe.
---
## 0. The 60-second mental model
Waggle's UI is **a single-page "desktop OS"**, not a multi-route app. React Router has exactly two routes
(`/` → `<Index>`, `*` → `<NotFound>`). `Index` shows a boot screen then mounts `<Desktop>`, which **is** the
shell: it owns a window manager, a Dock, draggable app windows, and overlays. Apps are opened by `appId`,
not by URL. Everything talks to the backend through **one singleton adapter** pointed at the local Fastify
sidecar.
```
Lovable App
└─ ServiceProvider (calls adapter.connect() once, exposes useService())
└─ Desktop shell
├─ Dock (tier-filtered app launcher)
├─ WindowManager (open/close/focus windows by appId)
│ └─ AppWindow × N → renderAppContent(appId) → <XxxApp />
└─ Overlays (modals, rails, wizards)
── all of the above import the SAME `adapter` singleton ──
└─ adapter → http://127.0.0.1:3333 (Fastify sidecar)
```
Build the adapter + ServiceProvider + Desktop shell **first**. Everything else is screens that call adapter
methods.
---
## 1. API base URL, transport, auth, headers, tier pattern
### 1.1 Base URL
| Concern | Value / behavior |
|---|---|
| Default server | `http://127.0.0.1:3333` (constant `DEFAULT_SERVER`) |
| Override | `localStorage["waggle:server-url"]`, settable via `adapter.setServerUrl(url)` |
| Resolution order | `ctorArg ?? localStorage["waggle:server-url"] ?? DEFAULT_SERVER` |
| Prefix | **None.** Every route hardcodes its own full path (`/api/...`, `/health`, `/ws`, `/v1/...`). Do **not** add a base prefix. |
| Same-origin reality | In the desktop binary the SPA is served BY the sidecar, so the page origin IS the API root (`http://localhost:3333` or `tauri://localhost`). For a Lovable web rebuild, hit `http://127.0.0.1:3333` explicitly; expect CORS to be pre-allowlisted for `localhost:5173/8080/8081/8082/3333/1420` and `tauri://localhost`. |
### 1.2 Auth handshake (two-step, no login form)
There is **no username/password UI** for the local sidecar. Auth is a boot-generated session token:
1. **Static shell loads token-free** — non-API GETs are auth-exempt.
2. **Fetch token once on connect:** `GET /api/auth/session-token` → `{ token }`. This route is auth-exempt
but **same-origin gated**. Store it as `authToken`.
3. **Send Bearer on everything else:** every `/api/*` request must carry
`Authorization: Bearer <token>`. Missing → `401 MISSING_TOKEN`; wrong → `401 INVALID_TOKEN`.
4. **WebSocket** (optional) uses the token in the query string: `GET /ws?token=<authToken>` (not a header).
Wrong token closes the socket with code `4001`.
```
connect():
GET /health → probe (fallback to DEFAULT_SERVER once if stored URL fails; persist working URL)
GET /api/auth/session-token → { token }; this.authToken = token
set _connected = true
```
### 1.3 The single `fetch` wrapper (reproduce exactly)
Every request goes through one `adapter.fetch(path, init)`. It MUST:
- Add `Content-Type: application/json` **only when a body is present** and no content-type was supplied.
(Body-less POSTs must NOT send a JSON content-type — this is a deliberate fix; sending it breaks some routes.)
- Add `Authorization: Bearer <authToken>` when the token is set.
- On HTTP **403** with body `{ error: 'TIER_INSUFFICIENT' }`, dispatch a global DOM event
`window.dispatchEvent(new CustomEvent('waggle:tier-insufficient', { detail: { required, actual, message } }))`.
This is the **only** trigger for the Upgrade modal.
- Use a timeout: **10s default, 30s for upload/ingest/harvest**. Throw `TimeoutError` / `NetworkError`.
- Handle rate limits: the sidecar enforces sliding-window limits (default **100/min**; `/api/chat` 120/min;
`/api/vault/*/reveal` 5/min; `/api/backup` & `/api/restore` 2/min; `/api/browse/local/mkdir` 10/min).
On `429`, read `Retry-After` and back off.
### 1.4 Read-side normalizers (reproduce or the UI crashes on `undefined`)
The backend and UI disagree on several field names. The adapter normalizes on read; you must too:
| Helper | Mapping |
|---|---|
| `unwrapArray<T>(data)` | accepts a raw array OR `{ results: [...] }` / `{ <key>: [...] }` envelope |
| `normalizeFrame(raw)` | frameType code `I/F/E/D/T/N` → `insight/fact/event/decision/task/entity`; `importance` string ↔ number 1–4 |
| `normalizeCronJob(raw)` | server `cronExpr/lastRunAt/nextRunAt` → client `schedule/lastRun/nextRun` |
| `getFleet()` | server `durationMs/tokensUsed` → client `duration/tokenUsage` |
| `getModelPricing()` | server `inputPer1k/outputPer1k` → client `inputCostPer1k/outputCostPer1k` |
| `getMemoryStats()` | server `frameCount/entityCount/relationCount` → client `frames/entities/relations`; tolerate `workspace:null` |
| `getModel()` | accepts raw `string` OR `{ model }` |
> **Tauri dual-path (skip for Lovable web).** A few memory methods (`addMemoryFrame`, `searchMemory`,
> `getKnowledgeGraph`, `getIdentity`) branch on `isTauri()` and use Rust IPC. A web rebuild always takes the
> `else` HTTP branch — ignore the IPC path entirely.
### 1.5 Tier gating pattern (read this before gating any feature)
- Fetch the user's tier from `GET /api/tier` → `{ tier, trialDaysRemaining?, trialExpired?, capabilities, usage }`.
- **Always run the stored tier through `getEffectiveTier(tier, trialStartedAt)` before reading
capabilities** — an expired TRIAL must collapse to FREE gating. (`getEffectiveTier`, `getCapabilities`,
`hasCapability` live in `@waggle/shared/tiers`; port them or reimplement.)
- Gate UI on `TierCapabilities` flags, not on the tier name. `-1` means **unlimited** (short-circuits to allowed).
- The **upgrade trigger is Skills + Connectors + Team features** — never agents/memory (`spawnAgents` is `true`
in every tier; memory/embedding quotas are `-1` everywhere). The full matrix is in `02c §15.2`.
- Two distinct tier axes exist:
- `BillingTier` = `'TRIAL'|'FREE'|'PRO'|'TEAMS'|'ENTERPRISE'` (entitlements).
- `UserTier` = `'simple'|'professional'|'power'|'admin'` (UI density / dock complexity).
- The Dock is computed from both via `getDockForTier(userTier, billingTier)`.
---
## 2. Real-time: which endpoints stream, and how to consume each
There are **five** streaming surfaces. Four are SSE (`EventSource`), one is the chat POST-SSE hybrid, plus an
optional WebSocket. **The chat stream is a POST, so you cannot use `EventSource` for it** — you parse the
response body manually.
### 2.1 Chat token stream — `POST /api/chat` (SSE-formatted body)
- **Not** a JSON endpoint and **not** `EventSource`-compatible (EventSource is GET-only). Issue a `fetch` POST,
read `response.body.getReader()`, decode, and split on `\n\n`; each frame is `event: <name>\ndata: <json>`.
- All validation/auth happen **before** the server hijacks the reply. So a `400`/`403` comes back as normal
JSON; anything after is SSE. Check `response.ok` / content-type before entering the stream loop.
- **Request body:** `{ message (required), workspace?/workspaceId?, model?, session?, persona?, autonomy?, shape? }`.
`autonomy = { level: 'normal'|'trusted'|'yolo', expiresAt? }`.
- **Event catalogue** (handle ALL of these):
| `event:` | `data` | Action |
|---|---|---|
| `token` | `{ content }` | append to streaming assistant text |
| `step` | `{ content }` | render a progress line ("Recalling memories…", budget notes) |
| `tool` | `{ name, input }` | show tool-call block (adapter normalizes name `tool`→`tool_start`) |
| `tool_result` | `{ name, result, duration?, isError }` | close tool block (normalized `tool_result`→`tool_end`) |
| `file_created` | `{ filePath, fileAction: 'write'|'edit'|'generate' }` | show "file created" affordance |
| `approval_required` | `{ requestId, toolName, input, sourceWorkspaceId, ...trustMeta }` | **PAUSE.** Render approve/deny; resolve via `POST /api/approval/:requestId` |
| `gepa_choices` | `{ original, expanded, clarifyingQuestions[], intent }` | offer ask-first clarification |
| `model_switch` | `{ model, reason, primary }` | toast "switched to <model>" |
| `notification` | `{ type:'workflow_captured', title, message, pattern }` | suggest saving a workflow |
| `done` | `{ content, usage{inputTokens,outputTokens}, toolsUsed[], model, cost?, tokens? }` | **terminal success** |
| `error` | `{ message }` | **terminal failure** (user-friendly only) |
Tolerate two `done` usage shapes: agent-loop `{ inputTokens, outputTokens }` and echo/command-path
`{ prompt_tokens, completion_tokens, total_tokens }`.
- **Approvals are blocking.** When `approval_required` arrives the agent is paused awaiting a server-side
Promise (auto-denies after **5 min**). POST `{ approved: boolean, always?: boolean, reason?, sourceWorkspaceId }`
to `/api/approval/:requestId`, echoing `sourceWorkspaceId` verbatim. `always:true` persists a grant.
### 2.2 The four GET SSE streams (`EventSource`)
| Endpoint | How to subscribe | Events you read |
|---|---|---|
| `GET /api/waggle/stream` | `new EventSource(url)` + `addEventListener('signal', …)` | `signal` → full `WaggleSignal` JSON. Initial `event: connected`. Heartbeat `: heartbeat` every 30s. |
| `GET /api/events/stream` | `addEventListener('audit', …)` | `audit` → full `AuditEvent` JSON. Initial `data:{"type":"connected"}`. |
| `GET /api/notifications/stream` | **mixed:** `onmessage` for unnamed frames + `addEventListener('subagent_status'|'workflow_suggestion', …)` | `notification` arrives as an **unnamed** `data:` frame (use `onmessage`); `subagent_status` and `workflow_suggestion` are **named** events. |
| `GET /api/harvest/progress` | `new EventSource(url)` | `{ phase, current, total, source }` progress frames. Adapter wraps as `{ ready: Promise, close }`. |
**EventSource caveat:** native `EventSource` cannot set an `Authorization` header. The sidecar gates these
streams by **same-origin** (the SPA is same-origin in production). For a cross-origin Lovable dev build you may
need to either (a) run behind a same-origin dev proxy to the sidecar, or (b) use a fetch-stream polyfill that
injects the Bearer header. The streams set `Access-Control-Allow-Origin` only for allowlisted origins.
### 2.3 WebSocket (optional) — `GET /ws?token=<authToken>`
Local sidecar `/ws` is an event-bus relay. Server→client frames `{ event, data }` for
`approval_required | step | tool | done | error | presence_update | notification`. Client→server
`{ type: 'approve'|'deny', requestId }`. The SSE streams already cover the UI's needs, so WS is **optional** for
the rebuild. (The cloud-server `/ws` is a separate Clerk+Redis team-chat gateway — not used by the local UI.)
### 2.4 Hook pattern for streams
Mirror the existing domain hooks: each owns local state + opens its stream on mount, returns close on unmount.
```ts
// useEvents → subscribeEvents (/api/events/stream)
// useNotifications → /api/notifications/stream (unread count, markRead, markAllRead)
// useRoomState → subscribeSubagentStatus (named 'subagent_status' on /api/notifications/stream)
// useWaggleDance → subscribeWaggleDance (/api/waggle/stream) + publish/ack
// useChat → POST /api/chat fetch-stream → parses into ContentBlock[]
```
---
## 3. Global state & providers the rebuild needs
**There is no Redux / Zustand / React Query.** State = React Context (one provider) + custom hooks + the
singleton adapter + a `window` CustomEvent bus. Reproduce these four layers.
### 3.1 The one provider
| Provider | Provides | Behavior |
|---|---|---|
| `ServiceProvider` / `useService()` | `{ adapter, connected, connecting, error, reconnect }` | calls `adapter.connect()` **once** on mount; wrap the whole app. (Plus shadcn `TooltipProvider` + a toast `Toaster`.) |
### 3.2 Domain hooks (the de-facto store)
Recreate these as the state layer — each wraps adapter calls + `useState`. The Desktop shell composes them.
| Hook | Owns |
|---|---|
| `useWorkspaces` | workspaces, **active workspace**, select/create/patch/delete/refresh — the workspace context every other call needs |
| `useChat` | `messages` (as `ContentBlock[]`), `isLoading`, `sendMessage`, `clearHistory`, `pendingApproval`, `approveAction`; parses the chat SSE stream |
| `useSessions` | per-workspace session CRUD (list/create/rename/delete/search/export) |
| `useMemory` | frames, stats, search, add/update/delete, `incrementFrameAccess` |
| `useKnowledgeGraph` | `{ nodes, edges }` |
| `useEvents` | audit steps + live SSE |
| `useNotifications` | notifications, `unreadCount`, markRead/markAllRead, live SSE |
| `useRoomState` | live sub-agent map (named `subagent_status` SSE) |
| `useWaggleDance` | signals, filter, ack, publish, live SSE |
| `useAgentStatus` | polls `/api/agent/status` |
| `useBilling` | **tier**, `startCheckout`, `openPortal`, `syncAfterCheckout` (auto-detects `?session_id=` on load) |
| `useFeatureGate` | `{ planTier, isEnabled, gate }` — the per-feature gate front-end |
| `useOnboarding` | persisted `OnboardingState` in `localStorage["waggle:onboarding"]` (`completed, step, tier, workspaceId, apiKeySet, templateId, personaId, tooltipsDismissed`); honors `?skipOnboarding=true`, `?forceWizard=true` |
| `useOfflineStatus` | polls `/health` for the offline pill |
| `useProviders` | `/api/providers` |
| `useWindowManager` | window open/close/focus/minimize + **per-window persona & autonomy** |
| `useOverlayState` | boolean flags for every overlay |
| `useKeyboardShortcuts` | global hotkeys → open app/overlay (Cmd+K global search, etc.) |
### 3.3 The four pieces of "global context" the rebuild must thread
1. **Workspace context** — `useWorkspaces.active`. Nearly every call accepts `workspace`/`workspaceId`
(alias-accepted; omit ⇒ `personal` mind). The active workspace id flows into chat, memory, files, sessions,
events. The window manager can override **persona** and **autonomy** per-window.
2. **Session** — the boot session token (`adapter.authToken`) for transport auth, plus the per-workspace
chat session id (`useSessions`, defaults to the workspace id then `'default'`).
3. **Persona** — workspace-default persona, overridable per-chat-window. Passed as `persona` in the chat body.
4. **Tier** — `useBilling.tier` → `getEffectiveTier` → capabilities, consumed by `useFeatureGate`, the Dock, and
the Upgrade/TrialExpired modals.
### 3.4 Cross-component `window` event bus (no library — wire these)
| Event | Dispatched by | Consumed by |
|---|---|---|
| `waggle:tier-insufficient` `{ required, actual, message }` | `adapter.fetch` on 403 `TIER_INSUFFICIENT` | `UpgradeModal` |
| `waggle:open-app` `{ appId, tab? }` | apps (e.g. `HarvestTab`) | `Desktop` → `wm.openApp(appId)` |
---
## 4. Screen-by-screen data contract
Each row = an OS app/overlay → the adapter methods/endpoints it calls → request/response summary. Paths are
verbatim and all `/api/*` are Bearer-gated. The `AppId` union (window content switch) is:
`chat, dashboard, memory, events, capabilities, connectors, cockpit, mission-control, settings, vault,
profile, terminal, calculator, notes, waggle-dance, files, agents, scheduled-jobs, marketplace, voice, room,
approvals, timeline, backup, telemetry, governance, launcher`.
(`terminal`/`calculator`/`notes` are declared but unimplemented; `voice` is a static "Coming Soon" placeholder.)
### 4.1 Dock apps
| Screen (appId) | Key endpoints | Request → Response summary |
|---|---|---|
| **Chat** (`chat`) | `POST /api/chat` (SSE); `GET /api/history?workspace=&session=`; `DELETE /api/chat/history?session=`; `POST /api/approval/:id`; `POST /api/feedback`; `GET /api/memory/search?q=&scope=`; pins `GET/POST/DELETE /api/workspaces/:id/pins`; `POST /api/ingest`; `GET/PUT /api/agent/model`; `GET /api/settings`; `GET /api/team/members`; `PATCH /api/workspaces/:id` | Send `{message,workspace,session?,persona?,autonomy?,shape}` → SSE stream (§2.1). History → `{ sessionId, messages[{id,role,content,timestamp}], count }`. Feedback `{sessionId,messageIndex,rating,reason?,detail?}` → fire-and-forget. Pin `{messageContent,messageRole,label?}`. Model PUT `{model}`. |
| **Dashboard / Home** (`dashboard`) | `GET /api/memory/stats`; `GET /api/tasks` (raw) | stats → `{ personal, workspace, total }` each `{frames,entities,relations}`. Tasks → task list. |
| **Memory** (`memory`) — tabs: Frames, Knowledge Graph, Harvest, Wiki, Evolution | frames `GET/POST/PUT/DELETE /api/memory/frames`, `PATCH …/:id/access?workspace=`; `GET /api/memory/search?q=&scope=`; `GET /api/memory/graph?workspace=` (or `?scope=all\|personal`); `GET /api/memory/stats`; **Harvest:** `GET /api/harvest/sources`, `POST /api/harvest/scan-claude-code`, `POST /api/harvest/preview`, `POST /api/harvest/commit`, `GET /api/harvest/runs/latest-interrupted`, `POST /api/harvest/runs/:id/abandon`, `POST /api/harvest/extract-identity`, `GET /api/harvest/progress` (SSE), `PATCH/DELETE /api/harvest/sources/:source`; **Wiki:** `GET /api/wiki/pages`, `…/:slug`, `…/:slug/content`, `POST /api/wiki/compile`, `GET /api/wiki/health`, `POST /api/wiki/export/{obsidian,notion}`; **Evolution:** `GET/POST /api/evolution/{runs,runs/:id,run,targets,baseline,status}` (raw); **Weaver:** `GET /api/weaver/status`, `POST /api/weaver/trigger` | Frame (normalized) `{id,content,source,frameType,importance,timestamp,score?,gop,accessCount,…}`. Graph → `{ nodes:KGNode[], edges:KGEdge[] }`. Harvest preview `{data,source}` → `{ knowledgeExtracted[] }`; commit `{data,source}` / `{resumeFromRun}`. Wiki compile `{ mode:'incremental'\|'full', concepts? }`. |
| **Events & Logs** (`events`) | `GET /api/events?workspaceId=`; `GET /api/events/stream` (SSE) | `AgentStep[]` + live `audit` events `{id,timestamp,workspaceId,eventType,toolName?,input?,output?,model?,cost?,…}`. |
| **Skills & Apps** (`capabilities`) incl. Marketplace tab | `GET /api/skills`; `POST /api/skills/create`; `GET /api/skills/starter-pack/catalog`; `POST /api/skills/starter-pack/:skillId`; `GET /api/skills/capability-packs/catalog`; `GET /api/marketplace/packs`; `POST /api/marketplace/install`; `GET /api/audit/installs` + `GET /api/skills/test` (raw) | Skills `SkillPack[]` (forced `installed:true`). Install `{packageId}` → raw `Response` (403 → UpgradeModal). Create `{name,description}`. |
| **Connectors** (`connectors`) | `GET /api/connectors`; `GET /api/connectors/:id/health`; `POST /api/connectors/:id/connect`; `POST /api/connectors/:id/disconnect`; `POST /api/vault` | `ConnectorDefinition[]` (card: `{id,name,description,service,authType,status,capabilities[],substrate,tools[],logoUrl?,category?,setupGuide?}`). Add secret `{name,value,type?}`. |
| **Cockpit / Command Center** (`cockpit`) incl. Compliance | `GET /health`; `GET /api/agent/cost`; `GET /api/connectors`; `GET /api/cron`; `GET /api/vault`; `GET /api/capabilities/status`; `GET /api/audit/installs`; `GET /api/cost/summary`; `GET /api/weaver/status`; `GET /api/events/stats`; **Compliance:** `GET /api/compliance/status?workspaceId=`, `GET /api/compliance/templates`, `POST/PATCH/DELETE /api/compliance/templates*`, `POST /api/compliance/export`, `POST /api/compliance/export-pdf` (Blob) | health `{status,uptime,services[]}`; cost summary; event stats `{totalEvents,period,byType,byDay,topTools}`. |
| **Mission Control** (`mission-control`) | `GET /api/fleet`; `GET /api/team/members`; `GET /api/team/activity`; `GET /api/tools/detect`; `POST /api/fleet/:workspaceId/(pause\|resume\|kill)` | Fleet (normalized) `{workspaceId,workspaceName,personaId,model,status,lastActivity,duration,toolCount,tokenUsage,costEstimate}` + `{count,maxSessions}`. |
| **Waggle Dance** (`waggle-dance`) | `GET /api/waggle/signals`; `GET /api/waggle/stream` (SSE); `POST /api/waggle/signals`; `PATCH /api/waggle/signals/:id/ack` | `WaggleSignal{id,type,workspaceId,content,metadata?,timestamp,acknowledged}`. Publish `{type,content,workspaceId?,metadata?}`. |
| **Personas (Agents)** (`agents`) | `GET /api/personas`; `POST /api/personas`; `PATCH/DELETE /api/personas/:id`; `POST /api/personas/generate`; `GET /api/capabilities/status`; `GET /api/agent-groups`; `POST /api/agent-groups`; `PATCH/DELETE /api/agent-groups/:id`; `POST /api/agent-groups/:id/run`; `GET /api/jobs/:jobId`; `POST /api/jobs/:jobId/cancel` | Persona create `{name,description,icon?,systemPrompt,tools?}`. Generate `{prompt}` → `{name,description,systemPrompt,tools[]}`. Group create `{name,description?,strategy:'parallel'\|'sequential'\|'coordinator',members[{agentId,roleInGroup:'lead'\|'worker',executionOrder}]}`. Group run `{task,teamId?}` → queued stub. |
| **Files** (`files`) | `GET /api/workspaces/:id/files/list?path=`; `POST …/files/upload` (FormData, 30s); `GET …/files/download?path=` (Blob); `POST …/files/{mkdir,delete,move,copy}`; `GET …/documents`; `GET …/documents/:name/versions` | `FileEntry[]`. Move/copy `{from,to}`; delete/mkdir `{path}`. Upload `FormData(file,path)`. |
| **Scheduled Jobs** (`scheduled-jobs`) | `GET /api/cron`; `POST /api/cron`; `PUT /api/cron/:id`; `DELETE /api/cron/:id`; `POST /api/cron/:id/trigger` | CronJob (normalized) `{id,name,schedule,jobType,jobConfig,workspaceId,enabled,lastRun,nextRun,createdAt}`. Create `{name,cronExpr,jobType,jobConfig?,workspaceId?,enabled?}`. Trigger → `{triggered,autoEnabled?,schedule?}`. |
| **Marketplace** (`marketplace`) | `GET /api/marketplace/search?query=&limit=`; `GET /api/marketplace/installed`; `POST /api/marketplace/install`; `POST /api/marketplace/uninstall` | search/install/uninstall return **raw `Response`** for 403-aware handling. |
| **AI Tools / Launcher** (`launcher`) | `GET /api/tools/detect`; `POST /api/tools/launch`; `GET /api/tools/processes`; `POST /api/tools/kill`; `POST /api/tools/hooks` | detect → `{platform,detectedAt,tools[{id,displayName,installed,installedPath,version,hooksInstalled,…}]}`. Launch `{id,installedPath,workspaceId?,args?,cwd?}` → `{ok,pid,error?}`. Hooks `{id,action:'install'\|'verify'\|'uninstall',cliPath?}`. |
| **Room** (`room`) | `GET /api/notifications/stream` named `subagent_status` (SSE) | live sub-agent canvas `{agents[{id,name,role,status:'pending'\|'running'\|'done'\|'failed',task,toolsUsed,…}]}`. |
| **Approvals** (`approvals`, TEAMS+) | `GET /api/approval/pending`; `GET /api/approval/grants`; `POST /api/approval/:id`; `DELETE /api/approval/grants/:id`; `POST /api/approval/grants/clear` | pending `{pending[{requestId,toolName,input,timestamp}],count}`. |
| **Timeline** (`timeline`) | `GET /api/events?workspaceId=&limit=&from=` | `TimelineEvent[]`. |
| **Backup & Restore** (`backup`) | `GET /api/backup/metadata`; `POST /api/backup`; `POST /api/restore` (all raw) | backup streams `application/octet-stream` (`.waggle-backup`, ≤500MB). Restore `{ backup:<base64>, preview? }`. |
| **Usage & Telemetry** (`telemetry`) | `GET /api/cost/by-workspace`; `GET /api/events/stats` (raw) | cost-by-workspace + stats aggregates. |
| **Team Governance** (`governance`, TEAMS+) | none direct (props-driven sub-components) | renders governance UI. |
| **Settings** (`settings`) | `GET/PUT /api/settings`; `GET/PUT /api/settings/permissions`; `POST /api/settings/test-key`; `GET /api/providers`; `GET /api/telemetry/status`; `POST /api/telemetry/toggle`; `DELETE /api/telemetry/events`; `GET /api/team/status`; `POST /api/team/{connect,disconnect}`; `GET /api/export`, `/api/debug/logs` (raw) | Settings object; permissions `{defaultAutonomy,externalGates[],workspaceOverrides}`. Test key `{provider,apiKey}` → `{valid}`. |
| **Vault** (`vault`) | `GET /api/vault`; `POST /api/vault`; `DELETE /api/vault/:id`; `GET /api/connectors`; `POST /api/connectors/:id/(connect\|disconnect)` | secret `{name,value,type?}`. Vault reveal is rate-limited to 5/min. |
| **My Profile** (`profile`) | `GET/PUT /api/profile`; `POST /api/profile/analyze-style` `{text}`; `POST /api/profile/analyze-brand` `{description}`; `POST /api/profile/research` `{}` | profile object + analysis results. |
### 4.2 Overlays (rendered by Desktop, not windowed)
| Overlay | Key endpoints |
|---|---|
| **Onboarding wizard** (8 steps: Welcome, WhyWaggle, Tier, ModelTier, Import, Template, Persona, ApiKey, Ready) | `connect`, `GET /api/vault`, `/health`, `GET /api/providers`, `/api/v1/models`, harvest preview/commit, `POST /api/harvest/scan-claude-code`, import preview/commit, `POST /api/personas`, `POST /api/vault`, `POST /api/workspaces`, `PUT /api/settings` |
| **Login briefing** (session-start digest) | `GET /api/identity`, `GET /api/workspaces`, `GET /api/memory/search`, `GET /api/memory/stats`, `GET /api/workspaces/:id/context` |
| **Global search (Cmd+K)** | `GET /api/workspaces`, sessions, `GET /api/skills`, `GET /api/memory/search` |
| **Create workspace dialog** | `GET /api/browse/local?path=`, `POST /api/browse/local/mkdir`, `GET/POST/PUT/DELETE /api/workspace-templates*`, `POST /api/workspace-templates/generate`, `GET /api/connectors`, `GET /api/agent-groups` |
| **Persona switcher** | `GET /api/personas`, `GET /api/agent-groups` |
| **Spawn agent dialog** | `GET /api/litellm/models`, `/api/litellm/pricing`, `GET /api/providers`, `GET /api/agent/model`, `POST /api/workspaces`, `POST /api/fleet/spawn` (`{task,persona?,model?,parentWorkspaceId?}`) |
| **Erase data dialog (GDPR)** | `POST /api/data/erase` — header `X-Confirm-Erase: yes` + body `{ confirmation:'I UNDERSTAND THIS IS PERMANENT' }` |
| **Upgrade modal** | triggered by `waggle:tier-insufficient`; actions → `POST /api/tier/start-trial`, `POST /api/stripe/create-checkout-session` `{tier:'PRO'\|'TEAMS', billingPeriod?}` → `{url}` |
| **Trial expired modal** | `POST /api/stripe/create-checkout-session` |
| Workspace switcher / Notification inbox / Context rail / Keyboard help / Tooltips | props-driven or `lib/context-rail-fetch.ts`; no/minor direct calls |
### 4.3 Billing / tier endpoints (used across overlays + `useBilling`)
| Method | Path | Body → Response |
|---|---|---|
| GET | `/api/tier` | → `{ tier, trialDaysRemaining?, trialExpired?, capabilities, usage }` |
| POST | `/api/tier/start-trial` | → `{ tier, rawTier, trialStartedAt, trialDaysRemaining, trialExpired, capabilities }` (409 if already started) |
| POST | `/api/stripe/create-checkout-session` | `{ tier:'PRO'\|'TEAMS', billingPeriod?:'monthly'\|'annual' }` → `{ url }` (503 `STRIPE_NOT_CONFIGURED` if unset) |
| POST | `/api/stripe/sync` | `{ sessionId }` → `{ tier, customerId }` — **call this after the checkout redirect** (webhooks unreliable behind NAT) |
| POST | `/api/stripe/create-portal-session` | — → `{ url }` (requires PRO+) |
**Stripe flow for the rebuild:** open `{url}` from create-checkout-session → user pays → Stripe redirects to
`/payment-success?session_id=...` → `useBilling` detects `?session_id=` on load → `POST /api/stripe/sync` →
refresh tier. Render upgrade UI defensively when `503 STRIPE_NOT_CONFIGURED`.
---
## 5. Design system — Hive DS (dark default, desktop-OS metaphor)
The visual contract lives in `apps/web/src/index.css` (`@layer base` source of truth) + `waggle-theme.css`
(component aliases). It's a **shadcn-style HSL-variable system + a literal Hive palette**, with a `[data-theme="light"]`
override. **Default theme is dark.** Tailwind 4 + shadcn primitives + lucide icons.
### 5.1 Core brand tokens (the three the brief names)
| Token | Dark value | Role |
|---|---|---|
| Honey `--honey-500` | `#e5a000` | Primary brand / accent / focus ring / `--primary` (`40 100% 45%`) |
| Hive-950 `--hive-950` | `#08090c` | Deepest background / status bar; `--background` ≈ `222 20% 4%` |
| Accent (AI) `--status-ai` / `--accent` | `#a78bfa` / `270 60% 68%` | Secondary accent (AI, knowledge-concept highlights) |
### 5.2 Full palette (use the CSS variables, never hard-code hex)
- **Hive grays** (cold undertone): `--hive-950 #08090c → --hive-50 #f0f2f7` (12 steps). Surfaces:
`--surface-card: var(--hive-850)`, `--surface-panel: var(--hive-800)`, `--surface-overlay: rgba(8,9,12,0.88)`.
- **Honey scale:** `--honey-600 #b87a00 … --honey-50 #fffbeb`, plus `--honey-glow rgba(229,160,0,0.12)`,
`--honey-pulse rgba(229,160,0,0.06)`.
- **Status:** `--status-healthy #34d399`, `--status-warning #fbbf24`, `--status-error #f87171`,
`--status-info #60a5fa`, `--status-ai #a78bfa`.
- **Knowledge-graph nodes:** `--kg-person #4A90D9`, `--kg-project #50C878`, `--kg-concept #9B59B6`,
`--kg-org #E67E22`, `--kg-default #95A5A6`.
- **shadcn semantic vars** (HSL triplets, consumed via `hsl(var(--x))`): `--background, --foreground, --card,
--popover, --primary (40 100% 45%), --secondary, --muted, --accent (270 60% 68%), --destructive, --border,
--input, --ring (40 100% 45%), --radius 0.75rem`. Sidebar + chart vars mirror these.
- **Step/event colors** (event stream): `--step-running/-success/-pending/-error/-thinking/-search/-web/-tool/-writing`.
### 5.3 Typography, radius, shadows, motion
- **Fonts:** headings `Space Grotesk`; body `DM Sans` (fallback Inter/system); mono `JetBrains Mono`. Imported
from Google Fonts in `index.css`. `--font-sans`, `--font-mono` aliases exist.
- **Type scale:** `--text-micro 11 → --text-display 24` (micro 11, caption 12, body-sm 13, body 14, title 16,
heading 20, display 24).
- **Radius:** `--radius: 0.75rem`.
- **Shadows:** `--shadow-card`, `--shadow-elevated`, `--shadow-overlay`, `--shadow-honey` (honey glow),
`--shadow-focus` (2px honey ring).
- **Signature motifs:** glassmorphism (`.glass` / `.glass-strong` — backdrop blur 20–30px), honeycomb hex
background (`.honeycomb-bg`, SVG data-URI at 3% honey opacity), hex avatar clip-path (`.hex-avatar`),
hex streaming cursor (`.hex-cursor`), honey-pulse on memory-save, heartbeat on health dot, float on the bee
mascot, token-fade on streamed text. Selection + thin 5px scrollbars are honey/hive themed.
- **Interaction utilities:** `.waggle-interactive`, `.waggle-card-lift`, `.waggle-nav-hover`, `.waggle-press`,
`.direction-d-card` (the canonical card: hive-700 border → honey-500 + honey-shadow on hover).
### 5.4 Light mode
`:root[data-theme="light"]` flips the hive scale (cream `#fdfcf9` bg, dark text), darkens honey + status +
KG colors for WCAG AA on cream, and softens shadows. Toggle by setting `data-theme="light"` on `:root`.
**Ship dark first**; light is a polish pass.
### 5.5 Layout — the desktop OS metaphor
- **Desktop**: full-viewport, `overflow:hidden` body, honeycomb wallpaper + `.desktop-overlay` wash.
- **Dock**: launcher rail (bottom/side) built from `lib/dock-tiers.ts`. Entries are `app | zone-parent
(collapsible group) | separator`, each with `icon` (lucide), `label`, `color`. `getDockForTier(userTier,
billingTier)` filters by `minBillingTier` (e.g. `governance` & `approvals` are TEAMS+; empty zone-parents are
dropped). `simple` tier shows ~6 apps; `power`/`admin` show the full set incl. Ops + Extend zone groups.
- **Windows**: each open app is a draggable/resizable `AppWindow` keyed by `appId`; content via a
`renderAppContent(appId)` switch. Windows carry per-window persona + autonomy.
- **Overlays**: modals/rails/wizards rendered directly by Desktop, gated by `useOverlayState` flags.
- **BootScreen**: shown until `localStorage["waggle-booted"]`, then Desktop mounts.
---
## 6. Prioritized rebuild order
Build in dependency order. Each phase is independently demoable.
**Phase 0 — Foundation (nothing renders without this).**
1. `adapter` singleton: base URL + `localStorage["waggle:server-url"]`, `connect()` (health probe + session-token
bootstrap), the `fetch` wrapper (conditional content-type, Bearer, 403→`waggle:tier-insufficient`,
10s/30s timeouts, 429 handling), and the §1.4 read normalizers.
2. `ServiceProvider` / `useService()` calling `adapter.connect()` once.
3. Hive DS tokens (`index.css` + `waggle-theme.css`), dark default, fonts, shadcn vars.
4. The `window` event bus (§3.4).
**Phase 1 — Shell.**
5. Desktop shell: `useWindowManager` (open by `appId`), Dock from `dock-tiers.ts`, `AppWindow` + `renderAppContent`
switch, BootScreen, `useOverlayState`, `useKeyboardShortcuts` (Cmd+K).
6. `useWorkspaces` (active workspace context) + Workspace switcher + Create-workspace dialog.
7. `useBilling` + tier gating (`getEffectiveTier` → capabilities → `useFeatureGate`) + Upgrade/TrialExpired modals.
**Phase 2 — The product's core loop (Chat).**
8. **Chat** (`chat`) end-to-end: `POST /api/chat` fetch-stream parser → `ContentBlock[]`, all SSE events,
inline approvals (`POST /api/approval/:id`), history, model switch, pins, feedback. This is the single
highest-value screen — do it first and well.
9. **Dashboard / Home** (`dashboard`) — cheap, gives a landing surface (`/api/memory/stats`, `/api/tasks`).
**Phase 3 — Memory moat (the strategic lock-in).**
10. **Memory** (`memory`): Frames + Knowledge Graph tabs first, then **Harvest** (import is the moat: scan +
preview/commit + progress SSE), then Wiki, then Evolution.
11. **Files** (`files`) — workspace file CRUD + upload/download.
**Phase 4 — Real-time ops + agents.**
12. **Events** (`events`, SSE), **Room** (`room`, `subagent_status` SSE), **Waggle Dance** (`waggle-dance`, SSE).
13. **Personas/Agents** (`agents`) + Spawn agent dialog + Mission Control (`mission-control`, fleet).
14. **Scheduled Jobs** (`scheduled-jobs`), **Notifications** inbox.
**Phase 5 — Extensibility + monetization surfaces.**
15. **Skills & Apps** (`capabilities`) + **Marketplace** (`marketplace`) + **Connectors** (`connectors`) +
**Vault** (`vault`) — these are the upgrade triggers; wire 403→Upgrade carefully.
16. **Settings** (`settings`), **My Profile** (`profile`), **Cockpit** (`cockpit`) + Compliance.
**Phase 6 — Governance / admin / polish.**
17. **Approvals** (`approvals`, TEAMS+), **Team Governance** (`governance`, TEAMS+), **Timeline** (`timeline`),
**Telemetry** (`telemetry`), **Backup** (`backup`), **Erase data** (GDPR).
18. **Onboarding wizard** (8 steps) + Login briefing + Global search polish.
19. **Light mode** pass.
**Rationale:** Phases 0–1 are non-negotiable scaffolding. Chat (Phase 2) is the product. Memory/Harvest (Phase 3)
is the strategic moat ("free forever" lock-in). Real-time + agents (Phase 4) prove the "OS" thesis.
Monetization surfaces (Phase 5) are where tier gating earns money. Governance + polish (Phase 6) come last.
---
## 7. Gotchas checklist (the things that bite a rebuild)
- [ ] Chat is **POST-SSE**, not `EventSource` — parse the body stream manually.
- [ ] Body-less POSTs must **not** send `Content-Type: application/json`.
- [ ] `EventSource` can't send Bearer headers — the SSE streams are **same-origin gated**; proxy or polyfill in dev.
- [ ] Run tier through **`getEffectiveTier`** before gating (expired TRIAL → FREE).
- [ ] `-1` in any capability/limit means **unlimited**, not "zero/disabled".
- [ ] Reproduce the §1.4 read normalizers or the UI breaks on renamed fields.
- [ ] Echo back `sourceWorkspaceId` verbatim when resolving approvals.
- [ ] After Stripe redirect, call `POST /api/stripe/sync { sessionId }` — do not trust the webhook for desktop.
- [ ] Handle two `done` usage shapes (agent-loop vs echo/command path).
- [ ] `/api/notifications/stream`: `notification` is an **unnamed** frame (`onmessage`); the others are named events.
- [ ] All `/api/*` paths are **flat** — no plugin prefix; the literal path in the table IS the path.
- [ ] Respect rate limits (chat 120/min, vault reveal 5/min, backup/restore 2/min) — handle `429` + `Retry-After`.

144
docs/backend-map/AUDIT.md Normal file
View File

@@ -0,0 +1,144 @@
# Backend Map — Completeness Audit
**Date:** 2026-06-06
**Scope:** Cross-check the written sections under `docs/backend-map/sections/` against ground truth enumerated from source (route files, schema tables, connectors).
**Method:** Glob/Grep enumeration of source → per-item cross-reference against the section files.
---
## 1. Coverage Summary Table
| Area | Expected (ground truth) | Documented? | Gaps |
|---|---|---|---|
| Local sidecar route files (`packages/server/src/local/routes/*.ts`) | 65 files | **65/65 endpoint-documented** | ✅ Closed 2026-06-06 — `ingest.ts` → 03b, `workflows.ts` → 03e |
| Cloud route files (`packages/server/src/routes/*.ts`) | 14 files | **6/14 endpoint-documented** | 4 documented in 03g (agents, jobs, scout, suggestions); 8 only named in the registration list (not endpoint-mapped) |
| Memory tables (`hive-mind-core/src/mind/schema.ts`, `CREATE TABLE`) | 14 base tables | **14/14** | None — fully covered in 02a |
| Relational tables (`server/src/db/schema.ts`, `pgTable(`) | 20 tables | **20/20** | None — fully covered in 02b |
| Connectors (`packages/agent/src/connectors/*.ts`) | 29 impls (+ `index.ts` barrel = 31 files) | **30 registered connectors covered in 05f §8.3** | None material — see §5 |
**Overall coverage estimate: ~96%** (was ~94% before the 2026-06-06 gap-close of `ingest.ts` + `workflows.ts`; the local-sidecar surface — the primary frontend-rebuild target — is now 65/65 = 100% endpoint-documented).
---
## 2. Local Routes — Full Cross-Reference (65 files)
Ground truth: 65 `.ts` files in `packages/server/src/local/routes/`. Of these, several are **helper modules with no HTTP handlers** (correctly NOT documented as standalone route surfaces), and 2 expose real endpoints that are **undocumented at the endpoint level**.
### 2.1 Documented route files (endpoints mapped in a section)
`agent.ts`, `agent-groups.ts`, `agent-run.ts` → **03a**
`approval.ts` → **03a / 03e**
`chat.ts` → **03a**
`commands.ts` → **03a**
`sessions.ts` → **03a**
`memory.ts`, `knowledge.ts`, `wiki.ts`, `harvest.ts`, `import.ts`, `identity.ts`, `mind.ts`, `documents.ts`, `data-erase.ts` → **03b**
`workspaces.ts`, `workspace-templates.ts`, `team.ts`, `personas.ts`, `settings.ts`, `profile.ts`, `pins.ts` → **03c**
`marketplace.ts`, `marketplace-dev.ts`, `skills.ts`, `connectors.ts`, `tools.ts`, `oauth.ts`, `vault.ts`, `providers.ts` → **03d** (connectors/capabilities also in **05f**)
`evolution.ts`, `feedback.ts`, `telemetry.ts`, `compliance.ts`, `cost.ts`, `capabilities.ts` → **03e** (capabilities also **05f §11**)
`waggle-signals.ts`, `waggle-dance.ts`, `events.ts`, `cron.ts`, `notifications.ts`, `offline.ts`, `backup.ts`, `fleet.ts`, `litellm.ts`, `local-inference.ts`, `anthropic-proxy.ts`, `browse.ts`, `browser-ext.ts`, `telegram.ts` → **03f**
`weaver.ts`, `tasks.ts`, `files.ts`, `export.ts` → **04-feature-map** (endpoint tables) + named in 03g registration list
### 2.2 Helper modules — no HTTP routes (correctly NOT standalone-documented)
`validate.ts` (path-traversal guards — explicitly noted in 03e §1), `session-utils.ts` (session shapes — surfaced in 03a §6), `chat-context.ts`, `chat-helpers.ts`, `chat-persistence.ts`, `chat-governance.ts` (chat-loop internals — 03a §9 references governance), `browse-helpers.ts`, `workspace-context.ts`, `workspace-templates.ts` helper bits. These are internal modules, not API surfaces; their absence from endpoint tables is correct.
> Note: `chat-governance.ts` and `session-utils.ts` are referenced in 03a but not given their own heading — acceptable since they are not route files.
### 2.3 GAPS — route files with real endpoints but NO endpoint-level docs
| File | Endpoint(s) | Status |
|---|---|---|
| **`ingest.ts`** | `POST /api/ingest` (base64 file ingestion: images, pdf/docx/pptx, xlsx/csv, code/text, zip listing) | **UNDOCUMENTED.** `ingestRoutes` appears only in the 03g registration list. 04-feature-map references `/api/ingest` timeout behavior in prose but never documents the endpoint, body shape, or response. The browser-ext note in 03f says "ingest flows reuse `/api/memory/frames`" — which is a *different* path and does not cover `POST /api/ingest`. |
| **`workflows.ts`** | `GET /api/workflows`, `POST /api/workflows`, `DELETE /api/workflows/:name` | **UNDOCUMENTED.** `workflowRoutes` appears only in the 03g registration list. Workflow *templates/steps* are described conceptually in 03e/05e, but these three CRUD endpoints are never mapped (method/path/body/response). |
---
## 3. Cloud Routes — Cross-Reference (14 files)
Ground truth: 14 `.ts` files in `packages/server/src/routes/` (the multi-tenant/Clerk-auth server).
| Cloud route file | Documented? | Where |
|---|---|---|
| `agents.ts` | Yes (endpoint table) | 03g §4a |
| `jobs.ts` | Yes (endpoint table) | 03g §4b |
| `scout.ts` | Yes (endpoint table) | 03g §4c |
| `suggestions.ts` | Yes (endpoint table) | 03g §4d |
| `teams.ts` | Partial — named in 03g §8 registration list; noted as the "second teamRoutes" in 03c §1.3, but its endpoints are NOT mapped | **GAP (endpoint-level)** |
| `messages.ts` | Partial — named in 03g §8 list only | **GAP** |
| `knowledge.ts` (cloud) | Partial — named in 03g §8 list only | **GAP** |
| `resources.ts` | Partial — named in 03g §8 list only | **GAP** |
| `tasks.ts` (cloud) | Partial — named in 03g §8 list only | **GAP** |
| `cron.ts` (cloud) | Partial — named in 03g §8 list only | **GAP** |
| `audit.ts` | Partial — named in 03g §8 list only | **GAP** |
| `analytics.ts` | Partial — named in 03g §8 list only | **GAP** |
| `capability-governance.ts` | Partial — named in 03g §8 list only | **GAP** |
| `webhooks.ts` | Partial — named in 03g §8 list only | **GAP** |
**Assessment:** 03g §8 enumerates the cloud server's full plugin registration order (so every file is *acknowledged*), and 03g explicitly frames the cloud server as secondary ("the frontend almost always talks to the Local Sidecar"). The 4 prompt-named files (agents/jobs/scout/suggestions) get full endpoint tables; the other 10 are listed but not endpoint-mapped. This is a deliberate scoping decision, but for a strict completeness measure these 8–10 cloud route files are **not endpoint-documented**. Their data shapes ARE largely covered indirectly via the 20 relational tables in 02b (teams, messages, tasks, agent_audit_log, scout_findings, suggestions_log, team_capability_* etc.).
---
## 4. Tables — Cross-Reference
### 4.1 Memory tables (14, `hive-mind-core/src/mind/schema.ts`)
All 14 `CREATE TABLE` statements are documented column-by-column in **02a**:
`meta`, `identity`, `awareness`, `sessions`, `memory_frames`, `knowledge_entities`, `knowledge_relations`, `improvement_signals`, `install_audit`, `procedures`, `ai_interactions`, `execution_traces`, `evolution_runs`, `harvest_sources`.
Plus the 2 virtual tables (`memory_frames_fts`, `memory_frames_vec`) and a note on the out-of-schema `kg_entity_frames` link table. **Coverage: 14/14 (100%).** No gaps.
### 4.2 Relational tables (20, `server/src/db/schema.ts`)
All 20 `pgTable(` definitions are documented column-by-column in **02b**:
`users`, `teams`, `team_members`, `agents`, `agent_groups`, `agent_group_members`, `tasks`, `messages`, `team_entities`, `team_relations`, `team_resources`, `team_capability_policies`, `team_capability_overrides`, `team_capability_requests`, `agent_jobs`, `cron_schedules`, `scout_findings`, `proactive_patterns`, `suggestions_log`, `agent_audit_log`. **Coverage: 20/20 (100%).** No gaps.
---
## 5. Connectors — Cross-Reference
Ground truth: 31 files in `packages/agent/src/connectors/` = 29 connector implementations + `index.ts` (barrel) + (the 29 includes `email-connector.ts` and `postgres-connector.ts`).
**05f §8.3** lists **30 connectors registered at startup** (`setup-connectors.ts → registerConnectors`):
GitHub, Slack, Jira, Email, Google Calendar, Discord, Linear, Asana, Trello, Monday, Notion, Confluence, Obsidian, HubSpot, Salesforce, Pipedrive, Airtable, GitLab, Bitbucket, Dropbox, Postgres, Gmail, Google Docs, Google Drive, Google Sheets, MS Teams, Outlook, OneDrive, OneNote, Composio.
**Reconciliation of counts:**
- 29 connector impl files map to 30 registered connectors. The discrepancy is **"Google Calendar"** — it is registered as a connector but is provided by the `gcal-connector.ts` file (one of the 29). All 29 impl files have a corresponding registered connector; the 30 registered count includes connectors whose impls share files / are SDK-defined.
- `index.ts` is a barrel (not a connector) — correctly excluded.
**Assessment:** Connectors are **well covered** in 05f — the registry, SDK, per-connector tool generation (`connector_<id>_<action>`), the full 30-name registration list, the `ConnectorDefinition`/`ConnectorHealth` shapes, and the 4 HTTP routes (`/api/connectors*`, also in 03d §3). The MCP catalog (~140 entries) is also covered (05f §12). **No material gap.** Minor: the doc says "30 connectors" without reconciling against the 29 impl files / `gcal` naming — a 1-line clarification would help but is not a coverage gap.
---
## 6. Subsystem Sections (05x) — Spot Check
All major subsystems have a dedicated section:
- 05a agent-runtime, 05b memory, 05c harvest, 05d evolution, 05e waggledance/AI-OS, 05f capabilities/connectors/tiers, 05g skills/marketplace/wiki.
No subsystem appears unrepresented. `weaver` (memory consolidation) is the lightest-covered subsystem — it has endpoints in 04-feature-map (`/api/weaver/status`, `/api/weaver/trigger`) but no dedicated subsystem deep-dive; acceptable given its small surface.
---
## 7. Findings — Prioritized
| # | Severity | Gap | Fix |
|---|---|---|---|
| 1 | ~~Medium~~ ✅ **RESOLVED 2026-06-06** | `POST /api/ingest` (`ingest.ts`) — multi-format file ingestion | Documented in **03b** ("File Ingestion") — body, result shape, validation/413, supported types, registry + memory-frame side effects |
| 2 | ~~Medium~~ ✅ **RESOLVED 2026-06-06** | `/api/workflows` CRUD (`workflows.ts`) — 3 endpoints | Documented in **03e** ("Workflow Templates CRUD") — GET/POST/DELETE table + `WorkflowTemplate` shape |
| 3 | **Low** | 8–10 cloud route files (`teams`, `messages`, `knowledge`, `resources`, `tasks`, `cron`, `audit`, `analytics`, `capability-governance`, `webhooks`) named in registration list but not endpoint-mapped | Either explicitly mark cloud server as out-of-scope for the frontend rebuild, or add a thin endpoint table per file (shapes already inferable from 02b) |
| 4 | **Trivial** | Connector count "30" not reconciled with 29 impl files / `gcal` naming | Add a 1-line note |
---
## 8. Overall Coverage
| Dimension | Score |
|---|---|
| Memory tables (02a) | 100% (14/14) |
| Relational tables (02b) | 100% (20/20) |
| Local routes (endpoint-level) | **100% (65/65 files)** — `ingest` + `workflows` closed 2026-06-06 |
| Cloud routes (endpoint-level) | ~43% (6/14 endpoint-mapped; rest registration-listed only — deliberately secondary) |
| Connectors / capabilities / tiers (05f) | ~98% |
| Subsystems (05x) | 100% (all represented) |
**Weighted overall coverage estimate: ~96%** (post 2026-06-06 gap-close).
The local-sidecar surface (the primary frontend-rebuild target) is now **complete (65/65 endpoint-documented)** — `POST /api/ingest` and the `/api/workflows` CRUD trio were the last two gaps and are now documented in 03b and 03e respectively. The data model is 100% covered. The remaining shortfall is the **cloud/multi-tenant server**, where 8–10 route files are acknowledged in the registration list but not endpoint-mapped; this is a deliberate scoping choice (cloud is secondary to the sidecar for a frontend rebuild) and the residual ~4% lives almost entirely there. Their data shapes are inferable from the 20 relational tables in 02b.

147
docs/backend-map/DIAGRAMS/01-system-architecture.md Normal file
View File

@@ -0,0 +1,147 @@
# System Architecture (Layers)
This diagram is the cross-cutting runtime map for a Lovable rebuild of the Waggle OS frontend. It traces one request from the Tauri Rust shell, through the `apps/web` React SPA, into the Fastify **Local Sidecar** (port 3333) that serves the SPA and exposes all `/api/*` routes, down through the workspace packages that own each responsibility, into the data stores (per-workspace SQLite `*.mind` files via better-sqlite3 + sqlite-vec, plus team/cloud Postgres + Redis via the separate Cloud Server on port 3100), and out to the LiteLLM router and the LLM providers. Every node and ownership label is grounded in the section files. Key facts to anchor on: the frontend origin **is** the sidecar (same-origin, two-step Bearer-token auth), the sidecar uses local SQLite while the optional Cloud Server uses Postgres + Clerk + Redis, and the memory substrate (`mind` + `harvest`) lives in `@waggle/hive-mind-core`, not `@waggle/core`.
## Full Runtime Stack
```mermaid
flowchart TD
subgraph Shell["Desktop shell — app/ (Tauri 2.0, Rust)"]
TAURI["Tauri WebView\nloads apps/web dist from sidecar\norigin = tauri://localhost"]
end
subgraph Web["Frontend — apps/web (React 19 + Vite + Tailwind 4)"]
SPA["SPA shell + assets\nauth-exempt GET"]
AUTH2["Two-step auth\n1. GET /api/auth/session-token\n2. Authorization Bearer on every /api/*"]
APPS["OS apps + overlays\nChat · Memory{Harvest,Evolution,Wiki}\nMarketplace · Connectors · Launcher\nWaggleDance · Room · Mission Control"]
WS["WebSocket client\nGET /ws?token="]
end
subgraph Sidecar["Local Sidecar :3333 — packages/server (Fastify, Node, bundled in Tauri)"]
SEC["securityMiddleware\nHost allowlist · Bearer · CORS · RateLimit · CSP"]
STATIC["@fastify/static + SPA fallback\nserves apps/web dist"]
ROUTES["62 route plugins, flat /api/* paths\nchat · memory · harvest · evolution · skills\nmarketplace · connectors · wiki · tools\nwaggle-dance · stripe · vault · personas"]
WSL["/ws event-bus relay\napprovals · steps · tools · notifications"]
end
subgraph Packages["Workspace packages (responsibilities)"]
AGENT["@waggle/agent\nrunAgentLoop · Orchestrator · personas\ntool-executor · completion gates · cost-tracker\nconnectors · capability/trust · KVARK tools"]
HMC["@waggle/hive-mind-core\nmind: Identity·Awareness·Frames·KnowledgeGraph\nHybridSearch·Cognify·embeddings\nharvest: source adapters + 4-pass pipeline"]
SHARED["@waggle/shared\ntypes · zod schemas · TIERS\nmcp-catalog · tool-detection"]
DANCE["@waggle/waggle-dance\nWaggleMessage protocol\ndispatcher · SignalBus"]
MKT["@waggle/marketplace\ncatalog · installer · SecurityGate · sync"]
WIKI["@waggle/wiki-compiler\nentity/concept/synthesis/index/health pages"]
OPT["@waggle/optimizer\nself-evolution: GEPA + EvolveSchema\njudge · gates · evolution runs"]
end
subgraph LocalData["Per-workspace local data (better-sqlite3 + sqlite-vec)"]
MIND[("*.mind SQLite per workspace\nmemory_frames + _fts + _vec float[1024]\nknowledge_entities/relations · sessions\nharvest_runs · evolution_runs · wiki_pages")]
CFG[("~/.waggle/config.json\ntier · stripe_customer_id")]
VAULT[("Vault\napi keys · connector creds · kvark:connection")]
MKTDB[("marketplace.db SQLite")]
SKILLS[("~/.waggle/skills/*.md\nplugins · hooks.json")]
end
subgraph Cloud["Optional Cloud Server :3100 — packages/server (team/SaaS only)"]
AUTHP["authPlugin — Clerk JWT verify"]
CROUTES["teams · agents · jobs · scout\nsuggestions · messages · audit · analytics"]
WSG["/ws gateway — team chat"]
PG[("Postgres via drizzle\nDATABASE_URL")]
REDIS[("Redis pub/sub\nteam:*:waggle · job:*:progress")]
end
LITELLM["LiteLLM router\nlitellm-config.yaml\nPOST /chat/completions"]
PROVIDERS["LLM providers\nAnthropic · OpenAI · Ollama (local)\nVoyage · others"]
STRIPEAPI["Stripe API"]
KVARK["KVARK sovereign FastAPI\nsearch · ask · actions (TEAMS/ENTERPRISE)"]
TAURI --> SPA
SPA --> AUTH2
AUTH2 --> APPS
APPS -->|"Bearer /api/*"| SEC
WS -->|"?token="| WSL
SPA -.->|"shell + assets"| STATIC
SEC --> ROUTES
ROUTES --> AGENT
ROUTES --> HMC
ROUTES --> DANCE
ROUTES --> MKT
ROUTES --> WIKI
ROUTES --> OPT
ROUTES --> SHARED
WSL --> AGENT
AGENT --> SHARED
HMC --> SHARED
OPT --> AGENT
WIKI --> HMC
MKT --> SKILLS
AGENT -->|"recall/write memory"| HMC
HMC --> MIND
AGENT --> VAULT
ROUTES --> CFG
MKT --> MKTDB
AGENT --> SKILLS
AGENT -->|"POST /chat/completions"| LITELLM
OPT -->|"judge + mutate (Haiku)"| LITELLM
HMC -->|"embeddings (litellm/ollama)"| LITELLM
WIKI -->|"synthesize (Haiku/Ollama)"| LITELLM
LITELLM --> PROVIDERS
ROUTES -->|"stripeRoutes"| STRIPEAPI
STRIPEAPI -->|"webhook -> write tier"| CFG
VAULT -->|"KvarkClient"| KVARK
APPS -. "team mode only" .-> AUTHP
AUTHP --> CROUTES
CROUTES --> PG
CROUTES --> REDIS
WSG <--> REDIS
```
## Request / Auth Path (Local Sidecar)
This is the bootstrap handshake the rebuilt frontend must reproduce. The API root is the page's own origin; there is no separate host to configure.
```mermaid
flowchart LR
A["1. Load shell\nauth-exempt GET"] --> B["2. GET /api/auth/session-token\nsame-origin gated"]
B --> C["token"]
C --> D["3. Authorization Bearer token\non every /api/* call"]
C --> E["4. GET /ws?token=\nWebSocket relay"]
D --> F["Sidecar route plugins\nrate-limited, tier-gated"]
F -->|"403 TIER_INSUFFICIENT"| G["render upgrade prompt\nrequired + upgradeUrl"]
F -->|"429 + Retry-After"| H["backoff and retry"]
```
## Package Ownership Map
Which package owns which responsibility, distilled from the subsystem sections.
```mermaid
flowchart TD
R["Fastify sidecar routes\npackages/server"] --> A
R --> H
R --> D
R --> M
R --> W
R --> O
A["@waggle/agent\nagent loop · orchestrator · 22 personas\ntool execution + 11-step chain\ncompletion gates D3/D4/D1\nconnectors (30) · capability + trust\nautonomy gating · KVARK tools · cost"]
H["@waggle/hive-mind-core\nMIND: identity · awareness · frames\nknowledge graph · hybrid search\nrelevance scoring · cognify · embeddings\nHARVEST: adapters · 4-pass pipeline · dedup"]
D["@waggle/waggle-dance\nWaggleMessage protocol\ndispatcher + combo validation\nSignalBus ring buffer · AI-OS signals"]
M["@waggle/marketplace\nSQLite catalog · installer\nSecurityGate (4 layers) · 9 sync adapters"]
W["@waggle/wiki-compiler\n5 page types from memory\nincremental compile · Obsidian/Notion export"]
O["@waggle/optimizer\nGEPA + EvolveSchema + judge\nevolution gates · run store · deploy"]
S["@waggle/shared\nwire types · zod schemas\n5-tier model + capabilities\nMCP catalog · tool detection"]
A --> S
H --> S
D --> S
M --> S
W --> S
O --> S
```

466
docs/backend-map/DIAGRAMS/02-master-er.md Normal file
View File

@@ -0,0 +1,466 @@
# Master Data Model (two ER diagrams)
Waggle OS keeps data in two physically separate stores. The **per-workspace memory layer** is a single SQLite file (`*.mind`) per workspace — 14 base tables plus 2 virtual search tables — holding one user's private frames, knowledge graph, identity, awareness, and audit. The **team/cloud relational layer** is a shared PostgreSQL database (20 tables, Drizzle ORM) holding users, teams, agents, tasks, jobs, and governance. The two stores share **no cross-database foreign keys**; the only bridge is the `users.mind_path` text column in Postgres, which points at where a given user's local SQLite `*.mind` file lives. The note after the diagrams explains how one workspace mind relates to a user/team row.
---
## Diagram 1 — Per-workspace memory layer (SQLite, one `*.mind` file)
One isolated SQLite file per workspace. Schema version `'1'`. 14 base tables plus `memory_frames_fts` (FTS5 keyword index) and `memory_frames_vec` (vec0, 1024-dim embeddings), both keyed by `rowid = memory_frames.id`. Foreign keys exist only inside this file (frames to sessions, frames self-ref, relations to entities). `ai_interactions` is append-only (DB triggers reject UPDATE/DELETE). The knowledge graph is bitemporal: active rows have `valid_to IS NULL`.
```mermaid
erDiagram
sessions ||--o{ memory_frames : "gop_id FK"
memory_frames ||--o{ memory_frames : "base_frame_id self-FK"
memory_frames ||--|| memory_frames_fts : "rowid=id FTS5"
memory_frames ||--|| memory_frames_vec : "rowid=id vec0"
knowledge_entities ||--o{ knowledge_relations : "source_id FK"
knowledge_entities ||--o{ knowledge_relations : "target_id FK"
meta {
TEXT key PK
TEXT value
}
identity {
INTEGER id PK "CHECK id=1, single row"
TEXT name
TEXT role
TEXT department
TEXT personality
TEXT capabilities
TEXT system_prompt
TEXT created_at
TEXT updated_at
}
awareness {
INTEGER id PK
TEXT category "task action pending flag"
TEXT content
INTEGER priority
TEXT metadata "JSON"
TEXT created_at
TEXT expires_at "nullable"
}
sessions {
INTEGER id PK
TEXT gop_id UK "join key for frames"
TEXT project_id "nullable"
TEXT status "active closed archived"
TEXT started_at
TEXT ended_at "nullable"
TEXT summary "nullable"
}
memory_frames {
INTEGER id PK
TEXT frame_type "I P B"
TEXT gop_id FK
INTEGER t "per-gop ordinal"
INTEGER base_frame_id FK "nullable self"
TEXT content "JSON for B-frames"
TEXT importance "critical..deprecated"
TEXT source "user_stated..system"
INTEGER access_count
TEXT created_at
TEXT last_accessed
}
memory_frames_fts {
TEXT content "FTS5 rowid=id"
}
memory_frames_vec {
FLOAT embedding "float 1024 rowid=id"
}
knowledge_entities {
INTEGER id PK
TEXT entity_type
TEXT name
TEXT properties "JSON"
TEXT valid_from
TEXT valid_to "nullable=active"
TEXT recorded_at
}
knowledge_relations {
INTEGER id PK
INTEGER source_id FK
INTEGER target_id FK
TEXT relation_type
REAL confidence
TEXT properties "JSON"
TEXT valid_from
TEXT valid_to "nullable=active"
TEXT recorded_at
}
procedures {
INTEGER id PK
TEXT name
TEXT model
TEXT template
INTEGER version
REAL success_rate
REAL avg_cost
TEXT created_at
TEXT updated_at
}
improvement_signals {
INTEGER id PK
TEXT category "capability_gap..skill_promotion"
TEXT pattern_key
TEXT detail
INTEGER count
TEXT first_seen
TEXT last_seen
INTEGER surfaced "0 or 1"
TEXT surfaced_at "nullable"
TEXT metadata "JSON"
}
install_audit {
INTEGER id PK
TEXT timestamp
TEXT capability_name
TEXT capability_type "native..marketplace"
TEXT source
TEXT version "nullable"
TEXT risk_level "low medium high"
TEXT trust_source
TEXT approval_class
TEXT action
TEXT initiator "agent user system"
TEXT detail
}
ai_interactions {
INTEGER id PK "APPEND-ONLY"
TEXT timestamp
TEXT workspace_id "nullable"
TEXT session_id "nullable"
TEXT model
TEXT provider
INTEGER input_tokens
INTEGER output_tokens
REAL cost_usd
TEXT tools_called "JSON"
TEXT human_action "nullable"
TEXT risk_context "nullable"
TEXT imported_from "nullable"
TEXT persona "nullable"
TEXT input_text "nullable"
TEXT output_text "nullable"
}
execution_traces {
INTEGER id PK
TEXT session_id "nullable"
TEXT persona_id "nullable"
TEXT workspace_id "nullable"
TEXT model "nullable"
TEXT task_shape "nullable"
TEXT outcome "success..pending"
TEXT trace_json "JSON"
REAL cost_usd
INTEGER duration_ms
TEXT created_at
TEXT finalized_at "nullable"
}
evolution_runs {
INTEGER id PK
TEXT run_uuid UK
TEXT target_kind
TEXT target_name "nullable"
TEXT baseline_text
TEXT winner_text
TEXT winner_schema_json "nullable"
REAL delta_accuracy
TEXT gate_verdict "pass fail"
TEXT gate_reasons_json "JSON"
TEXT status "proposed..failed"
TEXT artifacts_json "nullable"
TEXT user_note "nullable"
TEXT failure_reason "nullable"
TEXT created_at
TEXT decided_at "nullable"
TEXT deployed_at "nullable"
}
harvest_sources {
INTEGER id PK
TEXT source UK
TEXT display_name
TEXT source_path "nullable"
TEXT last_synced_at "nullable"
INTEGER items_imported
INTEGER frames_created
INTEGER auto_sync "0 or 1"
INTEGER sync_interval_hours
TEXT last_content_hash "nullable"
TEXT created_at
}
```
> Standalone tables (no DB-level FKs to others): `meta`, `identity`, `awareness`, `procedures`, `improvement_signals`, `install_audit`, `ai_interactions`, `execution_traces`, `evolution_runs`, `harvest_sources`. Their `session_id` / `workspace_id` columns are plain TEXT, not enforced foreign keys. A `kg_entity_frames` link table is referenced by `frames.delete()` but is not defined in `schema.ts`, so it may be absent — omitted here.
---
## Diagram 2 — Team/cloud relational layer (PostgreSQL + Drizzle)
Shared multi-user store. All primary keys are server-generated UUIDs (`gen_random_uuid()`); all timestamps are `timestamp with time zone`. Every foreign key is `ON DELETE no action` — there are **no cascades**, so deleting a parent is blocked while children reference it. Two junction tables use composite PKs (`team_members`, `agent_group_members`). `users.mind_path` is the only pointer out to the SQLite memory layer.
```mermaid
erDiagram
users ||--o{ teams : "owns owner_id"
users ||--o{ team_members : "is"
teams ||--o{ team_members : "has"
users ||--o{ agents : "owns user_id"
teams ||--o{ agents : "scopes team_id"
users ||--o{ agent_groups : "owns"
agent_groups ||--o{ agent_group_members : "contains"
agents ||--o{ agent_group_members : "joins"
teams ||--o{ tasks : "has"
users ||--o{ tasks : "creates created_by"
users ||--o{ tasks : "assigned assigned_to"
teams ||--o{ messages : "channel"
users ||--o{ messages : "sends sender_id"
teams ||--o{ team_entities : "owns"
users ||--o{ team_entities : "shares shared_by"
teams ||--o{ team_relations : "owns"
team_entities ||--o{ team_relations : "source_id"
team_entities ||--o{ team_relations : "target_id"
teams ||--o{ team_resources : "owns"
users ||--o{ team_resources : "shares"
teams ||--o{ team_capability_policies : "governs"
users ||--o{ team_capability_policies : "updates"
teams ||--o{ team_capability_overrides : "governs"
users ||--o{ team_capability_overrides : "decides"
teams ||--o{ team_capability_requests : "scopes"
users ||--o{ team_capability_requests : "requests-decides"
teams ||--o{ agent_jobs : "owns"
users ||--o{ agent_jobs : "runs"
teams ||--o{ cron_schedules : "owns"
users ||--o{ cron_schedules : "creates"
users ||--o{ scout_findings : "for-user"
teams ||--o{ scout_findings : "for-team"
proactive_patterns ||--o{ suggestions_log : "fires"
users ||--o{ suggestions_log : "receives"
users ||--o{ agent_audit_log : "acts"
teams ||--o{ agent_audit_log : "scopes"
users {
uuid id PK
text clerk_id UK
text display_name
text email UK
text avatar_url "nullable"
text mind_path "nullable to SQLite mind"
timestamptz created_at
timestamptz updated_at
}
teams {
uuid id PK
text name
text slug UK
uuid owner_id FK
timestamptz created_at
}
team_members {
uuid team_id PK_FK
uuid user_id PK_FK
text role "default member"
text role_description "nullable"
jsonb interests "nullable"
timestamptz joined_at
}
agents {
uuid id PK
uuid user_id FK
uuid team_id FK "nullable"
text name
text role "nullable"
text system_prompt "nullable"
text model "default claude-haiku-4-5"
jsonb tools "default empty"
jsonb config "default empty"
timestamptz created_at
}
agent_groups {
uuid id PK
uuid user_id FK
text name
text description "nullable"
text strategy "default parallel"
timestamptz created_at
}
agent_group_members {
uuid group_id PK_FK
uuid agent_id PK_FK
text role_in_group "default worker"
integer execution_order "default 0"
}
tasks {
uuid id PK
uuid team_id FK
text title
text description "nullable"
text status "default open"
text priority "default normal"
uuid created_by FK
uuid assigned_to FK "nullable"
uuid parent_task_id "no FK logical self-ref"
timestamptz created_at
timestamptz updated_at
}
messages {
uuid id PK
uuid team_id FK
uuid sender_id FK
text type
text subtype
jsonb content
uuid reference_id "no FK"
jsonb routing "nullable"
timestamptz created_at
}
team_entities {
uuid id PK
uuid team_id FK
text entity_type
text name
jsonb properties "default empty"
uuid shared_by FK
timestamptz valid_from
timestamptz valid_to "nullable open-ended"
timestamptz created_at
}
team_relations {
uuid id PK
uuid team_id FK
uuid source_id FK
uuid target_id FK
text relation_type
real confidence "default 1.0"
jsonb properties "default empty"
timestamptz created_at
}
team_resources {
uuid id PK
uuid team_id FK
text resource_type
text name
text description "nullable"
jsonb config
uuid shared_by FK
real rating "default 0"
integer use_count "default 0"
timestamptz created_at
}
team_capability_policies {
uuid id PK
uuid team_id FK
text role
jsonb allowed_sources "default empty"
jsonb blocked_tools "default empty"
text approval_threshold "default none"
uuid updated_by FK "nullable"
timestamptz created_at
timestamptz updated_at
}
team_capability_overrides {
uuid id PK
uuid team_id FK
text capability_name
text capability_type
text decision
text reason "default empty"
uuid decided_by FK
timestamptz created_at
timestamptz decided_at
}
team_capability_requests {
uuid id PK
uuid team_id FK
uuid requested_by FK
text capability_name
text capability_type
text justification
text status "default pending"
uuid decided_by FK "nullable"
text decision_reason "nullable"
timestamptz created_at
timestamptz decided_at "nullable"
}
agent_jobs {
uuid id PK
uuid team_id FK
uuid user_id FK
text job_type
text status "default queued"
jsonb input
jsonb output "nullable"
timestamptz started_at "nullable"
timestamptz completed_at "nullable"
timestamptz created_at
}
cron_schedules {
uuid id PK
uuid team_id FK
uuid created_by FK
text name
text cron_expr
text job_type
jsonb job_config "default empty"
boolean enabled "default true"
timestamptz last_run_at "nullable"
timestamptz next_run_at "nullable"
timestamptz created_at
}
scout_findings {
uuid id PK
uuid user_id FK "nullable"
uuid team_id FK "nullable"
text source
text category
text title
text summary "nullable"
real relevance_score "default 0"
text url "nullable"
text status "default new"
timestamptz created_at
}
proactive_patterns {
uuid id PK "config-only no FK"
text name
jsonb trigger
text suggestion_type
text template
boolean enabled "default true"
}
suggestions_log {
uuid id PK
uuid user_id FK
uuid pattern_id FK
jsonb context
text status "default pending"
timestamptz created_at
}
agent_audit_log {
uuid id PK
uuid user_id FK
uuid team_id FK "nullable"
text agent_name
text action_type
text description
jsonb before_state "nullable"
jsonb after_state "nullable"
boolean requires_approval "default false"
boolean approved "nullable tri-state"
uuid approved_by FK "nullable"
timestamptz created_at
}
```
---
## How a workspace mind relates to a user/team row
A **workspace** is, physically, one SQLite `*.mind` file on the local machine (e.g. `personal.mind`). It is self-contained: switching workspace means opening a different file, and nothing inside that file joins across workspaces. There is no `workspace` table in Postgres — workspaces live as files, not relational rows.
The single connection point between the two layers is the **`users.mind_path`** column in Postgres: a nullable `text` pointer to where that user's private SQLite mind file lives. There are **no cross-database foreign keys**; the relationship is resolved only in application code. Concretely:
- A **user** row in Postgres (`users.id`, Clerk-backed via `clerk_id`) owns at most one `mind_path`. Following that path opens the user's personal `*.mind` SQLite file (Diagram 1), whose `identity` table (single CHECK-pinned row) describes that same user from the memory side.
- Inside the mind file, `ai_interactions.workspace_id` and `execution_traces.workspace_id` are plain TEXT labels for the originating workspace — they are **not** foreign keys to anything in Postgres. The same is true of `session_id`: it is a logical label, not an enforced cross-DB reference.
- **Team-shared knowledge** is duplicated in concept but not in storage: each user keeps a private SQLite `knowledge_entities` / `knowledge_relations` graph (Diagram 1), while the team keeps a separate Postgres `team_entities` / `team_relations` graph (Diagram 2). They are distinct tables in distinct engines; promoting a private fact to the team graph is an application-level copy, not a join.
- The relational layer **never stores memory frames**. All `memory_frames`, embeddings, awareness, and harvest tracking stay local in the workspace mind file; Postgres only holds the team/collaboration/governance metadata and the `mind_path` breadcrumb back to each local file.

80
docs/backend-map/DIAGRAMS/03-chat-turn-sequence.md Normal file
View File

@@ -0,0 +1,80 @@
# Chat Turn Sequence
This diagram traces one conversational chat turn end to end: the frontend POSTs to the real endpoint `POST /api/chat`, which is an SSE stream (the server validates, then calls `reply.hijack()` and writes a raw `text/event-stream`). The route handler (`packages/server/src/local/routes/chat.ts`) assembles the layered system prompt via the per-session `Orchestrator` (`buildSystemPrompt` + `recallMemory`, where recall runs `HybridSearch` over the workspace mind), filters tools by persona/context, then calls `runAgentLoop`, which POSTs to LiteLLM at `${litellmUrl}/chat/completions`. Tool calls pass through the 11-step `executeToolCall` middleware chain (governance, hooks, LoopGuard, injection scan); a `TraceRecorder` wires additive trace callbacks. Tokens and tool events stream back to the UI as named SSE events, and after the loop `autoSaveFromExchange` runs the `CognifyPipeline` to write new memory frames. All claims are grounded in `sections/03a-api-chat-agents.md`, `sections/05a-subsystem-agent-runtime.md`, and `sections/05b-subsystem-memory.md`.
```mermaid
sequenceDiagram
autonumber
participant UI as "Frontend (chat UI)"
participant Route as "POST /api/chat\nchat.ts (SSE)"
participant Orch as "Orchestrator\nbuildSystemPrompt + recallMemory"
participant Search as "HybridSearch\n(workspace mind)"
participant Loop as "runAgentLoop\nagent-loop.ts"
participant LLM as "LiteLLM\n/chat/completions"
participant Tools as "executeToolCall\n11-step chain"
participant Trace as "TraceRecorder"
participant Cog as "CognifyPipeline\nautoSaveFromExchange"
participant Disk as "session .jsonl"
UI->>Route: "POST { message, workspace, session, model?, persona?, autonomy? }"
Note over Route: "validate · injection scan score < 0.7 · RBAC · path guard (all pre-hijack)"
alt rejected
Route-->>UI: "400 / 403 JSON error (MESSAGE_TOO_LONG, INJECTION_DETECTED, ...)"
else accepted
Note over Route: "reply.hijack() · write SSE headers · wire AbortController to client close"
Route->>Disk: "persistMessage user turn (append .jsonl)"
Route->>Route: "generateTurnId UUID v4 · resolve model fallback chain"
Route->>Orch: "recallMemory(query, turnId)"
Note over Orch: "catch-up vs semantic · drop temporary/deprecated · injection scan"
Orch->>Search: "search(query) keyword + vector"
Search-->>Orch: "SearchResult[] ranked by finalScore = rrfScore * relevanceScore"
Orch-->>Route: "recalledContext text"
Route-->>UI: "event: step Recalling relevant memories..."
Route-->>UI: "event: tool / tool_result (auto_recall)"
Route->>Orch: "buildSystemPrompt()"
Orch-->>Route: "identity + self-awareness + preloaded context"
Note over Route: "layer profile + runtime facts + activeSpec.rules + skills + Workspace Now + corrections, then composePersonaPrompt"
Note over Route: "filterToolsForContext + filterAvailableTools + persona allow/deny"
Route->>Loop: "runAgentLoop { systemPrompt, tools, messages, stream:true, maxTurns:200, turnId, traceRecording }"
loop "each turn up to maxTurns"
Loop->>LLM: "POST /chat/completions { model, messages, tools, stream }"
Note over Loop,LLM: "signal = client-abort + 300s timeout · 429/5xx -> backoff, turn--, retry"
LLM-->>Loop: "stream chunks: content + tool_calls + usage"
Loop-->>UI: "event: token xN assistant text"
alt "tool_calls present"
Loop-->>UI: "event: tool { name, input }"
Loop->>Tools: "executeToolCall(name, args)"
Note over Tools: "parse args -> onToolUse -> governance blockedTools -> pre:tool -> pre:memory-write -> LoopGuard.check -> execute -> scanForInjection -> onToolResult -> post hooks"
opt "gated tool"
Loop-->>UI: "event: approval_required { requestId, toolName, input }"
UI->>Route: "POST /api/approval/:requestId { approved, always? }"
Route-->>Loop: "resolve(approved) (auto-deny after 5 min)"
end
Tools->>Trace: "record tool call + sanitized result"
Tools-->>Loop: "role:tool result message (sanitized)"
Loop-->>UI: "event: tool_result { name, result, isError }"
else "no tool_calls (final answer)"
Note over Loop: "maybeFireCompletionGate: D3 verification -> D4 phantom-write -> D1 skill-distillation"
alt "a gate fired"
Note over Loop: "push corrective directive, continue one more turn"
else "none fired"
Loop-->>Trace: "finalize trace"
Loop-->>Route: "AgentResponse { content, toolsUsed, usage }"
end
end
end
Note over Route: "cost tracking · KG entity extraction · disclaimers"
Route->>Cog: "autoSaveFromExchange(message, result.content)"
Cog->>Disk: "cognify writes new memory frame (I/P) + FTS + vec index + entities"
Route->>Orch: "commitSurfacedSignals()"
Route->>Disk: "persistMessage assistant turn"
Route-->>UI: "event: done { content, usage, toolsUsed, model, cost? }"
end
Note over Route: "on failure -> event: error { message } (raw turn still persisted) · finally: raw.end()"
```

218
docs/backend-map/DIAGRAMS/04-feature-api-map.md Normal file
View File

@@ -0,0 +1,218 @@
# Feature -> API Map (rebuild blueprint)
This diagram is the rebuild contract for a Lovable reconstruction of the Waggle OS web client. The left column lists every implemented OS app and overlay (windows opened by `appId` plus the modals/rails `Desktop.tsx` renders). The right column lists the backend endpoint GROUPS each one depends on, grouped exactly as the section files split them (03a Chat/Agents, 03b Memory/Knowledge/Wiki/Harvest, 03c Workspace/Team/Persona/Settings/Profile/Pins, 03d Marketplace/Skills/Connectors/Tools/Vault/Providers, 03e Evolution/Governance/Compliance/Costs/Approvals, 03f Realtime/Ops, 03g Billing/Stripe/KVARK). Every app first goes through the single shared `adapter` singleton and `ServiceProvider`; auth and tier-gating are cross-cutting and apply to all calls. Edges are sourced verbatim from section 04 "Feature -> UI Component -> Backend Endpoints" (and §3 endpoint reference). Apps with no backend calls (Voice, Team Governance, Keyboard Shortcuts) are shown wired only to the shared layer.
## 1. Apps and overlays mapped to endpoint groups
```mermaid
flowchart LR
classDef app fill:#1b2330,stroke:#a78bfa,color:#e8e8ef
classDef ovl fill:#23202e,stroke:#e5a000,color:#f3ead0
classDef shared fill:#0f1622,stroke:#7dd3fc,color:#dff1ff
classDef grp fill:#10271b,stroke:#34d399,color:#d7f7e6
%% Shared layer every feature passes through
ADAPTER["adapter singleton\nlib/adapter.ts\n127.0.0.1:3333"]:::shared
SVC["ServiceProvider\nadapter.connect"]:::shared
AUTH["Auth + Health\nGET /api/auth/session-token\nGET /health"]:::shared
TIER["Tier gating bus\n403 TIER_INSUFFICIENT\nwaggle:tier-insufficient"]:::shared
SVC --> ADAPTER
ADAPTER --> AUTH
ADAPTER --> TIER
%% Endpoint GROUPS (right side)
G_CHAT["03a Chat + Agents\n/api/chat SSE\n/api/history /api/agent/*\n/api/sessions/* /api/jobs/*"]:::grp
G_MEM["03b Memory + Knowledge\n/api/memory/* /api/identity\n/api/mind/* /api/documents"]:::grp
G_WIKI["03b Wiki + Harvest + Import\n/api/wiki/* /api/harvest/*\n/api/import/*"]:::grp
G_WS["03c Workspace + Templates\n/api/workspaces/* /api/files/*\n/api/workspace-templates/* /api/browse"]:::grp
G_TEAM["03c Team + Persona + Settings + Profile + Pins\n/api/personas/* /api/agent-groups/*\n/api/team/* /api/settings/* /api/profile/* /api/pins"]:::grp
G_MKT["03d Marketplace + Skills + Capabilities\n/api/marketplace/* /api/skills/*\n/api/capabilities/status"]:::grp
G_CONN["03d Connectors + Vault\n/api/connectors/* /api/vault/*"]:::grp
G_TOOLS["03d AI-OS Tool Launcher\n/api/tools/detect|launch|processes|kill|hooks"]:::grp
G_EVO["03e Evolution + Compliance + Costs + Feedback + Approvals\n/api/evolution/* /api/compliance/*\n/api/cost*|/api/costs /api/feedback /api/approval/*"]:::grp
G_RT["03f Realtime + Ops\n/api/events* SSE /api/notifications/*\n/api/cron/* /api/fleet/* /api/litellm/* /api/local-inference/* /api/backup|restore"]:::grp
G_WD["03f WaggleDance signals\n/api/waggle/signals\n/api/waggle/stream SSE"]:::grp
G_BILL["03g Billing + Tier + GDPR\n/api/tier /api/tier/start-trial\n/api/stripe/* /api/data/erase"]:::grp
G_TEL["03e Telemetry\n/api/telemetry/*"]:::grp
%% ---- Dock apps (left) ----
A_CHAT["Chat"]:::app
A_DASH["Dashboard / Home"]:::app
A_MEM["Memory\nframes/KG/harvest/wiki/evolution"]:::app
A_EVT["Events and Logs"]:::app
A_CAP["Skills and Apps"]:::app
A_CONN["Connectors"]:::app
A_COCK["Cockpit / Command Center"]:::app
A_MC["Mission Control"]:::app
A_WD["Waggle Dance"]:::app
A_AGT["Personas / Agents"]:::app
A_FILE["Files"]:::app
A_CRON["Scheduled Jobs"]:::app
A_MKT["Marketplace"]:::app
A_LAUN["AI Tools / Launcher"]:::app
A_VOICE["Voice\nstatic placeholder"]:::app
A_ROOM["Room\nsub-agent canvas"]:::app
A_APPR["Approvals"]:::app
A_TL["Timeline"]:::app
A_BAK["Backup and Restore"]:::app
A_TEL["Usage and Telemetry"]:::app
A_GOV["Team Governance\nTEAMS tier"]:::app
A_SET["Settings"]:::app
A_VAULT["Vault"]:::app
A_PROF["My Profile"]:::app
%% All apps go through the shared adapter
A_CHAT & A_DASH & A_MEM & A_EVT & A_CAP & A_CONN & A_COCK & A_MC & A_WD & A_AGT & A_FILE & A_CRON & A_MKT & A_LAUN & A_VOICE & A_ROOM & A_APPR & A_TL & A_BAK & A_TEL & A_GOV & A_SET & A_VAULT & A_PROF --> ADAPTER
%% Chat
A_CHAT --> G_CHAT
A_CHAT --> G_MEM
A_CHAT --> G_TEAM
A_CHAT --> G_WIKI
%% Dashboard
A_DASH --> G_MEM
A_DASH --> G_CHAT
%% Memory app
A_MEM --> G_MEM
A_MEM --> G_WIKI
A_MEM --> G_EVO
A_MEM --> G_RT
%% Events
A_EVT --> G_RT
%% Skills and Apps
A_CAP --> G_MKT
%% Connectors
A_CONN --> G_CONN
%% Cockpit + Compliance
A_COCK --> G_RT
A_COCK --> G_EVO
A_COCK --> G_CONN
A_COCK --> G_MKT
A_COCK --> G_WIKI
%% Mission Control
A_MC --> G_RT
A_MC --> G_TEAM
A_MC --> G_TOOLS
%% Waggle Dance
A_WD --> G_WD
%% Personas / Agents
A_AGT --> G_TEAM
A_AGT --> G_CHAT
A_AGT --> G_MKT
%% Files
A_FILE --> G_WS
%% Scheduled Jobs
A_CRON --> G_RT
%% Marketplace
A_MKT --> G_MKT
%% Launcher
A_LAUN --> G_TOOLS
%% Room
A_ROOM --> G_RT
%% Approvals
A_APPR --> G_EVO
%% Timeline
A_TL --> G_RT
%% Backup
A_BAK --> G_RT
%% Telemetry
A_TEL --> G_EVO
A_TEL --> G_TEL
%% Settings
A_SET --> G_TEAM
A_SET --> G_TEL
A_SET --> G_RT
%% Vault
A_VAULT --> G_CONN
%% Profile
A_PROF --> G_TEAM
```
## 2. Overlays mapped to endpoint groups
```mermaid
flowchart LR
classDef ovl fill:#23202e,stroke:#e5a000,color:#f3ead0
classDef grp fill:#10271b,stroke:#34d399,color:#d7f7e6
classDef shared fill:#0f1622,stroke:#7dd3fc,color:#dff1ff
ADAPTER["adapter singleton\nlib/adapter.ts"]:::shared
GO_WS["03c Workspace + Templates\n/api/workspaces/* /api/workspace-templates/*\n/api/browse"]:::grp
GO_TEAM["03c Persona + Settings + Profile\n/api/personas/* /api/agent-groups/*\n/api/settings /api/profile"]:::grp
GO_MEM["03b Memory + Identity\n/api/memory/search /api/memory/stats\n/api/identity"]:::grp
GO_HARV["03b Harvest + Import\n/api/harvest/* /api/import/*"]:::grp
GO_PROV["03d Vault + Providers + Skills\n/api/vault /api/providers\n/api/skills /v1/models"]:::grp
GO_FLEET["03f Fleet + LiteLLM\n/api/fleet/spawn /api/litellm/models"]:::grp
GO_NOTIF["03f Notifications\n/api/notifications/*"]:::grp
GO_BILL["03g Billing + GDPR\n/api/tier/start-trial\n/api/stripe/* /api/data/erase"]:::grp
GO_TEL["03e Telemetry\n/api/telemetry/track"]:::grp
GO_SVC["Connect + Health\n/api/auth/session-token /health"]:::shared
O_ONB["Onboarding wizard\n8 steps"]:::ovl
O_LOGIN["Login briefing"]:::ovl
O_SEARCH["Global search Cmd+K"]:::ovl
O_CWS["Create workspace dialog"]:::ovl
O_PSW["Persona switcher"]:::ovl
O_SPAWN["Spawn agent dialog"]:::ovl
O_WSW["Workspace switcher\nprops-driven"]:::ovl
O_NINBOX["Notification inbox"]:::ovl
O_CRAIL["Context rail"]:::ovl
O_ERASE["Erase data dialog GDPR"]:::ovl
O_UPG["Upgrade modal"]:::ovl
O_TRIAL["Trial expired modal"]:::ovl
O_KB["Keyboard shortcuts help\nno calls"]:::ovl
O_TOUR["Onboarding tooltips\nno calls"]:::ovl
O_ONB & O_LOGIN & O_SEARCH & O_CWS & O_PSW & O_SPAWN & O_ERASE & O_UPG & O_TRIAL & O_NINBOX & O_CRAIL --> ADAPTER
O_ONB --> GO_SVC
O_ONB --> GO_PROV
O_ONB --> GO_HARV
O_ONB --> GO_TEAM
O_ONB --> GO_WS
O_ONB --> GO_TEL
O_LOGIN --> GO_MEM
O_LOGIN --> GO_WS
O_SEARCH --> GO_WS
O_SEARCH --> GO_MEM
O_SEARCH --> GO_PROV
O_CWS --> GO_WS
O_CWS --> GO_TEAM
O_PSW --> GO_TEAM
O_SPAWN --> GO_FLEET
O_SPAWN --> GO_PROV
O_SPAWN --> GO_WS
O_NINBOX --> GO_NOTIF
O_CRAIL --> GO_MEM
O_ERASE --> GO_BILL
O_UPG --> GO_BILL
O_TRIAL --> GO_BILL
```

159
docs/backend-map/DIAGRAMS/05-tier-gating.md Normal file
View File

@@ -0,0 +1,159 @@
# Tier + Trust Gating
Waggle gates every action on **two orthogonal axes**. The **subscription tier** (TRIAL / FREE / PRO / TEAMS / ENTERPRISE) decides whether a feature exists for this user at all — enforced on HTTP routes via `requireTier()` returning `403 TIER_INSUFFICIENT`, and on tool registration (KVARK tools only exist when KVARK is configured). The **autonomy / trust level** (Normal / Trusted / YOLO) decides, per tool call, whether the UI must show an approval prompt — a pure runtime UX lever with a hardcoded critical blacklist that always wins, even at YOLO. The rule of thumb: *tier decides whether the door exists; trust decides whether it needs a key turn each time you walk through.* Always run a stored tier through `getEffectiveTier(tier, trialStartedAt)` before gating, because an expired TRIAL collapses to FREE.
---
## 1. The two gating axes (overview)
```mermaid
flowchart TD
USER["User stored tier plus trialStartedAt"] --> EFF["getEffectiveTier\nTRIAL expired collapses to FREE"]
EFF --> CAPS["getCapabilities\nTierCapabilities flag set"]
CAPS --> AXIS1["AXIS 1 — Tier gate\nfeature exists for this plan?"]
SESSION["Session AutonomyLevel\nnormal / trusted / yolo"] --> AXIS2["AXIS 2 — Trust gate\nclick needed for this call?"]
AXIS1 --> ROUTES["requireTier preHandler\n403 TIER_INSUFFICIENT on fail"]
AXIS1 --> KREG["KVARK tools registered\nonly when KVARK configured"]
AXIS2 --> CONF["needsConfirmationWithAutonomy\nper tool call"]
ROUTES --> GATED["Marketplace, Personas, Admin,\nTeam, Cloud-sync, Enterprise packs"]
KREG --> KVARK["kvark_search, kvark_feedback,\nkvark_action, kvark_ask_document"]
CONF --> TOOLS["Tools, connectors, bash,\ninstall_capability"]
```
---
## 2. The 5 tiers and what each unlocks
```mermaid
flowchart LR
subgraph TRIAL["TRIAL — 0 USD / 15 days"]
T1["All features unlocked\nmirrors TEAMS plus ENTERPRISE\nselfHosted OFF\ndecays to FREE after 15 days"]
end
subgraph FREE["FREE — 0 USD forever"]
F1["5 workspaces, 5 connectors\nspawnAgents ON, memory free\nbuilt-in skills only\nexport txt and md\nauditLog none, kvarkCta subtle"]
end
subgraph PRO["PRO — 19 USD / mo (solo)"]
P1["Unlimited workspaces plus connectors\ncustomSkills ON, marketplace\nexport txt md pdf json\nteamMembersLimit 1, auditLog basic\nno sharedWorkspaces, no cloudSync"]
end
subgraph TEAMS["TEAMS — 49 USD / seat"]
M1["sharedWorkspaces, teamSkillLibrary\ncloudSync, adminPanel\nauditLog full, selfHosted\nmanagedModelPool, priorityModels\nWaggleDance, kvarkCta active"]
end
subgraph ENT["ENTERPRISE — consultative"]
E1["KVARK sovereign on-prem\nall TEAMS capabilities\nselfHosted ON, kvarkCta none\ngovernance permissions route"]
end
TRIAL -->|"trial expires"| FREE
FREE -->|"upgrade 19/mo"| PRO
PRO -->|"upgrade 49/seat"| TEAMS
TEAMS -->|"sales contact"| ENT
```
---
## 3. Tier x Capability matrix (verbatim from `TIER_CAPABILITIES`)
`-1` means **unlimited**. Values quoted from `packages/shared/src/tiers.ts`.
| Capability | TRIAL | FREE | PRO | TEAMS | ENTERPRISE |
|---|---|---|---|---|---|
| `connectorLimit` | -1 | **5** | -1 | -1 | -1 |
| `workspaceLimit` | -1 | **5** | -1 | -1 | -1 |
| `embeddingProviders` | all 6 | inprocess, mock, ollama | + voyage, openai | all 6 | all 6 |
| `embeddingQuotaPerMonth` | -1 | -1 | -1 | -1 | -1 |
| `messageHistoryLimit` | -1 | -1 | -1 | -1 | -1 |
| `spawnAgents` | yes | yes | yes | yes | yes |
| `customSkills` | yes | **no** | yes | yes | yes |
| `teamSkillLibrary` | yes | **no** | **no** | yes | yes |
| `cloudSync` | yes | **no** | **no** | yes | yes |
| `exportFormats` | txt md pdf json | **txt md** | txt md pdf json | txt md pdf json | txt md pdf json |
| `teamMembersLimit` | -1 | **1** | **1** | -1 | -1 |
| `sharedWorkspaces` | yes | **no** | **no** | yes | yes |
| `adminPanel` | yes | **no** | **no** | yes | yes |
| `auditLog` | full | **none** | **basic** | full | full |
| `selfHosted` | **no** | **no** | **no** | yes | yes |
| `managedModelPool` | yes | **no** | **no** | yes | yes |
| `priorityModels` | yes | **no** | **no** | yes | yes |
| `kvarkCta` | subtle | subtle | subtle | **active** | **none** |
| `stripePriceId` | null | null | `STRIPE_PRICE_PRO` | `STRIPE_PRICE_TEAMS` | null |
`all 6` embedding providers = `inprocess, mock, ollama, voyage, openai, litellm`. Tier ordering (`TIER_ORDER`): `FREE 0, PRO 1, TEAMS 2, ENTERPRISE 3, TRIAL 3` — TRIAL ties ENTERPRISE because it has max capabilities, but is time-limited.
---
## 4. Tier-gated HTTP routes (Axis 1)
The frontend must catch `403 TIER_INSUFFICIENT` and show an upgrade prompt using `required` and `upgradeUrl`.
| Method | Path | Min tier |
|---|---|---|
| POST | `/api/marketplace/install` | PRO |
| POST | `/api/marketplace/publish` | PRO |
| GET | `/api/marketplace/enterprise-packs` | ENTERPRISE (plus KVARK configured) |
| POST | `/api/personas` | PRO |
| POST | `/api/personas/generate` | PRO |
| GET | `/api/cost/by-workspace` | TEAMS |
| POST | `/api/cloud-sync/toggle` | TEAMS |
| GET | `/api/admin/overview` | TEAMS |
| POST | `/api/admin/audit-export` | TEAMS |
| POST | `/api/team/connect` | TEAMS |
| GET | `/api/team/governance/permissions` | ENTERPRISE |
KVARK tools (`kvark_search`, `kvark_feedback`, `kvark_action`, `kvark_ask_document`) are registered only when `getKvarkConfig(vault)` returns a connection — 4 tools when configured, 0 otherwise.
---
## 5. The Normal / Trusted / YOLO autonomy gate (Axis 2)
```mermaid
flowchart TD
CALL["Tool call name plus args"] --> NC{"needsConfirmation?\nALWAYS_CONFIRM, connector write,\ndestructive bash"}
NC -- no --> RUN["Run silently"]
NC -- yes --> LVL{"AutonomyLevel"}
LVL -- normal --> PROMPT["Show approval prompt"]
LVL -- "trusted or yolo" --> CRIT{"isCriticalNeverAutopass?"}
CRIT -- yes --> PROMPT
CRIT -- no --> WHICH{"which level?"}
WHICH -- yolo --> AUTO["Auto-approve plus audit step"]
WHICH -- trusted --> TAP{"in TRUSTED_AUTOPASS\nor safe bash?"}
TAP -- yes --> AUTO
TAP -- no --> PROMPT
```
**Level behavior:**
| Level | Behavior |
|---|---|
| `normal` | Gate everything `needsConfirmation` flags (writes, connector writes, git, install, destructive bash). |
| `trusted` | Auto-pass `TRUSTED_AUTOPASS` (`write_file, edit_file, generate_docx, read_other_workspace, read_other_workspace_file`) and non-critical bash. Still gate git push/commit/pr/merge, `install_capability`, connector writes, cross-workspace writes. |
| `yolo` | Auto-pass everything **except** `isCriticalNeverAutopass`. |
**`isCriticalNeverAutopass` (never auto-passes, even at YOLO):** `rm -rf /` or `~` or `$HOME` or `/*`, any `sudo`, `format C:`, `mkfs`, `reg delete`, `dd if=… of=/dev`, forced `git push --force` to main/master/production, fork bomb; `install_capability` with `_riskLevel === 'high'`.
Autonomy override travels in the chat request body `{ "autonomy": { "level": "trusted"|"yolo", "expiresAt": <ms epoch> } }`; if `expiresAt` is absent or past, the server falls back to `normal`.
---
## 6. How tier + trust combine (worked examples)
```mermaid
flowchart TD
A["FREE user — connector_github_create_issue"] --> A1["Tier: no route gate"] --> A2["Trust: write so Normal prompts,\nTrusted gates, YOLO auto-passes"] --> A3["Runs after approval if connected"]
B["FREE user — Install marketplace pack"] --> B1["Tier: 403 needs PRO"] --> B3["Blocked, upgrade prompt"]
C["ENTERPRISE user — kvark_action"] --> C1["Tier: KVARK configured so registered"] --> C2["Trust: requires approval"] --> C3["Governed action runs with audit ref"]
D["Any tier, YOLO — bash sudo rm -rf /"] --> D1["Tier: none"] --> D2["Trust: isCriticalNeverAutopass"] --> D3["Still prompts even at YOLO"]
E["PRO user — install_capability starter-pack"] --> E1["Tier: none"] --> E2["Trust: ALWAYS_CONFIRM,\nclass from _riskLevel"] --> E3["Prompts, critical if high-risk"]
```
| Scenario | Tier check | Trust check | Net result |
|---|---|---|---|
| FREE — `connector_github_create_issue` | none (not a `requireTier` route) | write → Normal prompts; Trusted gates; YOLO auto-passes | Runs after approval if connected |
| FREE — install marketplace pack | `403 TIER_INSUFFICIENT` (needs PRO) | n/a (blocked first) | Blocked → upgrade prompt |
| ENTERPRISE — `kvark_action` (KVARK configured) | tools registered | requires approval | Governed action runs with audit ref |
| Any tier, YOLO — `bash sudo rm -rf /` | none | `isCriticalNeverAutopass` | Still prompts even at YOLO |
| PRO — `install_capability` (starter-pack) | none | in `ALWAYS_CONFIRM` | Prompts (critical if high-risk) |

110
docs/backend-map/DIAGRAMS/06-api-domains.md Normal file
View File

@@ -0,0 +1,110 @@
# API Domain Overview
This is the cross-cutting map of every HTTP/SSE/WebSocket surface a Lovable rebuild of the Waggle frontend must talk to. Almost everything lives in the **Local Fastify Sidecar** (default loopback `:3333`, flat `/api/*` paths, Bearer session-token + same-origin guards); a smaller set of Clerk-authenticated routes live in the separate **Cloud Server** (`:3100`, used only in SaaS/team-server deployments). Each domain node below is annotated with its endpoint count (counts are derived from the route tables in sections `03a`-`03g`; the approvals group is shared between the chat and governance sections, so it is shown once and noted). KVARK has no Fastify routes — it is reached only through the in-process `KvarkClient` (its method count is shown for completeness). Use this as the index; drill into the matching `03*` section for exact request/response shapes.
```mermaid
mindmap
root(("Waggle API\n7 domains"))
("Chat / Agents / Sessions\n~29 endpoints\nsec 03a")
("POST /api/chat SSE + DELETE history\n2")
("GET /api/history\n1")
("Agent status / cost / model / active\n6")
("Slash commands /api/commands/execute\n1")
("Sessions CRUD + search + export + timeline\n8")
("Agent groups CRUD + run\n5")
("POST /api/agent/run one-shot SSE\n1")
("Approvals SSE-paused gate\nshared with 03e")
("Memory / Wiki / Harvest\n~38 endpoints\nsec 03b")
("Memory frames search/CRUD/stats\n7")
("Knowledge graph read /api/memory/graph\n1")
("Wiki pages / compile / export\n8")
("Harvest preview/commit/sources/runs SSE\n11")
("Legacy import preview/commit\n2")
("Identity record + mind context\n5")
("Document version registry\n3")
("GDPR data erase\n1")
("Workspace / Team / Personas\n~76 endpoints\nsec 03c")
("Workspaces lifecycle + storage + context\n16")
("Workspace templates 15 built-in\n5")
("Team remote-proxy + local CRUD\n25")
("Personas catalog + create + generate\n5")
("Settings / tier / cloud-sync / admin\n14")
("User profile + style + brand\n7")
("Pins per workspace\n4")
("Marketplace / Skills / Connectors\n~60 endpoints\nsec 03d")
("Marketplace search/install/sources PRO\n15")
("Skills + plugins + hooks\n28")
("Connectors connect/disconnect/health\n4")
("AI-OS tool launcher detect/launch/hooks\n5")
("OAuth 5 providers\n3")
("Vault list/add/delete/reveal\n4")
("LLM + search providers catalog\n1")
("Evolution / Governance\n~39 endpoints\nsec 03e")
("Evolution runs accept/reject/run SSE\n8")
("Feedback thumbs + stats\n2")
("Telemetry local-only\n6")
("Compliance EU AI Act + templates + PDF\n12")
("Cost dashboard + by-workspace TEAMS\n3")
("Capabilities status + plugin toggles\n3")
("Approvals inbox + grants\n5")
("Real-time / Ops\n~52 endpoints\nsec 03f")
("WaggleDance UI signals + stream SSE\n4")
("WaggleDance v2 protocol bus\n2")
("Audit events + stats + stream SSE\n3")
("Cron schedules + trigger + history\n7")
("Notifications stream SSE + store\n6")
("Offline message queue\n5")
("Backup / restore / metadata\n3")
("Agent fleet spawn/pause/resume/kill\n5")
("LiteLLM control + pricing\n4")
("Local inference hardware/models/pull\n4")
("Anthropic proxy /v1/chat/completions SSE\n2")
("Filesystem browse local-only\n2")
("Browser extension health\n1")
("Telegram outbound push\n4")
("Cloud / Billing / KVARK\n~35 surfaces\nsec 03g")
("Sidecar inline auth/docs/health/ws\n6")
("Stripe checkout/webhook/sync/portal\n4")
("Cloud agents + groups Clerk\n10")
("Cloud jobs queue Clerk\n4")
("Cloud scout findings Clerk\n2")
("Cloud suggestions Clerk\n2")
("KVARK client methods no routes\n5")
("WebSocket sidecar + cloud\n2")
```
## Domains as a graph (server split + auth model)
The same domains, grouped by which server hosts them and what auth each requires. The sidecar serves the SPA and almost every domain; the cloud server is Clerk-gated and optional.
```mermaid
graph LR
FE["Frontend SPA\napps/web served by sidecar"]
subgraph SIDECAR["Local Sidecar :3333 - Bearer session-token + same-origin"]
D1["Chat / Agents / Sessions\n~29"]
D2["Memory / Wiki / Harvest\n~38"]
D3["Workspace / Team / Personas\n~76"]
D4["Marketplace / Skills / Connectors\n~60"]
D5["Evolution / Governance\n~39"]
D6["Real-time / Ops SSE+WS\n~52"]
BILL["Stripe billing\n4"]
BOOT["Auth token / docs / health / ws\n6"]
end
subgraph CLOUD["Cloud Server :3100 - Clerk JWT - optional SaaS"]
C1["Agents + Groups\n10"]
C2["Jobs queue\n4"]
C3["Scout findings\n2"]
C4["Suggestions\n2"]
C5["Team WebSocket gateway\n1"]
end
KVARK["KVARK client\nno routes - vault-credentialed\n5 methods - TEAMS/ENTERPRISE"]
FE -->|"same-origin Bearer"| SIDECAR
FE -.->|"team mode Clerk JWT"| CLOUD
D5 -->|"agent tools"| KVARK
D3 -->|"teamServerUrl proxy"| CLOUD
BILL -->|"writes tier to config.json"| D3
```

109
docs/backend-map/README.md Normal file
View File

@@ -0,0 +1,109 @@
# Waggle OS — Backend Map
This is the complete, source-grounded map of the Waggle OS backend, written so the frontend can be
**rebuilt from scratch (e.g. in Lovable) against the existing, unchanged Fastify sidecar**. The
mental model is simple and worth internalizing before anything else: Waggle is a single-page
"desktop OS" (window manager + dock + ~18 apps, opened by `appId`, not by URL) that is a *thin
client* — it renders state, streams responses over SSE, and POSTs user intent through one shared
`adapter` singleton to a local Node.js Fastify **sidecar** (`http://127.0.0.1:3333`). The sidecar is
the brain stem: it resolves the workspace session, builds the layered system prompt, runs the agent
tool-calling loop, executes tools through a governance middleware chain, and streams results back.
Two physically separate data stores sit behind it — per-workspace **SQLite `*.mind` files** (private
memory, knowledge graph, identity, awareness; the moat) and an optional team/cloud **Postgres** layer
(users, teams, tasks, governance). Everything is gated on two orthogonal axes: **subscription tier**
(does the feature exist?) and **trust/autonomy** (does this action need an approval click?). The
backend is the contract; the frontend is replaceable.
---
## Recommended Reading Order
Read these in sequence for a clean ramp from concept → contracts → build playbook:
1. **[00-MENTAL-MODEL.md](00-MENTAL-MODEL.md)** — the concept map and six-layer request lifecycle. Read this first; keep it in your head while reading everything else.
2. **[DIAGRAMS/01-system-architecture.md](DIAGRAMS/01-system-architecture.md)** — the cross-cutting runtime stack: shell → SPA → sidecar → packages → data stores → LLM router.
3. **[DIAGRAMS/02-master-er.md](DIAGRAMS/02-master-er.md)** — the two ER diagrams (SQLite memory layer + Postgres relational layer) and how they bridge.
4. **[sections/03a-api-chat-agents.md](sections/03a-api-chat-agents.md)** … **[sections/03g-api-cloud-billing-kvark.md](sections/03g-api-cloud-billing-kvark.md)** — the full HTTP/SSE/WS API contract, split by domain (start at 03a, the conversational core).
5. **[sections/04-feature-map.md](sections/04-feature-map.md)** — the canonical app ↔ endpoint matrix: every screen and exactly which routes it calls.
6. **[sections/05a-subsystem-agent-runtime.md](sections/05a-subsystem-agent-runtime.md)** … **[sections/05g-subsystem-skills-marketplace-wiki.md](sections/05g-subsystem-skills-marketplace-wiki.md)** — the subsystem deep-dives explaining what the API actually does internally.
7. **[07-FRONTEND-REBUILD-GUIDE.md](07-FRONTEND-REBUILD-GUIDE.md)** — the action-oriented, build-in-order playbook for the Lovable rebuild.
---
## Full File Index
### Top-level
| File | Description |
|---|---|
| [00-MENTAL-MODEL.md](00-MENTAL-MODEL.md) | The concept map: what Waggle OS is, the six-layer request lifecycle, and the two or three ideas everything hangs off. |
| [07-FRONTEND-REBUILD-GUIDE.md](07-FRONTEND-REBUILD-GUIDE.md) | Action-oriented Lovable rebuild playbook — build the adapter + ServiceProvider + Desktop shell first, then screens that call adapter methods. |
| [AUDIT.md](AUDIT.md) | Completeness audit cross-checking the written sections against ground truth (route files, schema tables, connectors); ~94% coverage with named gaps. |
| [WAGGLE-BACKEND-VISUAL.html](WAGGLE-BACKEND-VISUAL.html) | Self-contained dark-themed visual one-pager of the whole backend map (Hive DS palette) — open in a browser for the diagrams. |
### Diagrams (`DIAGRAMS/`)
| File | Description |
|---|---|
| [DIAGRAMS/01-system-architecture.md](DIAGRAMS/01-system-architecture.md) | Full runtime stack as a layered flowchart: Tauri shell → React SPA → Fastify sidecar (:3333) → packages → SQLite/Postgres/Redis → LiteLLM → LLM providers. |
| [DIAGRAMS/02-master-er.md](DIAGRAMS/02-master-er.md) | Two ER diagrams — per-workspace SQLite memory (`*.mind`, 14+2 tables) and team/cloud Postgres (20 tables) — sharing no FKs except `users.mind_path`. |
| [DIAGRAMS/03-chat-turn-sequence.md](DIAGRAMS/03-chat-turn-sequence.md) | End-to-end sequence of one chat turn: `POST /api/chat` → prompt assembly → tool loop → 11-step tool middleware → SSE stream → cognify write-back. |
| [DIAGRAMS/04-feature-api-map.md](DIAGRAMS/04-feature-api-map.md) | Rebuild-contract flowchart mapping every OS app/overlay to the backend endpoint groups (03a–03g) it depends on. |
| [DIAGRAMS/05-tier-gating.md](DIAGRAMS/05-tier-gating.md) | The two orthogonal gating axes — subscription tier (does the door exist?) vs trust/autonomy (does it need a key turn?), plus `getEffectiveTier()`. |
| [DIAGRAMS/06-api-domains.md](DIAGRAMS/06-api-domains.md) | Mind-map index of all 7 API domains with endpoint counts; local sidecar (:3333) vs cloud server (:3100), KVARK via in-process client only. |
### Sections — Data Model (`sections/02*`)
| File | Description |
|---|---|
| [sections/02a-data-model-memory.md](sections/02a-data-model-memory.md) | Per-workspace SQLite `*.mind` schema — frames, knowledge graph, identity, awareness, sessions; the canonical shape of everything the memory APIs return. |
| [sections/02b-data-model-relational.md](sections/02b-data-model-relational.md) | Team/cloud Postgres + Drizzle schema (20 tables): users, teams, agents, tasks, jobs, governance — column names/types/FKs verbatim from source. |
| [sections/02c-shared-types-tiers.md](sections/02c-shared-types-tiers.md) | The wire contract: `@waggle/shared` interfaces, enums, Zod request schemas, and the 5-tier capability matrix consumed by both sidecar and web. |
### Sections — API Contracts (`sections/03*`)
| File | Description |
|---|---|
| [sections/03a-api-chat-agents.md](sections/03a-api-chat-agents.md) | Conversational core: `POST /api/chat` SSE stream, the event catalogue, mid-stream tool approvals, sessions CRUD/export, and slash commands. |
| [sections/03b-api-memory.md](sections/03b-api-memory.md) | Memory subsystem API: recall/save frames, knowledge graph, wiki compile/read, harvest external AI exports, import, identity, documents, GDPR erasure. |
| [sections/03c-api-workspace-team.md](sections/03c-api-workspace-team.md) | Management plane: workspaces + templates, teams/members, personas, settings (models/budgets/autonomy/tier), user profile, and message pins. |
| [sections/03d-api-marketplace-skills.md](sections/03d-api-marketplace-skills.md) | Capability layer API: marketplace, skills, connectors, tools, OAuth, vault, and providers. |
| [sections/03e-api-evolution-governance.md](sections/03e-api-evolution-governance.md) | "Self-improves + you stay in control": evolution runs, feedback, telemetry, EU AI Act compliance (PDF), cost tracking, capability status, approvals inbox. |
| [sections/03f-api-realtime-ops.md](sections/03f-api-realtime-ops.md) | Real-time + ops: the 4 SSE streams (WaggleDance v1/v2, events, notifications), cron, offline queue, backup/restore, fleet, LLM proxies, browse, Telegram. |
| [sections/03g-api-cloud-billing-kvark.md](sections/03g-api-cloud-billing-kvark.md) | The two HTTP surfaces, API-root discovery + auth/origin guards, Stripe billing flow (checkout → webhook → tier), KVARK client, and WebSocket channels. |
### Sections — Feature Map (`sections/04`)
| File | Description |
|---|---|
| [sections/04-feature-map.md](sections/04-feature-map.md) | Canonical app↔endpoint matrix: every OS app, overlay, and page, the shared `adapter`/providers, and the exact backend endpoints each feature calls. |
### Sections — Subsystem Deep-Dives (`sections/05*`)
| File | Description |
|---|---|
| [sections/05a-subsystem-agent-runtime.md](sections/05a-subsystem-agent-runtime.md) | The agent runtime: layered system-prompt assembly, the tool-calling loop, the tool middleware chain, and completion-time verification/file-write/skill gates. |
| [sections/05b-subsystem-memory.md](sections/05b-subsystem-memory.md) | The persistent memory engine (the moat): Identity → Awareness → Frames → Knowledge Graph, hybrid search, cognify write path, combined retrieval. |
| [sections/05c-subsystem-harvest.md](sections/05c-subsystem-harvest.md) | Harvest ingestion pipeline: external AI exports + files/URLs → normalized `UniversalImportItem` → memory frames; powers the Memory app HarvestTab. |
| [sections/05d-subsystem-evolution.md](sections/05d-subsystem-evolution.md) | Self-evolution loop: mine execution history → GEPA + EvolveSchema mutation → LLM-judge → safety gates → auditable `EvolutionRun` → user accept/reject. |
| [sections/05e-subsystem-waggledance-aios.md](sections/05e-subsystem-waggledance-aios.md) | WaggleDance multi-agent coordination + the AI-OS arc: detect 7 external AI tools, install reversible hooks, launch with workspace context, stream signals back. |
| [sections/05f-subsystem-capabilities-tiers.md](sections/05f-subsystem-capabilities-tiers.md) | What an agent can/may do: capability discovery + acquisition, connectors-as-tools, MCP catalog, and the trust vs tier gates (incl. KVARK gating). |
| [sections/05g-subsystem-skills-marketplace-wiki.md](sections/05g-subsystem-skills-marketplace-wiki.md) | Skills lifecycle, the marketplace catalog/installer/security gate, and the wiki-compiler — the PRO+ upgrade trigger (Memory/Harvest/wiki stay free). |
---
## How to Feed This Into Lovable
For the rebuild, paste the **contract-bearing** docs directly into Lovable and keep the visual open
on the side:
- **Paste these as context** (the binding contracts the new frontend must honor):
- [00-MENTAL-MODEL.md](00-MENTAL-MODEL.md) — so the model holds the desktop-OS / thin-client mental model.
- [sections/02c-shared-types-tiers.md](sections/02c-shared-types-tiers.md) — the exact wire types, Zod request schemas, and tier capability matrix.
- [sections/03a-api-chat-agents.md](sections/03a-api-chat-agents.md) through [sections/03g-api-cloud-billing-kvark.md](sections/03g-api-cloud-billing-kvark.md) — the full endpoint/SSE contract (paste the 03* sections relevant to the screens you're building).
- [07-FRONTEND-REBUILD-GUIDE.md](07-FRONTEND-REBUILD-GUIDE.md) — the build-in-order playbook (adapter + ServiceProvider + Desktop shell first, then screens).
- **Open [WAGGLE-BACKEND-VISUAL.html](WAGGLE-BACKEND-VISUAL.html) in a browser** for the diagrams — it renders the architecture, ER, chat-turn, feature-API, and tier-gating views as one navigable one-pager, which is faster to skim than the raw Mermaid blocks.
Start by having Lovable build the `adapter` singleton + `ServiceProvider` + `Desktop` shell against
the sidecar at `http://127.0.0.1:3333`; everything else is screens that call adapter methods listed
in [sections/04-feature-map.md](sections/04-feature-map.md).

1552
docs/backend-map/WAGGLE-BACKEND-VISUAL.html Normal file
View File

File diff suppressed because it is too large Load Diff

542
docs/backend-map/sections/02a-data-model-memory.md Normal file
View File

@@ -0,0 +1,542 @@
# 02a · Data Model — Per-Workspace Memory Layer (`*.mind`)
## Purpose
This section documents the **per-workspace memory database** — the persistent "mind" of Waggle OS. Every workspace gets **one isolated SQLite file** (the `*.mind` file, e.g. `personal.mind`); there is **no shared/global memory database**. All schema below comes from `packages/hive-mind-core/src/mind/schema.ts` (the `SCHEMA_SQL` + `VEC_TABLE_SQL` constants), with column semantics cross-read from `frames.ts`, `knowledge.ts`, `identity.ts`, `awareness.ts`, and `sessions.ts`. For the frontend rebuild, treat this as the **canonical shape of everything the memory APIs return** — the server routes (documented in the API sections) read and write exactly these rows.
> **Schema version:** `SCHEMA_VERSION = '1'` (constant exported from `schema.ts`). Stored in the `meta` table under `key = 'schema_version'`.
> **One DB per workspace.** Each workspace is a separate `*.mind` SQLite file. Switching workspace = opening a different file. Nothing in this schema joins across workspaces.
---
## Layer overview
The mind is organized into numbered "layers" (the comments in `schema.ts` label them). They are NOT separate databases — just a conceptual grouping of tables inside the one `*.mind` file:
| Layer | Table(s) | Role |
|---|---|---|
| — | `meta` | Schema versioning + key/value flags |
| 0 | `identity` | Who this mind belongs to (single row) |
| 1 | `awareness` | Active working state (≤10 live items) |
| — | `sessions` | Maps GOPs (groups-of-prompts) to projects |
| 2 | `memory_frames` (+ `memory_frames_fts`, `memory_frames_vec`) | The actual memories (I/P/B frames) + full-text + vector search |
| 3 | `knowledge_entities`, `knowledge_relations` | Knowledge graph (entities + edges) |
| 4 | `procedures` | GEPA-optimized prompt templates |
| 5 | `improvement_signals` | Recurring patterns that should change behavior |
| 6 | `install_audit` | Capability-install trust trail |
| 7 | `ai_interactions` | EU AI Act Art. 12 audit log (append-only) |
| 8 | `harvest_sources` | Memory-harvest sync tracking |
| 9 | `execution_traces` | Agent run history (self-evolution input) |
| 10 | `evolution_runs` | Proposed/accepted self-evolution runs |
**14 base tables** + 2 virtual tables (`memory_frames_fts` FTS5, `memory_frames_vec` vec0). A `kg_entity_frames` link table is referenced by `frames.delete()` (`packages/hive-mind-core/src/mind/frames.ts:330`) but is **NOT defined in `schema.ts`** — it is created elsewhere (knowledge-graph wiring) and its DELETE is wrapped in try/catch, so it may be absent.
---
## Tables (every column)
### `meta` — schema versioning / key-value flags
Primary key: `key`. No indexes beyond the PK.
| Column | SQLite type | Null? | Default | Meaning |
|---|---|---|---|---|
| `key` | TEXT | NO | — | **PK.** Flag name (e.g. `schema_version`). |
| `value` | TEXT | NO | — | String value for that key. |
---
### `identity` — Layer 0 (single row, `<500` tokens)
Primary key: `id`, hard-pinned to `1` via `CHECK (id = 1)` — there is **exactly one identity row per mind**. Backed by `IdentityLayer` (`identity.ts`).
| Column | SQLite type | Null? | Default | Meaning |
|---|---|---|---|---|
| `id` | INTEGER | NO | — | **PK.** Always `1` (CHECK enforced). |
| `name` | TEXT | NO | — | Display name of the mind's owner/agent. |
| `role` | TEXT | NO | `''` | Job/role. |
| `department` | TEXT | NO | `''` | Department / org unit. |
| `personality` | TEXT | NO | `''` | Personality description. |
| `capabilities` | TEXT | NO | `''` | Free-text capability summary. |
| `system_prompt` | TEXT | NO | `''` | Base system prompt fragment for this identity. |
| `created_at` | TEXT | NO | `datetime('now')` | Created timestamp (SQLite UTC string). |
| `updated_at` | TEXT | NO | `datetime('now')` | Last-update timestamp; bumped on every `update()`. |
`IdentityLayer.toContext()` flattens these into the prompt. Empty-string fields are skipped in that rendering.
---
### `awareness` — Layer 1 (active working state, capped at 10)
Primary key: `id` (AUTOINCREMENT). Backed by `AwarenessLayer` (`awareness.ts`). Reads cap results to `MAX_ITEMS = 10` and filter out expired rows.
| Column | SQLite type | Null? | Default | Meaning |
|---|---|---|---|---|
| `id` | INTEGER | NO | autoincrement | **PK.** |
| `category` | TEXT | NO | — | **CHECK IN** `('task','action','pending','flag')`. UI labels: task→"Active Tasks", action→"Recent Actions", pending→"Pending Items", flag→"Context Flags". |
| `content` | TEXT | NO | — | The awareness item text. |
| `priority` | INTEGER | NO | `0` | Higher = surfaced first (`ORDER BY priority DESC`). |
| `metadata` | TEXT | NO | `'{}'` | JSON blob. Known keys (`AwarenessMetadata`): `context`, `status`, `result`, `priority` (+ arbitrary). Added via runtime `ALTER TABLE` migration if missing. |
| `created_at` | TEXT | NO | `datetime('now')` | Created timestamp. |
| `expires_at` | TEXT | YES | NULL | Optional expiry; rows past `expires_at` are filtered out of all read queries. |
---
### `sessions` — maps GOPs to projects
Primary key: `id` (AUTOINCREMENT). **`gop_id` is UNIQUE** and is the logical join key for `memory_frames`. Backed by `SessionStore` (`sessions.ts`). Index: `idx_sessions_project (project_id, started_at)`.
| Column | SQLite type | Null? | Default | Meaning |
|---|---|---|---|---|
| `id` | INTEGER | NO | autoincrement | **PK.** |
| `gop_id` | TEXT | NO | — | **UNIQUE.** "Group Of Prompts" id. Generated as `session:<ISO timestamp>:<rand6>`, or a stable id like `harvest` for long-lived logical sessions. Referenced by `memory_frames.gop_id`. |
| `project_id` | TEXT | YES | NULL | Optional project grouping. |
| `status` | TEXT | NO | `'active'` | **CHECK IN** `('active','closed','archived')`. |
| `started_at` | TEXT | NO | `datetime('now')` | Session start. |
| `ended_at` | TEXT | YES | NULL | Set on `close()`. |
| `summary` | TEXT | YES | NULL | Optional close-time summary. |
---
### `memory_frames` — Layer 2 (the actual memories: I/P/B)
Primary key: `id` (AUTOINCREMENT). The core of the mind. Backed by `FrameStore` (`frames.ts`). Frames use an **I/P/B model** organized per-GOP with a monotonic `t` ordinal.
| Column | SQLite type | Null? | Default | Meaning |
|---|---|---|---|---|
| `id` | INTEGER | NO | autoincrement | **PK.** Also the rowid linking `memory_frames_fts` and `memory_frames_vec`. |
| `frame_type` | TEXT | NO | — | **CHECK IN** `('I','P','B')`. **I** = Initial/state frame; **P** = Progress/delta frame (has `base_frame_id`); **B** = Bundle/cross-reference frame (content is JSON `{description, references:[ids]}`). |
| `gop_id` | TEXT | NO | — | **FK → `sessions(gop_id)`.** Groups frames into a session. |
| `t` | INTEGER | NO | `0` | Per-GOP monotonic ordinal (`MAX(t)+1` within the gop). Defines frame order. |
| `base_frame_id` | INTEGER | YES | NULL | **Self-FK → `memory_frames(id)`.** The I-frame a P/B frame builds on. Nulled on delete of the base. |
| `content` | TEXT | NO | — | Frame body. For B-frames this is JSON. May carry a `[hm …]` provenance prefix that dedup strips before hashing. |
| `importance` | TEXT | NO | `'normal'` | **CHECK IN** `('critical','important','normal','temporary','deprecated')`. Drives a retrieval multiplier (critical 2.0 / important 1.5 / normal 1.0 / temporary 0.7 / deprecated 0.3) and compaction (temporary pruned >30d, deprecated pruned >90d). |
| `source` | TEXT | NO | `'user_stated'` | **CHECK IN** `('user_stated','tool_verified','agent_inferred','import','system')`. ⚠️ The TS `FrameSource` type in `frames.ts` ALSO lists `'personal'`, `'workspace'`, `'team_sync'` — these are **not** in the DB CHECK constraint, so writing them would fail at the DB level. Treat the 5 CHECK values as authoritative for persisted rows. |
| `access_count` | INTEGER | NO | `0` | Incremented by `touch()` on every read/dedup hit. |
| `created_at` | TEXT | NO | `datetime('now')` | Created timestamp. Harvest can override with the original source timestamp if it passes strict ISO-8601 validation. |
| `last_accessed` | TEXT | NO | `datetime('now')` | Updated by `touch()`. |
**Indexes:** `idx_frames_gop_t (gop_id, t)`, `idx_frames_type (frame_type, gop_id)`, `idx_frames_base (base_frame_id)`.
**Dedup behavior (important for the frontend):** Inserting content identical (SHA-256 of `[hm …]`-stripped + trimmed body) to one of the **last 500 frames** does NOT create a new row — it increments `access_count` on the existing frame and returns it. So a "save" can be a silent no-op-with-bump.
---
### `memory_frames_fts` — FTS5 virtual table (keyword search)
```sql
CREATE VIRTUAL TABLE memory_frames_fts USING fts5(
content, content_rowid='id', tokenize='porter unicode61'
);
```
Mirrors `memory_frames.content`, keyed by `rowid = memory_frames.id`. Kept in sync on insert/update/delete by `FrameStore`. Powers keyword/hybrid search. Not directly queried by the frontend — it's an internal index.
---
### `memory_frames_vec` — vec0 virtual table (vector search)
Defined in the separate `VEC_TABLE_SQL` constant (created only when `sqlite-vec` is available):
```sql
CREATE VIRTUAL TABLE memory_frames_vec USING vec0(
embedding float[1024]
);
```
**1024-dim** float embeddings, keyed by `rowid = memory_frames.id`. All writes are wrapped in try/catch in `FrameStore` because the vec extension may be absent at runtime. Powers semantic search half of HybridSearch.
---
### `knowledge_entities` — Layer 3 (graph nodes)
Primary key: `id` (AUTOINCREMENT). Backed by `KnowledgeGraph` (`knowledge.ts`). **Bitemporal**: rows are versioned via `valid_from`/`valid_to` rather than hard-deleted — "active" = `valid_to IS NULL`.
| Column | SQLite type | Null? | Default | Meaning |
|---|---|---|---|---|
| `id` | INTEGER | NO | autoincrement | **PK.** |
| `entity_type` | TEXT | NO | — | Entity category (free-text; e.g. person/project/tool). |
| `name` | TEXT | NO | — | Entity name. Searched via `LIKE` (escaped). |
| `properties` | TEXT | NO | `'{}'` | JSON property bag. |
| `valid_from` | TEXT | NO | `datetime('now')` | Start of validity window. |
| `valid_to` | TEXT | YES | NULL | End of validity. **NULL = currently active.** `retireEntity()` sets this instead of deleting. |
| `recorded_at` | TEXT | NO | `datetime('now')` | When the row was written/last updated. |
**Indexes:** `idx_entities_type (entity_type)`, `idx_entities_name (name)`.
---
### `knowledge_relations` — Layer 3 (graph edges)
Primary key: `id` (AUTOINCREMENT). Directed edges between entities. Same bitemporal model.
| Column | SQLite type | Null? | Default | Meaning |
|---|---|---|---|---|
| `id` | INTEGER | NO | autoincrement | **PK.** |
| `source_id` | INTEGER | NO | — | **FK → `knowledge_entities(id)`.** Edge tail. |
| `target_id` | INTEGER | NO | — | **FK → `knowledge_entities(id)`.** Edge head. |
| `relation_type` | TEXT | NO | — | Edge label (free-text; e.g. `works_on`, `knows`). |
| `confidence` | REAL | NO | `1.0` | Edge confidence 0–1. |
| `properties` | TEXT | NO | `'{}'` | JSON property bag. |
| `valid_from` | TEXT | NO | `datetime('now')` | Validity start. |
| `valid_to` | TEXT | YES | NULL | NULL = active; `retireRelation()` sets it. |
| `recorded_at` | TEXT | NO | `datetime('now')` | Write timestamp. |
**Indexes:** `idx_relations_source (source_id, relation_type)`, `idx_relations_target (target_id, relation_type)`.
---
### `improvement_signals` — Layer 5 (behavior-change patterns)
Primary key: `id` (AUTOINCREMENT). Counts recurring patterns; surfaced to the user when frequent.
| Column | SQLite type | Null? | Default | Meaning |
|---|---|---|---|---|
| `id` | INTEGER | NO | autoincrement | **PK.** |
| `category` | TEXT | NO | — | **CHECK IN** `('capability_gap','correction','workflow_pattern','skill_promotion')`. |
| `pattern_key` | TEXT | NO | — | Stable dedup key (unique within category). |
| `detail` | TEXT | NO | `''` | Human-readable detail. |
| `count` | INTEGER | NO | `1` | Times observed; incremented on repeat. |
| `first_seen` | TEXT | NO | `datetime('now')` | First observation. |
| `last_seen` | TEXT | NO | `datetime('now')` | Most recent observation. |
| `surfaced` | INTEGER | NO | `0` | Boolean (0/1): has this been shown to the user. |
| `surfaced_at` | TEXT | YES | NULL | When surfaced. |
| `metadata` | TEXT | NO | `'{}'` | JSON. |
**Indexes:** `idx_signals_category_key (category, pattern_key)` **UNIQUE** (one row per category+key), `idx_signals_category (category, count DESC)`.
---
### `install_audit` — Layer 6 (capability install trust trail)
Primary key: `id` (AUTOINCREMENT). Records every capability-install decision. The CHECK lists must stay in sync with `packages/core/src/install-audit.ts` (a documented drift once crashed `acquire_capability`).
| Column | SQLite type | Null? | Default | Meaning |
|---|---|---|---|---|
| `id` | INTEGER | NO | autoincrement | **PK.** |
| `timestamp` | TEXT | NO | `datetime('now')` | When the event occurred. |
| `capability_name` | TEXT | NO | — | Capability identifier. |
| `capability_type` | TEXT | NO | — | **CHECK IN** `('native','skill','plugin','mcp','connector','marketplace')`. |
| `source` | TEXT | NO | — | Origin (registry/url/etc). |
| `version` | TEXT | YES | NULL | Capability version. |
| `risk_level` | TEXT | NO | — | **CHECK IN** `('low','medium','high')`. |
| `trust_source` | TEXT | NO | — | Where trust derives from. |
| `approval_class` | TEXT | NO | — | **CHECK IN** `('standard','elevated','critical','blocked')`. |
| `action` | TEXT | NO | — | **CHECK IN** `('proposed','approved','installed','rejected','failed','blocked')`. |
| `initiator` | TEXT | NO | — | **CHECK IN** `('agent','user','system')`. |
| `detail` | TEXT | NO | `''` | Free-text detail. |
**Indexes:** `idx_audit_capability (capability_name, action)`, `idx_audit_timestamp (timestamp DESC)`.
---
### `procedures` — Layer 4 (GEPA-optimized prompt templates)
Primary key: `id` (AUTOINCREMENT). Versioned prompt templates with measured performance.
| Column | SQLite type | Null? | Default | Meaning |
|---|---|---|---|---|
| `id` | INTEGER | NO | autoincrement | **PK.** |
| `name` | TEXT | NO | — | Procedure/template name. |
| `model` | TEXT | NO | — | Model the template targets. |
| `template` | TEXT | NO | — | The prompt template text. |
| `version` | INTEGER | NO | `1` | Template version. |
| `success_rate` | REAL | NO | `0.0` | Measured success rate. |
| `avg_cost` | REAL | NO | `0.0` | Measured average cost (USD). |
| `created_at` | TEXT | NO | `datetime('now')` | Created. |
| `updated_at` | TEXT | NO | `datetime('now')` | Updated. |
**Index:** `idx_procedures_name_model (name, model)`.
---
### `ai_interactions` — Layer 7 (EU AI Act Art. 12 audit log)
Primary key: `id` (AUTOINCREMENT). **APPEND-ONLY** — two triggers (`ai_interactions_no_delete`, `ai_interactions_no_update`) `RAISE(ABORT, …)` on any UPDATE or DELETE. The frontend can only INSERT and SELECT these rows; never edit or remove.
| Column | SQLite type | Null? | Default | Meaning |
|---|---|---|---|---|
| `id` | INTEGER | NO | autoincrement | **PK.** |
| `timestamp` | TEXT | NO | `datetime('now')` | Event time. |
| `workspace_id` | TEXT | YES | NULL | Originating workspace. |
| `session_id` | TEXT | YES | NULL | Originating session. |
| `model` | TEXT | NO | — | Model used. |
| `provider` | TEXT | NO | — | LLM provider. |
| `input_tokens` | INTEGER | NO | `0` | Prompt tokens. |
| `output_tokens` | INTEGER | NO | `0` | Completion tokens. |
| `cost_usd` | REAL | NO | `0` | Cost in USD. |
| `tools_called` | TEXT | NO | `'[]'` | JSON array of tool names. |
| `human_action` | TEXT | YES | NULL | **CHECK IN** `('approved','denied','modified','none')` (nullable). |
| `risk_context` | TEXT | YES | NULL | Risk annotation. |
| `imported_from` | TEXT | YES | NULL | Source if imported. |
| `persona` | TEXT | YES | NULL | Persona that ran. |
| `input_text` | TEXT | YES | NULL | Actual input (Art. 12.1(a); added 2026-04-15 via migration). |
| `output_text` | TEXT | YES | NULL | Actual output (Art. 12.1(a)). |
**Indexes:** `idx_interactions_workspace (workspace_id, timestamp)`, `idx_interactions_timestamp (timestamp DESC)`, `idx_interactions_model (model)`.
---
### `execution_traces` — Layer 9 (agent run history)
Primary key: `id` (AUTOINCREMENT). Raw agent-run records; the dataset that feeds self-evolution.
| Column | SQLite type | Null? | Default | Meaning |
|---|---|---|---|---|
| `id` | INTEGER | NO | autoincrement | **PK.** |
| `session_id` | TEXT | YES | NULL | Session id. |
| `persona_id` | TEXT | YES | NULL | Persona that ran. |
| `workspace_id` | TEXT | YES | NULL | Workspace. |
| `model` | TEXT | YES | NULL | Model used. |
| `task_shape` | TEXT | YES | NULL | Coarse task category. |
| `outcome` | TEXT | NO | `'pending'` | **CHECK IN** `('success','corrected','abandoned','verified','pending')`. |
| `trace_json` | TEXT | NO | `'{}'` | JSON of the full trace. |
| `cost_usd` | REAL | NO | `0` | Run cost. |
| `duration_ms` | INTEGER | NO | `0` | Run duration (ms). |
| `created_at` | TEXT | NO | `datetime('now')` | Start. |
| `finalized_at` | TEXT | YES | NULL | When outcome was finalized. |
**Indexes:** `idx_traces_session (session_id, created_at)`, `idx_traces_persona (persona_id, outcome)`, `idx_traces_outcome (outcome, created_at DESC)`, `idx_traces_workspace (workspace_id, created_at DESC)`.
---
### `evolution_runs` — Layer 10 (self-evolution proposals)
Primary key: `id` (AUTOINCREMENT). **`run_uuid` is UNIQUE.** Each row is a proposed prompt/schema mutation with a gate verdict and lifecycle status.
| Column | SQLite type | Null? | Default | Meaning |
|---|---|---|---|---|
| `id` | INTEGER | NO | autoincrement | **PK.** |
| `run_uuid` | TEXT | NO | — | **UNIQUE.** Stable run identifier. |
| `target_kind` | TEXT | NO | — | What's being evolved (e.g. persona / behavioral-spec). |
| `target_name` | TEXT | YES | NULL | Specific target name. |
| `baseline_text` | TEXT | NO | — | Original text before mutation. |
| `winner_text` | TEXT | NO | — | Winning mutated text. |
| `winner_schema_json` | TEXT | YES | NULL | Winning schema (JSON), if schema-evolution. |
| `delta_accuracy` | REAL | NO | `0` | Accuracy gain vs baseline. |
| `gate_verdict` | TEXT | NO | `'pass'` | **CHECK IN** `('pass','fail')`. |
| `gate_reasons_json` | TEXT | NO | `'[]'` | JSON array of gate reasons. |
| `status` | TEXT | NO | `'proposed'` | **CHECK IN** `('proposed','accepted','rejected','deployed','failed')`. |
| `artifacts_json` | TEXT | YES | NULL | JSON artifacts. |
| `user_note` | TEXT | YES | NULL | User decision note. |
| `failure_reason` | TEXT | YES | NULL | Why it failed (if `failed`). |
| `created_at` | TEXT | NO | `datetime('now')` | Proposed at. |
| `decided_at` | TEXT | YES | NULL | Accept/reject time. |
| `deployed_at` | TEXT | YES | NULL | Deploy time. |
**Indexes:** `idx_evo_runs_status (status, created_at DESC)`, `idx_evo_runs_target (target_kind, target_name, created_at DESC)`, `idx_evo_runs_created (created_at DESC)`.
---
### `harvest_sources` — Layer 8 (Memory Harvest sync tracking)
Primary key: `id` (AUTOINCREMENT). **`source` is UNIQUE** — one row per import source (chatgpt/claude/gemini/etc).
| Column | SQLite type | Null? | Default | Meaning |
|---|---|---|---|---|
| `id` | INTEGER | NO | autoincrement | **PK.** |
| `source` | TEXT | NO | — | **UNIQUE.** Source key (e.g. `claude`, `chatgpt`). |
| `display_name` | TEXT | NO | — | Human label for the source. |
| `source_path` | TEXT | YES | NULL | Filesystem path / location of the export. |
| `last_synced_at` | TEXT | YES | NULL | Last successful sync. |
| `items_imported` | INTEGER | NO | `0` | Items pulled from source. |
| `frames_created` | INTEGER | NO | `0` | `memory_frames` rows produced. |
| `auto_sync` | INTEGER | NO | `0` | Boolean (0/1): auto-resync enabled. |
| `sync_interval_hours` | INTEGER | NO | `24` | Auto-sync interval. |
| `last_content_hash` | TEXT | YES | NULL | Hash of last-imported content (skip-if-unchanged). |
| `created_at` | TEXT | NO | `datetime('now')` | Created. |
No explicit secondary indexes (UNIQUE on `source` provides the lookup index).
---
## Key relationships (FKs and logical joins)
- `memory_frames.gop_id` → `sessions.gop_id` (**FK**). Frames belong to a session/GOP.
- `memory_frames.base_frame_id` → `memory_frames.id` (**self-FK**). P/B frames reference their base I-frame.
- `memory_frames_fts.rowid` = `memory_frames.id` (logical, FTS5 `content_rowid`).
- `memory_frames_vec.rowid` = `memory_frames.id` (logical, vec0).
- `knowledge_relations.source_id` → `knowledge_entities.id` (**FK**).
- `knowledge_relations.target_id` → `knowledge_entities.id` (**FK**).
- `kg_entity_frames.frame_id` → `memory_frames.id` (**referenced in `frames.delete()` but table not in `schema.ts`** — created by KG wiring elsewhere; may be absent).
- The remaining tables (`identity`, `awareness`, `procedures`, `improvement_signals`, `install_audit`, `ai_interactions`, `execution_traces`, `evolution_runs`, `harvest_sources`, `meta`) are **standalone** — no DB-level FKs between them. `ai_interactions.session_id` / `execution_traces.session_id` are plain TEXT, not FK-constrained to `sessions`.
---
## ER diagram
```mermaid
erDiagram
sessions ||--o{ memory_frames : "gop_id"
memory_frames ||--o{ memory_frames : "base_frame_id (self)"
memory_frames ||--|| memory_frames_fts : "rowid=id (FTS5)"
memory_frames ||--|| memory_frames_vec : "rowid=id (vec0)"
knowledge_entities ||--o{ knowledge_relations : "source_id"
knowledge_entities ||--o{ knowledge_relations : "target_id"
memory_frames }o..o{ kg_entity_frames : "frame_id (table not in schema.ts)"
meta {
TEXT key PK
TEXT value
}
identity {
INTEGER id PK "CHECK id=1"
TEXT name
TEXT role
TEXT department
TEXT personality
TEXT capabilities
TEXT system_prompt
TEXT created_at
TEXT updated_at
}
awareness {
INTEGER id PK
TEXT category "task|action|pending|flag"
TEXT content
INTEGER priority
TEXT metadata "JSON"
TEXT created_at
TEXT expires_at "nullable"
}
sessions {
INTEGER id PK
TEXT gop_id UK
TEXT project_id "nullable"
TEXT status "active|closed|archived"
TEXT started_at
TEXT ended_at "nullable"
TEXT summary "nullable"
}
memory_frames {
INTEGER id PK
TEXT frame_type "I|P|B"
TEXT gop_id FK
INTEGER t
INTEGER base_frame_id FK "nullable self"
TEXT content
TEXT importance "critical..deprecated"
TEXT source "user_stated..system"
INTEGER access_count
TEXT created_at
TEXT last_accessed
}
memory_frames_fts {
TEXT content "FTS5 rowid=id"
}
memory_frames_vec {
FLOAT embedding "float[1024] rowid=id"
}
knowledge_entities {
INTEGER id PK
TEXT entity_type
TEXT name
TEXT properties "JSON"
TEXT valid_from
TEXT valid_to "nullable=active"
TEXT recorded_at
}
knowledge_relations {
INTEGER id PK
INTEGER source_id FK
INTEGER target_id FK
TEXT relation_type
REAL confidence
TEXT properties "JSON"
TEXT valid_from
TEXT valid_to "nullable=active"
TEXT recorded_at
}
improvement_signals {
INTEGER id PK
TEXT category
TEXT pattern_key
TEXT detail
INTEGER count
TEXT first_seen
TEXT last_seen
INTEGER surfaced
TEXT surfaced_at "nullable"
TEXT metadata "JSON"
}
install_audit {
INTEGER id PK
TEXT timestamp
TEXT capability_name
TEXT capability_type
TEXT source
TEXT version "nullable"
TEXT risk_level "low|medium|high"
TEXT trust_source
TEXT approval_class
TEXT action
TEXT initiator
TEXT detail
}
procedures {
INTEGER id PK
TEXT name
TEXT model
TEXT template
INTEGER version
REAL success_rate
REAL avg_cost
TEXT created_at
TEXT updated_at
}
ai_interactions {
INTEGER id PK
TEXT timestamp
TEXT workspace_id "nullable"
TEXT session_id "nullable"
TEXT model
TEXT provider
INTEGER input_tokens
INTEGER output_tokens
REAL cost_usd
TEXT tools_called "JSON"
TEXT human_action "nullable"
TEXT risk_context "nullable"
TEXT imported_from "nullable"
TEXT persona "nullable"
TEXT input_text "nullable"
TEXT output_text "nullable"
}
execution_traces {
INTEGER id PK
TEXT session_id "nullable"
TEXT persona_id "nullable"
TEXT workspace_id "nullable"
TEXT model "nullable"
TEXT task_shape "nullable"
TEXT outcome "success..pending"
TEXT trace_json "JSON"
REAL cost_usd
INTEGER duration_ms
TEXT created_at
TEXT finalized_at "nullable"
}
evolution_runs {
INTEGER id PK
TEXT run_uuid UK
TEXT target_kind
TEXT target_name "nullable"
TEXT baseline_text
TEXT winner_text
TEXT winner_schema_json "nullable"
REAL delta_accuracy
TEXT gate_verdict "pass|fail"
TEXT gate_reasons_json "JSON"
TEXT status "proposed..failed"
TEXT artifacts_json "nullable"
TEXT user_note "nullable"
TEXT failure_reason "nullable"
TEXT created_at
TEXT decided_at "nullable"
TEXT deployed_at "nullable"
}
harvest_sources {
INTEGER id PK
TEXT source UK
TEXT display_name
TEXT source_path "nullable"
TEXT last_synced_at "nullable"
INTEGER items_imported
INTEGER frames_created
INTEGER auto_sync
INTEGER sync_interval_hours
TEXT last_content_hash "nullable"
TEXT created_at
}
```
---
## Frontend-relevant gotchas
- **Timestamps are SQLite strings**, not epoch numbers — `datetime('now')` yields `'YYYY-MM-DD HH:MM:SS'` (UTC, space separator). Harvest-overridden frame timestamps may instead be strict ISO-8601 with `T` + tz. Parse defensively.
- **Booleans are INTEGER 0/1** (`awareness`… none; `improvement_signals.surfaced`, `harvest_sources.auto_sync`). No real boolean type.
- **JSON-in-TEXT columns** must be parsed client-side: `awareness.metadata`, `*.properties`, `improvement_signals.metadata`, `ai_interactions.tools_called`, `execution_traces.trace_json`, `evolution_runs.*_json`, and B-frame `memory_frames.content`.
- **`ai_interactions` is immutable** — the UI must not offer edit/delete on audit rows; the DB triggers will reject the write.
- **Knowledge graph is bitemporal** — "current" entities/relations are those with `valid_to IS NULL`; "deletes" are retirements (set `valid_to`), so a hidden node may still exist with a closed validity window.

575
docs/backend-map/sections/02b-data-model-relational.md Normal file
View File

@@ -0,0 +1,575 @@
# 02b · Data Model — Relational Layer (Team / Cloud, Postgres + Drizzle)
## Purpose
This section documents the **TEAM / CLOUD relational layer** of Waggle OS: a PostgreSQL database accessed through Drizzle ORM, defined in `packages/server/src/db/schema.ts` (20 tables). This is the **shared, multi-user, team-scoped** store — distinct from the **per-workspace SQLite memory layer** (`packages/hive-mind-core/src/mind/`, `better-sqlite3` + `sqlite-vec`) which holds a single user's private memory frames, knowledge graph, identity, and awareness. Treat the shapes below as a hard contract: every column name, Postgres type, default, and foreign key here is copied verbatim from the schema and its generated migrations — do not invent fields.
---
## Two distinct data layers (do not confuse them)
| Aspect | Relational layer (THIS section) | SQLite memory layer (separate) |
|---|---|---|
| Engine | PostgreSQL | SQLite (`better-sqlite3` + `sqlite-vec`) |
| Access | Drizzle ORM (`drizzle-orm/postgres-js`) | Direct `@waggle/hive-mind-core` API |
| Scope | Teams, users, agents, tasks, jobs, governance | One workspace's private memory |
| Location | Cloud / server (`DATABASE_URL`) | Local file per workspace (`mind_path`) |
| Defined in | `packages/server/src/db/schema.ts` | `packages/hive-mind-core/src/mind/` |
| Link between them | `users.mindPath` (text) points at the user's local SQLite mind file |
The only bridge column is `users.mind_path` — a nullable `text` pointer to where a user's private SQLite "mind" lives. There are **no cross-database foreign keys**; the layers are joined only in application code.
---
## Connection & migration mechanics
From `packages/server/src/db/connection.ts`:
```ts
export function createDb(connectionString: string) {
const client = postgres(connectionString);
return drizzle(client, { schema });
}
export type Db = ReturnType<typeof createDb>;
export type DbTransaction = Parameters<Parameters<Db['transaction']>[0]>[0];
export type DbExecutor = Db | DbTransaction; // root db OR an open transaction
```
- Driver: `postgres` (postgres-js) wrapped by `drizzle()`.
- `DbExecutor` is the type passed around so repository functions accept **either** the root `db` **or** a live transaction — useful to know if the frontend's backend-for-frontend wraps multi-step writes in transactions.
- Migrations (`packages/server/src/db/migrate.ts`): requires `DATABASE_URL` env var (throws if absent), runs `migrate(db, { migrationsFolder: './drizzle' })`, then `process.exit(0)`.
- Config (`packages/server/drizzle.config.ts`): dialect `postgresql`, schema `./src/db/schema.ts`, output `./drizzle`, dev fallback URL `postgres://waggle:waggle_dev@localhost:5434/waggle` (port **5434**).
- Two migration files exist: `0000_wild_glorian.sql` (18 tables) and `0001_redundant_sauron.sql` (the 3 `team_capability_*` tables).
> **FK delete behavior (uniform):** every foreign key in both migrations is `ON DELETE no action ON UPDATE no action`. There are **no cascades** — deleting a `users` or `teams` row will be **blocked** by Postgres if any child row references it. The frontend must not assume cascading cleanup.
---
## Column type legend
| Drizzle builder | Postgres type | Notes |
|---|---|---|
| `uuid().defaultRandom()` | `uuid DEFAULT gen_random_uuid()` | All primary keys |
| `text()` | `text` | Strings |
| `timestamp({ withTimezone: true })` | `timestamp with time zone` | All timestamps are TZ-aware |
| `boolean()` | `boolean` | |
| `real()` | `real` | Floats (scores, confidence, ratings) |
| `integer()` | `integer` | Counts / ordering |
| `jsonb().$type<T>()` | `jsonb` | Typed JSON blobs — shapes given per column |
`.notNull()` = NOT NULL. `.unique()` = UNIQUE constraint. `.default(x)` = column default. `.references(() => t.col)` = foreign key.
---
## The 20 tables
### Identity & membership
#### `users`
The account record. `clerkId` ties to Clerk auth; `mindPath` points at the private SQLite mind.
| Column | Type | Constraints / Default |
|---|---|---|
| `id` | uuid | PK, `defaultRandom()` |
| `clerk_id` | text | **UNIQUE**, NOT NULL |
| `display_name` | text | NOT NULL |
| `email` | text | **UNIQUE**, NOT NULL |
| `avatar_url` | text | nullable |
| `mind_path` | text | nullable — pointer to user's SQLite memory file |
| `created_at` | timestamptz | NOT NULL, `defaultNow()` |
| `updated_at` | timestamptz | NOT NULL, `defaultNow()` |
#### `teams`
A workspace/organization. `slug` is the URL-safe unique key; `ownerId` is the founding user.
| Column | Type | Constraints / Default |
|---|---|---|
| `id` | uuid | PK, `defaultRandom()` |
| `name` | text | NOT NULL |
| `slug` | text | **UNIQUE**, NOT NULL |
| `owner_id` | uuid | NOT NULL → `users.id` |
| `created_at` | timestamptz | NOT NULL, `defaultNow()` |
#### `team_members` (junction, composite PK)
Who belongs to which team, with role + interests. **Composite primary key `(team_id, user_id)`** — a user can appear once per team.
| Column | Type | Constraints / Default |
|---|---|---|
| `team_id` | uuid | PK part, NOT NULL → `teams.id` |
| `user_id` | uuid | PK part, NOT NULL → `users.id` |
| `role` | text | NOT NULL, default `'member'` |
| `role_description` | text | nullable |
| `interests` | jsonb (`string[]`) | nullable |
| `joined_at` | timestamptz | NOT NULL, `defaultNow()` |
### Agents
#### `agents`
A configured agent owned by a user, optionally scoped to a team.
| Column | Type | Constraints / Default |
|---|---|---|
| `id` | uuid | PK, `defaultRandom()` |
| `user_id` | uuid | NOT NULL → `users.id` |
| `team_id` | uuid | nullable → `teams.id` |
| `name` | text | NOT NULL |
| `role` | text | nullable |
| `system_prompt` | text | nullable |
| `model` | text | NOT NULL, default `'claude-haiku-4-5'` |
| `tools` | jsonb (`string[]`) | NOT NULL, default `[]` |
| `config` | jsonb (`Record<string, unknown>`) | NOT NULL, default `{}` |
| `created_at` | timestamptz | NOT NULL, `defaultNow()` |
#### `agent_groups`
A named collection of agents with an execution `strategy`.
| Column | Type | Constraints / Default |
|---|---|---|
| `id` | uuid | PK, `defaultRandom()` |
| `user_id` | uuid | NOT NULL → `users.id` |
| `name` | text | NOT NULL |
| `description` | text | nullable |
| `strategy` | text | NOT NULL, default `'parallel'` |
| `created_at` | timestamptz | NOT NULL, `defaultNow()` |
#### `agent_group_members` (junction, composite PK)
Maps agents into groups with an in-group role and ordering. **Composite PK `(group_id, agent_id)`.**
| Column | Type | Constraints / Default |
|---|---|---|
| `group_id` | uuid | PK part, NOT NULL → `agent_groups.id` |
| `agent_id` | uuid | PK part, NOT NULL → `agents.id` |
| `role_in_group` | text | NOT NULL, default `'worker'` |
| `execution_order` | integer | NOT NULL, default `0` |
### Collaboration: tasks & messages
#### `tasks`
Team work items. Self-references for subtasks via `parent_task_id` (note: `parentTaskId` is a plain `uuid` column with **no `.references()`** — it is NOT an enforced FK in the schema).
| Column | Type | Constraints / Default |
|---|---|---|
| `id` | uuid | PK, `defaultRandom()` |
| `team_id` | uuid | NOT NULL → `teams.id` |
| `title` | text | NOT NULL |
| `description` | text | nullable |
| `status` | text | NOT NULL, default `'open'` |
| `priority` | text | NOT NULL, default `'normal'` |
| `created_by` | uuid | NOT NULL → `users.id` |
| `assigned_to` | uuid | nullable → `users.id` |
| `parent_task_id` | uuid | nullable — **no FK constraint** (logical self-ref only) |
| `created_at` | timestamptz | NOT NULL, `defaultNow()` |
| `updated_at` | timestamptz | NOT NULL, `defaultNow()` |
#### `messages`
Team message bus. `content` is a typed jsonb blob; `routing` records targeted delivery with reasons.
| Column | Type | Constraints / Default |
|---|---|---|
| `id` | uuid | PK, `defaultRandom()` |
| `team_id` | uuid | NOT NULL → `teams.id` |
| `sender_id` | uuid | NOT NULL → `users.id` |
| `type` | text | NOT NULL |
| `subtype` | text | NOT NULL |
| `content` | jsonb (`Record<string, unknown>`) | NOT NULL |
| `reference_id` | uuid | nullable — **no FK** (generic reference) |
| `routing` | jsonb (`Array<{ userId: string; reason: string }>`) | nullable |
| `created_at` | timestamptz | NOT NULL, `defaultNow()` |
### Shared team knowledge graph
> This is a **team-shared, Postgres** mini knowledge graph — separate from each user's private SQLite KnowledgeGraph. Entities are bi-temporal (`valid_from` / `valid_to`).
#### `team_entities`
Shared facts/entities contributed by team members.
| Column | Type | Constraints / Default |
|---|---|---|
| `id` | uuid | PK, `defaultRandom()` |
| `team_id` | uuid | NOT NULL → `teams.id` |
| `entity_type` | text | NOT NULL |
| `name` | text | NOT NULL |
| `properties` | jsonb (`Record<string, unknown>`) | NOT NULL, default `{}` |
| `shared_by` | uuid | NOT NULL → `users.id` |
| `valid_from` | timestamptz | NOT NULL, `defaultNow()` |
| `valid_to` | timestamptz | nullable (open-ended validity) |
| `created_at` | timestamptz | NOT NULL, `defaultNow()` |
#### `team_relations`
Edges between `team_entities` with a confidence score.
| Column | Type | Constraints / Default |
|---|---|---|
| `id` | uuid | PK, `defaultRandom()` |
| `team_id` | uuid | NOT NULL → `teams.id` |
| `source_id` | uuid | NOT NULL → `team_entities.id` |
| `target_id` | uuid | NOT NULL → `team_entities.id` |
| `relation_type` | text | NOT NULL |
| `confidence` | real | NOT NULL, default `1.0` |
| `properties` | jsonb (`Record<string, unknown>`) | NOT NULL, default `{}` |
| `created_at` | timestamptz | NOT NULL, `defaultNow()` |
#### `team_resources`
Shared reusable assets (prompts, tools, docs) with social signals.
| Column | Type | Constraints / Default |
|---|---|---|
| `id` | uuid | PK, `defaultRandom()` |
| `team_id` | uuid | NOT NULL → `teams.id` |
| `resource_type` | text | NOT NULL |
| `name` | text | NOT NULL |
| `description` | text | nullable |
| `config` | jsonb (`Record<string, unknown>`) | NOT NULL (no default) |
| `shared_by` | uuid | NOT NULL → `users.id` |
| `rating` | real | NOT NULL, default `0` |
| `use_count` | integer | NOT NULL, default `0` |
| `created_at` | timestamptz | NOT NULL, `defaultNow()` |
### Team governance / capability control
> These three tables (added in migration `0001`) implement team-level capability governance: standing policies per role, one-off allow/deny overrides, and a request→decision approval queue.
#### `team_capability_policies`
Standing policy per team role.
| Column | Type | Constraints / Default |
|---|---|---|
| `id` | uuid | PK, `defaultRandom()` |
| `team_id` | uuid | NOT NULL → `teams.id` |
| `role` | text | NOT NULL |
| `allowed_sources` | jsonb (`string[]`) | NOT NULL, default `[]` |
| `blocked_tools` | jsonb (`string[]`) | NOT NULL, default `[]` |
| `approval_threshold` | text | NOT NULL, default `'none'` |
| `updated_by` | uuid | nullable → `users.id` |
| `created_at` | timestamptz | NOT NULL, `defaultNow()` |
| `updated_at` | timestamptz | NOT NULL, `defaultNow()` |
#### `team_capability_overrides`
Explicit per-capability allow/deny decisions (`decision`).
| Column | Type | Constraints / Default |
|---|---|---|
| `id` | uuid | PK, `defaultRandom()` |
| `team_id` | uuid | NOT NULL → `teams.id` |
| `capability_name` | text | NOT NULL |
| `capability_type` | text | NOT NULL |
| `decision` | text | NOT NULL |
| `reason` | text | NOT NULL, default `''` |
| `decided_by` | uuid | NOT NULL → `users.id` |
| `created_at` | timestamptz | NOT NULL, `defaultNow()` |
| `decided_at` | timestamptz | NOT NULL, `defaultNow()` |
#### `team_capability_requests`
A member's request for a capability + its approval lifecycle.
| Column | Type | Constraints / Default |
|---|---|---|
| `id` | uuid | PK, `defaultRandom()` |
| `team_id` | uuid | NOT NULL → `teams.id` |
| `requested_by` | uuid | NOT NULL → `users.id` |
| `capability_name` | text | NOT NULL |
| `capability_type` | text | NOT NULL |
| `justification` | text | NOT NULL |
| `status` | text | NOT NULL, default `'pending'` |
| `decided_by` | uuid | nullable → `users.id` |
| `decision_reason` | text | nullable |
| `created_at` | timestamptz | NOT NULL, `defaultNow()` |
| `decided_at` | timestamptz | nullable |
### Background work & scheduling
#### `agent_jobs`
Async agent job queue with input/output blobs and lifecycle timestamps.
| Column | Type | Constraints / Default |
|---|---|---|
| `id` | uuid | PK, `defaultRandom()` |
| `team_id` | uuid | NOT NULL → `teams.id` |
| `user_id` | uuid | NOT NULL → `users.id` |
| `job_type` | text | NOT NULL |
| `status` | text | NOT NULL, default `'queued'` |
| `input` | jsonb (`Record<string, unknown>`) | NOT NULL |
| `output` | jsonb (`Record<string, unknown>`) | nullable |
| `started_at` | timestamptz | nullable |
| `completed_at` | timestamptz | nullable |
| `created_at` | timestamptz | NOT NULL, `defaultNow()` |
#### `cron_schedules`
Recurring job definitions (cron expression + config) with run bookkeeping.
| Column | Type | Constraints / Default |
|---|---|---|
| `id` | uuid | PK, `defaultRandom()` |
| `team_id` | uuid | NOT NULL → `teams.id` |
| `created_by` | uuid | NOT NULL → `users.id` |
| `name` | text | NOT NULL |
| `cron_expr` | text | NOT NULL |
| `job_type` | text | NOT NULL |
| `job_config` | jsonb (`Record<string, unknown>`) | NOT NULL, default `{}` |
| `enabled` | boolean | NOT NULL, default `true` |
| `last_run_at` | timestamptz | nullable |
| `next_run_at` | timestamptz | nullable |
| `created_at` | timestamptz | NOT NULL, `defaultNow()` |
### Proactive / scout intelligence
#### `scout_findings`
Discovered items (news, signals) scoped to a user and/or team. **Both `user_id` and `team_id` are nullable** → a finding can be global, user-only, or team-only.
| Column | Type | Constraints / Default |
|---|---|---|
| `id` | uuid | PK, `defaultRandom()` |
| `user_id` | uuid | nullable → `users.id` |
| `team_id` | uuid | nullable → `teams.id` |
| `source` | text | NOT NULL |
| `category` | text | NOT NULL |
| `title` | text | NOT NULL |
| `summary` | text | nullable |
| `relevance_score` | real | NOT NULL, default `0` |
| `url` | text | nullable |
| `status` | text | NOT NULL, default `'new'` |
| `created_at` | timestamptz | NOT NULL, `defaultNow()` |
#### `proactive_patterns`
Reusable trigger→suggestion templates. **Has no `created_at` and no FKs** — it is a standalone config/lookup table.
| Column | Type | Constraints / Default |
|---|---|---|
| `id` | uuid | PK, `defaultRandom()` |
| `name` | text | NOT NULL |
| `trigger` | jsonb (`Record<string, unknown>`) | NOT NULL |
| `suggestion_type` | text | NOT NULL |
| `template` | text | NOT NULL |
| `enabled` | boolean | NOT NULL, default `true` |
#### `suggestions_log`
Records suggestions fired for a user from a `proactive_patterns` row.
| Column | Type | Constraints / Default |
|---|---|---|
| `id` | uuid | PK, `defaultRandom()` |
| `user_id` | uuid | NOT NULL → `users.id` |
| `pattern_id` | uuid | NOT NULL → `proactive_patterns.id` |
| `context` | jsonb (`Record<string, unknown>`) | NOT NULL |
| `status` | text | NOT NULL, default `'pending'` |
| `created_at` | timestamptz | NOT NULL, `defaultNow()` |
### Audit
#### `agent_audit_log`
Immutable audit trail of agent actions with before/after snapshots and an approval gate.
| Column | Type | Constraints / Default |
|---|---|---|
| `id` | uuid | PK, `defaultRandom()` |
| `user_id` | uuid | NOT NULL → `users.id` |
| `team_id` | uuid | nullable → `teams.id` |
| `agent_name` | text | NOT NULL |
| `action_type` | text | NOT NULL |
| `description` | text | NOT NULL |
| `before_state` | jsonb (`Record<string, unknown>`) | nullable |
| `after_state` | jsonb (`Record<string, unknown>`) | nullable |
| `requires_approval` | boolean | NOT NULL, default `false` |
| `approved` | boolean | nullable (tri-state: null = undecided) |
| `approved_by` | uuid | nullable → `users.id` |
| `created_at` | timestamptz | NOT NULL, `defaultNow()` |
---
## `text`-as-enum reference (no DB-level enums)
There are **no Postgres `enum` types** in this schema — every "enum-like" field is a free `text` column with a default. The frontend should treat these as the canonical values it sends/reads (defaults shown):
| Table.column | Default | Meaning |
|---|---|---|
| `team_members.role` | `'member'` | membership role |
| `agents.model` | `'claude-haiku-4-5'` | default LLM |
| `agent_groups.strategy` | `'parallel'` | group execution strategy |
| `agent_group_members.role_in_group` | `'worker'` | role within a group |
| `tasks.status` | `'open'` | task lifecycle |
| `tasks.priority` | `'normal'` | task priority |
| `agent_jobs.status` | `'queued'` | job lifecycle |
| `team_capability_policies.approval_threshold` | `'none'` | when approval is required |
| `team_capability_requests.status` | `'pending'` | request lifecycle |
| `scout_findings.status` | `'new'` | finding triage state |
| `suggestions_log.status` | `'pending'` | suggestion state |
| `cron_schedules.enabled` | `true` (boolean) | schedule on/off |
| `proactive_patterns.enabled` | `true` (boolean) | pattern on/off |
> The set of allowed values beyond the default is NOT constrained in the database; consult the route/service layer for the full vocabularies.
---
## Entity-relationship diagram
```mermaid
erDiagram
users ||--o{ teams : "owns (owner_id)"
users ||--o{ team_members : "is"
teams ||--o{ team_members : "has"
users ||--o{ agents : "owns"
teams ||--o{ agents : "scopes"
users ||--o{ agent_groups : "owns"
agent_groups ||--o{ agent_group_members : "contains"
agents ||--o{ agent_group_members : "joins"
teams ||--o{ tasks : "has"
users ||--o{ tasks : "creates (created_by)"
users ||--o{ tasks : "assigned (assigned_to)"
teams ||--o{ messages : "channel"
users ||--o{ messages : "sends"
teams ||--o{ team_entities : "owns"
users ||--o{ team_entities : "shares"
teams ||--o{ team_relations : "owns"
team_entities ||--o{ team_relations : "source"
team_entities ||--o{ team_relations : "target"
teams ||--o{ team_resources : "owns"
users ||--o{ team_resources : "shares"
teams ||--o{ team_capability_policies : "governs"
users ||--o{ team_capability_policies : "updates"
teams ||--o{ team_capability_overrides : "governs"
users ||--o{ team_capability_overrides : "decides"
teams ||--o{ team_capability_requests : "scopes"
users ||--o{ team_capability_requests : "requests/decides"
teams ||--o{ agent_jobs : "owns"
users ||--o{ agent_jobs : "runs"
teams ||--o{ cron_schedules : "owns"
users ||--o{ cron_schedules : "creates"
users ||--o{ scout_findings : "for-user"
teams ||--o{ scout_findings : "for-team"
proactive_patterns ||--o{ suggestions_log : "fires"
users ||--o{ suggestions_log : "receives"
users ||--o{ agent_audit_log : "acts"
teams ||--o{ agent_audit_log : "scopes"
users {
uuid id PK
text clerk_id UK
text email UK
text display_name
text mind_path "→ SQLite mind"
}
teams {
uuid id PK
text slug UK
uuid owner_id FK
}
team_members {
uuid team_id PK_FK
uuid user_id PK_FK
text role
jsonb interests
}
agents {
uuid id PK
uuid user_id FK
uuid team_id FK "nullable"
text model
jsonb tools
jsonb config
}
agent_groups {
uuid id PK
uuid user_id FK
text strategy
}
agent_group_members {
uuid group_id PK_FK
uuid agent_id PK_FK
integer execution_order
}
tasks {
uuid id PK
uuid team_id FK
uuid created_by FK
uuid assigned_to FK "nullable"
uuid parent_task_id "no FK"
text status
}
messages {
uuid id PK
uuid team_id FK
uuid sender_id FK
jsonb content
jsonb routing
}
team_entities {
uuid id PK
uuid team_id FK
uuid shared_by FK
timestamptz valid_to "nullable"
}
team_relations {
uuid id PK
uuid source_id FK
uuid target_id FK
real confidence
}
team_resources {
uuid id PK
uuid team_id FK
uuid shared_by FK
real rating
}
team_capability_policies {
uuid id PK
uuid team_id FK
jsonb blocked_tools
}
team_capability_overrides {
uuid id PK
uuid team_id FK
text decision
}
team_capability_requests {
uuid id PK
uuid team_id FK
text status
}
agent_jobs {
uuid id PK
uuid team_id FK
uuid user_id FK
text status
jsonb input
}
cron_schedules {
uuid id PK
uuid team_id FK
text cron_expr
boolean enabled
}
scout_findings {
uuid id PK
uuid user_id FK "nullable"
uuid team_id FK "nullable"
real relevance_score
}
proactive_patterns {
uuid id PK
jsonb trigger
boolean enabled
}
suggestions_log {
uuid id PK
uuid user_id FK
uuid pattern_id FK
}
agent_audit_log {
uuid id PK
uuid user_id FK
uuid team_id FK "nullable"
boolean approved "nullable"
}
```
---
## Notes for the frontend rebuild
- **All IDs are server-generated UUIDs** (`gen_random_uuid()`). The client never invents IDs; it reads them back from create responses.
- **All timestamps are `timestamp with time zone`** — expect ISO-8601 strings with offsets; render in the user's locale.
- **JSONB columns are opaque blobs with documented TS shapes** (`tools: string[]`, `config: Record<string, unknown>`, `routing: {userId, reason}[]`, `interests: string[]`, etc.). Send/receive these as plain JSON objects.
- **Two junction tables use composite PKs** (`team_members`, `agent_group_members`) — there is no surrogate id, so update/delete by the pair of FK columns.
- **No cascade deletes anywhere** — the UI must surface "cannot delete, still referenced" errors and/or delete children first.
- **`proactive_patterns` is config-only** (no FK, no timestamp) — likely seeded/admin-managed, not user-CRUD.
- **`users.mind_path`** is the only link from this cloud DB to a user's private local SQLite memory; the relational layer never stores memory frames.

646
docs/backend-map/sections/02c-shared-types-tiers.md Normal file
View File

@@ -0,0 +1,646 @@
# 02c · Shared Types, Zod Schemas & the 5-Tier Model (`@waggle/shared`)
**Purpose.** This section is the **wire contract** for the Waggle OS frontend. Every interface, enum, Zod request schema, and tier-capability flag documented here lives in `packages/shared/src/` and is consumed by both the Node.js sidecar (Fastify) and the web bundle. If you are rebuilding the frontend in Lovable, these are the **exact shapes you receive from and send to the backend** — the field names, types, and allowed enum values are quoted verbatim from source. Nothing here is invented; every claim is grounded in the four files read.
**Source files (all present, none empty):**
| File | What it holds |
|---|---|
| `packages/shared/src/types.ts` | Domain interfaces + string-literal union types (the wire shapes) |
| `packages/shared/src/schemas.ts` | Zod schemas validating inbound API request bodies |
| `packages/shared/src/constants.ts` | Canonical enum arrays + a few tuning constants |
| `packages/shared/src/tiers.ts` | `TIERS`, `TierCapabilities`, `TIER_CAPABILITIES`, tier helper functions |
> The barrel `packages/shared/src/index.ts` re-exports these four plus `mcp-catalog.js`, `connector-recommendations.js`, and `tool-detection.js` (out of scope for this section). Import everything below from `@waggle/shared`.
---
## 1. Conventions you must know before reading the tables
- **`Date` fields are serialized as ISO 8601 strings over the wire.** The TypeScript interfaces declare `createdAt: Date`, etc. — but JSON has no `Date`, so on the frontend you receive strings (e.g. `"2026-06-06T12:00:00.000Z"`) and must parse them. The `tiers.ts` helpers (`isTrialExpired`, `trialDaysRemaining`) explicitly take **ISO date strings**, confirming this serialization boundary.
- **`Record<string, unknown>` = arbitrary JSON object.** Many fields (`config`, `content`, `properties`, `input`, `output`, `jobConfig`, `context`, `trigger`, `beforeState`, `afterState`) are open-ended JSON blobs. The backend does not constrain their inner shape at the type level.
- **`| null` vs `?` (optional).** Interfaces use `| null` for fields that are **always present in the row but may be empty** (e.g. `avatarUrl: string | null`). Zod schemas use `.optional()` for fields that **may be absent from the request body**. These are different — respect both.
- **`-1` means "unlimited"** throughout `TierCapabilities` (e.g. `connectorLimit: -1`). This is enforced in `hasCapability()` where `cap === -1` short-circuits to `true`.
---
## 2. Auth & Users
### `User` (`types.ts`)
| Field | Type | Notes |
|---|---|---|
| `id` | `string` | Internal Waggle user id |
| `clerkId` | `string` | Clerk auth provider id (auth is Clerk-backed) |
| `displayName` | `string` | |
| `email` | `string` | |
| `avatarUrl` | `string \| null` | |
| `mindPath` | `string \| null` | Filesystem path to the user's personal `.mind` memory DB |
| `createdAt` | `Date` | ISO string on the wire |
| `updatedAt` | `Date` | ISO string on the wire |
---
## 3. Teams
### `Team` (`types.ts`)
| Field | Type | Notes |
|---|---|---|
| `id` | `string` | |
| `name` | `string` | |
| `slug` | `string` | URL-safe; used as the WebSocket room key (`teamSlug`) |
| `ownerId` | `string` | FK → `User.id` |
| `createdAt` | `Date` | |
### `TeamRole` (`types.ts`)
```ts
type TeamRole = 'owner' | 'admin' | 'member';
```
Canonical array in `constants.ts`: `TEAM_ROLES = ['owner', 'admin', 'member']`.
### `TeamMember` (`types.ts`)
| Field | Type | Notes |
|---|---|---|
| `teamId` | `string` | |
| `userId` | `string` | |
| `role` | `TeamRole` | `'owner' \| 'admin' \| 'member'` |
| `roleDescription` | `string \| null` | Free-text role blurb |
| `interests` | `string[] \| null` | Used for WaggleDance routing/matching |
| `joinedAt` | `Date` | |
---
## 4. Agents & Agent Groups
### `AgentDef` (`types.ts`)
| Field | Type | Notes |
|---|---|---|
| `id` | `string` | |
| `userId` | `string` | Owner |
| `teamId` | `string \| null` | Null = personal agent |
| `name` | `string` | |
| `role` | `string \| null` | |
| `systemPrompt` | `string \| null` | |
| `model` | `string` | e.g. `"claude-haiku-4-5"` |
| `tools` | `string[]` | Tool ids the agent may call |
| `config` | `Record<string, unknown>` | Open JSON |
| `createdAt` | `Date` | |
### `AgentGroupStrategy` (`types.ts`)
```ts
type AgentGroupStrategy = 'parallel' | 'sequential' | 'coordinator';
```
Canonical array: `AGENT_GROUP_STRATEGIES = ['parallel', 'sequential', 'coordinator']`.
### `AgentGroup` (`types.ts`)
| Field | Type |
|---|---|
| `id` | `string` |
| `userId` | `string` |
| `name` | `string` |
| `description` | `string \| null` |
| `strategy` | `AgentGroupStrategy` |
| `createdAt` | `Date` |
### `AgentGroupMember` (`types.ts`)
| Field | Type | Notes |
|---|---|---|
| `groupId` | `string` | |
| `agentId` | `string` | |
| `roleInGroup` | `'lead' \| 'worker'` | |
| `executionOrder` | `number` | Ordering within `sequential` strategy |
---
## 5. Tasks
### Enums (`types.ts` + `constants.ts`)
```ts
type TaskStatus = 'open' | 'claimed' | 'in_progress' | 'done' | 'cancelled';
type TaskPriority = 'critical' | 'high' | 'normal' | 'low';
```
Arrays: `TASK_STATUSES`, `TASK_PRIORITIES`.
### `Task` (`types.ts`)
| Field | Type | Notes |
|---|---|---|
| `id` | `string` | |
| `teamId` | `string` | Tasks are team-scoped |
| `title` | `string` | |
| `description` | `string \| null` | |
| `status` | `TaskStatus` | |
| `priority` | `TaskPriority` | |
| `createdBy` | `string` | FK → `User.id` |
| `assignedTo` | `string \| null` | |
| `parentTaskId` | `string \| null` | Subtask tree |
| `createdAt` | `Date` | |
| `updatedAt` | `Date` | |
---
## 6. WaggleDance Messages (multi-agent coordination bus)
### Enums (`types.ts` + `constants.ts`)
```ts
type MessageType = 'broadcast' | 'request' | 'response';
type MessageSubtype =
| 'knowledge_check' | 'task_delegation' | 'skill_request'
| 'model_recommendation' | 'knowledge_match' | 'task_claim'
| 'discovery' | 'routed_share' | 'skill_share' | 'model_recipe';
```
`constants.ts` only exports `MESSAGE_TYPES = ['broadcast','request','response']` — the **10 subtypes are NOT in `constants.ts`**; their canonical list is the `sendMessageSchema` enum in `schemas.ts` (and the `MessageSubtype` union).
### `WaggleMessage` (`types.ts`)
| Field | Type | Notes |
|---|---|---|
| `id` | `string` | |
| `teamId` | `string` | |
| `senderId` | `string` | |
| `type` | `MessageType` | |
| `subtype` | `MessageSubtype` | |
| `content` | `Record<string, unknown>` | Subtype-specific payload (open JSON) |
| `referenceId` | `string \| null` | Links a `response` to its `request` |
| `routing` | `Array<{ userId: string; reason: string }> \| null` | Targeted-share recipients + reasons |
| `createdAt` | `Date` | |
---
## 7. Team Knowledge Graph
### `TeamEntity` (`types.ts`)
| Field | Type | Notes |
|---|---|---|
| `id` | `string` | |
| `teamId` | `string` | |
| `entityType` | `string` | Free-form type label |
| `name` | `string` | |
| `properties` | `Record<string, unknown>` | |
| `sharedBy` | `string` | FK → `User.id` |
| `validFrom` | `Date` | Bitemporal validity start |
| `validTo` | `Date \| null` | Null = still valid |
| `createdAt` | `Date` | |
### `TeamRelation` (`types.ts`)
| Field | Type | Notes |
|---|---|---|
| `id` | `string` | |
| `teamId` | `string` | |
| `sourceId` | `string` | FK → `TeamEntity.id` |
| `targetId` | `string` | FK → `TeamEntity.id` |
| `relationType` | `string` | |
| `confidence` | `number` | `0.0`–`1.0` (default `1.0` in schema) |
| `properties` | `Record<string, unknown>` | |
| `createdAt` | `Date` | |
---
## 8. Team Resources (shared model recipes / skills / configs)
### `ResourceType` (`types.ts` + `constants.ts`)
```ts
type ResourceType = 'model_recipe' | 'skill' | 'tool_config' | 'prompt_template';
```
Array: `RESOURCE_TYPES`.
### `TeamResource` (`types.ts`)
| Field | Type | Notes |
|---|---|---|
| `id` | `string` | |
| `teamId` | `string` | |
| `resourceType` | `ResourceType` | |
| `name` | `string` | |
| `description` | `string \| null` | |
| `config` | `Record<string, unknown>` | |
| `sharedBy` | `string` | |
| `rating` | `number` | |
| `useCount` | `number` | |
| `createdAt` | `Date` | |
---
## 9. Jobs & Cron
### Enums (`types.ts` + `constants.ts`)
```ts
type JobType = 'chat' | 'task' | 'cron' | 'waggle';
type JobStatus = 'queued' | 'running' | 'completed' | 'failed';
```
Arrays: `JOB_TYPES`, `JOB_STATUSES`.
### `AgentJob` (`types.ts`)
| Field | Type | Notes |
|---|---|---|
| `id` | `string` | |
| `teamId` | `string` | |
| `userId` | `string` | |
| `jobType` | `JobType` | |
| `status` | `JobStatus` | |
| `input` | `Record<string, unknown>` | |
| `output` | `Record<string, unknown> \| null` | Null until complete |
| `startedAt` | `Date \| null` | |
| `completedAt` | `Date \| null` | |
| `createdAt` | `Date` | |
### `CronSchedule` (`types.ts`)
| Field | Type | Notes |
|---|---|---|
| `id` | `string` | |
| `teamId` | `string` | |
| `createdBy` | `string` | |
| `name` | `string` | |
| `cronExpr` | `string` | Standard cron expression |
| `jobType` | `string` | Free-form (note: NOT the `JobType` union here) |
| `jobConfig` | `Record<string, unknown>` | |
| `enabled` | `boolean` | |
| `lastRunAt` | `Date \| null` | |
| `nextRunAt` | `Date \| null` | |
| `createdAt` | `Date` | |
> Default cron for the hive-mind compile job: `HIVE_MIND_CRON = '0 9 * * 1'` (weekly, Monday 9am) — from `constants.ts`.
---
## 10. Intelligence: Scout Findings & Proactive Suggestions
### Enums (`types.ts`)
```ts
type ScoutSource = 'marketplace' | 'mcp_registry' | 'model_provider' | 'team';
type ScoutCategory = 'skill' | 'mcp' | 'model' | 'feature' | 'practice';
type FindingStatus = 'new' | 'presented' | 'adopted' | 'dismissed';
type SuggestionType = 'dashboard' | 'cron' | 'share' | 'skill' | 'upgrade';
type SuggestionStatus = 'pending' | 'accepted' | 'dismissed' | 'snoozed';
```
`SUGGESTION_TYPES` array is in `constants.ts`. (`ScoutSource`/`ScoutCategory`/`FindingStatus`/`SuggestionStatus` have no array constants — the union types are canonical.)
### `ScoutFinding` (`types.ts`)
| Field | Type | Notes |
|---|---|---|
| `id` | `string` | |
| `userId` | `string \| null` | |
| `teamId` | `string \| null` | At least one of user/team scopes it |
| `source` | `ScoutSource` | |
| `category` | `ScoutCategory` | |
| `title` | `string` | |
| `summary` | `string \| null` | |
| `relevanceScore` | `number` | |
| `url` | `string \| null` | |
| `status` | `FindingStatus` | |
| `createdAt` | `Date` | |
### `ProactivePattern` (`types.ts`)
| Field | Type |
|---|---|
| `id` | `string` |
| `name` | `string` |
| `trigger` | `Record<string, unknown>` |
| `suggestionType` | `SuggestionType` |
| `template` | `string` |
| `enabled` | `boolean` |
### `SuggestionEntry` (`types.ts`)
| Field | Type | Notes |
|---|---|---|
| `id` | `string` | |
| `userId` | `string` | |
| `patternId` | `string` | FK → `ProactivePattern.id` |
| `context` | `Record<string, unknown>` | |
| `status` | `SuggestionStatus` | |
| `createdAt` | `Date` | |
> Tuning constants (`constants.ts`): `MAX_SUGGESTIONS_PER_INTERACTION = 1`, `SCOUT_DEFAULT_INTERVAL_MS = 86_400_000` (daily), `SUBCONSCIOUS_INTERACTION_THRESHOLD = 10` (reflect every 10 tasks).
---
## 11. Audit
### `AuditEntry` (`types.ts`)
| Field | Type | Notes |
|---|---|---|
| `id` | `string` | |
| `userId` | `string` | |
| `teamId` | `string \| null` | |
| `agentName` | `string` | |
| `actionType` | `string` | |
| `description` | `string` | |
| `beforeState` | `Record<string, unknown> \| null` | |
| `afterState` | `Record<string, unknown> \| null` | |
| `requiresApproval` | `boolean` | Drives the approvals-inbox UI |
| `approved` | `boolean \| null` | Null = pending decision |
| `approvedBy` | `string \| null` | |
| `createdAt` | `Date` | |
---
## 12. WebSocket Event Contract
These discriminated unions (`types.ts`) define the **real-time channel** the frontend opens. Discriminate on the `type` field.
### Client → Server: `WsClientEvent`
| `type` | Payload fields |
|---|---|
| `'authenticate'` | `token: string` |
| `'join_team'` | `teamSlug: string` |
| `'send_message'` | `teamSlug: string`, `messageType: MessageType`, `subtype: MessageSubtype`, `content: Record<string, unknown>` |
### Server → Client: `WsServerEvent`
| `type` | Payload fields |
|---|---|
| `'waggle_message'` | `message: WaggleMessage` |
| `'task_update'` | `task: Task` |
| `'agent_status'` | `userId: string`, `status: 'running' \| 'idle' \| 'completed'` |
| `'suggestion'` | `suggestion: SuggestionEntry` |
| `'scout_finding'` | `finding: ScoutFinding` |
| `'job_progress'` | `jobId: string`, `progress: Record<string, unknown>` |
> Note: WebSocket `agent_status` uses `'running' \| 'idle' \| 'completed'` — a **different** set than `JobStatus`. Do not conflate them.
---
## 13. Connectors
### `ConnectorCredential` (`types.ts`) — stored in vault, not normally sent to UI
| Field | Type | Notes |
|---|---|---|
| `type` | `'api_key' \| 'oauth2' \| 'bearer' \| 'basic'` | |
| `accessToken?` | `string` | oauth2 |
| `refreshToken?` | `string` | oauth2 |
| `expiresAt?` | `string` | ISO timestamp |
| `scopes?` | `string[]` | oauth2 |
| `apiKey?` | `string` | api_key/bearer |
| `username?` | `string` | basic |
### `ConnectorStatus` (`types.ts`)
```ts
type ConnectorStatus = 'connected' | 'disconnected' | 'expired' | 'error';
```
### `ConnectorActionMeta` (`types.ts`)
| Field | Type |
|---|---|
| `name` | `string` |
| `description` | `string` |
| `riskLevel` | `'low' \| 'medium' \| 'high'` |
### `ConnectorDefinition` (`types.ts`) — **the connector card the user sees**
| Field | Type | Notes |
|---|---|---|
| `id` | `string` | |
| `name` | `string` | |
| `description` | `string` | |
| `service` | `string` | Which external service |
| `authType` | `'api_key' \| 'oauth2' \| 'bearer' \| 'basic'` | |
| `status` | `ConnectorStatus` | Whether creds exist in vault |
| `capabilities` | `('read' \| 'write' \| 'search')[]` | |
| `substrate` | `'waggle' \| 'kvark'` | Which substrate manages it |
| `tools` | `string[]` | Agent tools unlocked when connected |
| `config?` | `Record<string, unknown>` | |
| `actions?` | `ConnectorActionMeta[]` | Present only when SDK connector loaded |
| `logoUrl?` | `string` | CDN SVG logo |
| `category?` | `'productivity' \| 'development' \| 'crm' \| 'data' \| 'communication' \| 'storage' \| 'integration'` | |
| `setupGuide?` | `string` | 1–2 sentences: what credential + where to get it |
### `ConnectorHealth` (`types.ts`) — cockpit health row
| Field | Type |
|---|---|
| `id` | `string` |
| `name` | `string` |
| `status` | `ConnectorStatus` |
| `lastChecked` | `string` (ISO) |
| `error?` | `string` |
| `tokenExpiresAt?` | `string` (ISO) |
---
## 14. Zod Request Schemas (`schemas.ts`)
These validate **inbound request bodies**. The frontend should construct payloads that satisfy them; the backend `.parse()`s them and returns a 400 on failure. Columns: required vs optional, constraints, and defaults the server fills if you omit a field.
| Schema | Field | Type / Enum | Required? | Constraints / Default |
|---|---|---|---|---|
| `createTeamSchema` | `name` | string | required | 1–100 chars |
| | `slug` | string | required | 1–50 chars, regex `^[a-z0-9-]+$` |
| `inviteMemberSchema` | `email` | string | required | valid email |
| | `role` | enum | required | `'admin' \| 'member'` (NOT `'owner'`) |
| `updateMemberSchema` | `role` | enum | optional | `'admin' \| 'member'` |
| | `roleDescription` | string | optional | max 500 |
| | `interests` | string[] | optional | |
| `createTaskSchema` | `title` | string | required | 1–200 |
| | `description` | string | optional | max 5000 |
| | `priority` | enum | optional | `critical\|high\|normal\|low`, **default `'normal'`** |
| | `parentTaskId` | string | optional | UUID |
| `updateTaskSchema` | `title` | string | optional | 1–200 |
| | `description` | string | optional | max 5000 |
| | `status` | enum | optional | `open\|claimed\|in_progress\|done\|cancelled` |
| | `priority` | enum | optional | `critical\|high\|normal\|low` |
| | `assignedTo` | string \| null | optional | UUID or null |
| `sendMessageSchema` | `type` | enum | required | `broadcast\|request\|response` |
| | `subtype` | enum | required | all 10 `MessageSubtype` values |
| | `content` | record | required | arbitrary JSON object |
| | `referenceId` | string | optional | UUID |
| | `routing` | array | optional | `{ userId: uuid, reason: string }[]` |
| `createAgentSchema` | `name` | string | required | 1–100 |
| | `role` | string | optional | max 500 |
| | `systemPrompt` | string | optional | max 10000 |
| | `model` | string | optional | min 1, **default `'claude-haiku-4-5'`** |
| | `tools` | string[] | optional | **default `[]`** |
| | `config` | record | optional | **default `{}`** |
| | `teamId` | string | optional | UUID |
| `createAgentGroupSchema` | `name` | string | required | 1–100 |
| | `description` | string | optional | max 500 |
| | `strategy` | enum | required | `parallel\|sequential\|coordinator` |
| | `members` | array | required | `{ agentId: uuid, roleInGroup: 'lead'\|'worker' (def 'worker'), executionOrder: int≥0 (def 0) }[]` |
| `createEntitySchema` | `entityType` | string | required | 1–100 |
| | `name` | string | required | 1–200 |
| | `properties` | record | optional | **default `{}`** |
| | `validFrom` | string | optional | ISO datetime |
| | `validTo` | string | optional | ISO datetime |
| `createRelationSchema` | `sourceId` | string | required | UUID |
| | `targetId` | string | required | UUID |
| | `relationType` | string | required | 1–100 |
| | `confidence` | number | optional | 0–1, **default `1.0`** |
| | `properties` | record | optional | **default `{}`** |
| `createResourceSchema` | `resourceType` | enum | required | `model_recipe\|skill\|tool_config\|prompt_template` |
| | `name` | string | required | 1–200 |
| | `description` | string | optional | max 1000 |
| | `config` | record | required | arbitrary JSON object |
| `createCronSchema` | `name` | string | required | 1–200 |
| | `cronExpr` | string | required | min 1 |
| | `jobType` | string | required | min 1 (free-form string) |
| | `jobConfig` | record | optional | **default `{}`** |
| `queueJobSchema` | `jobType` | enum | required | `chat\|task\|cron\|waggle` |
| | `input` | record | required | arbitrary JSON object |
| | `teamId` | string | optional | UUID |
**Frontend takeaways from the schemas:**
- Invites can only set `'admin'` or `'member'` — there is no API path to invite an `'owner'`.
- Omitting `priority` on task create yields `'normal'`; omitting `model` on agent create yields `'claude-haiku-4-5'`.
- `createCronSchema.jobType` is a free string, whereas `queueJobSchema.jobType` is the strict 4-value enum.
---
## 15. The 5-Tier Model (`tiers.ts`)
```ts
export const TIERS = ['TRIAL', 'FREE', 'PRO', 'TEAMS', 'ENTERPRISE'] as const;
export type Tier = typeof TIERS[number];
export const TRIAL_DURATION_DAYS = 15;
```
**Pricing (from the file header comment, "confirmed April 12, 2026"):**
| Tier | Price | One-liner |
|---|---|---|
| TRIAL | $0 / 15 days | All features unlocked; falls back to FREE after 15 days |
| FREE | $0 forever | 5 workspaces, agents, built-in skills only |
| PRO | $19/mo | Unlimited, marketplace, all connectors |
| TEAMS | $49/mo per seat | Shared workspaces, WaggleDance, governance |
| ENTERPRISE | Consultative | KVARK sovereign on-prem |
### 15.1 `TierCapabilities` interface — every flag the UI can gate on
| Capability | Type | Meaning for the UI |
|---|---|---|
| `connectorLimit` | `number` | Max connectors; `-1` = unlimited |
| `workspaceLimit` | `number` | Max workspaces; `-1` = unlimited |
| `embeddingProviders` | `EmbeddingProviderType[]` | Allowed embedders (`'inprocess'\|'ollama'\|'voyage'\|'openai'\|'litellm'\|'mock'`) |
| `embeddingQuotaPerMonth` | `number` | `-1` = unlimited everywhere |
| `messageHistoryLimit` | `number` | `-1` = unlimited everywhere |
| `spawnAgents` | `boolean` | Can spawn agents (true in ALL tiers — agents are free) |
| `customSkills` | `boolean` | Author custom skills |
| `teamSkillLibrary` | `boolean` | Shared team skill library |
| `cloudSync` | `boolean` | Cloud sync of memory |
| `exportFormats` | `ExportFormat[]` | Allowed exports (`'txt'\|'md'\|'pdf'\|'json'`) |
| `teamMembersLimit` | `number` | Seats; `-1` = unlimited, `1` = solo |
| `sharedWorkspaces` | `boolean` | Team-shared workspaces |
| `adminPanel` | `boolean` | Show admin panel |
| `auditLog` | `'none' \| 'basic' \| 'full'` | Audit log depth |
| `selfHosted` | `boolean` | On-prem / sovereign |
| `managedModelPool` | `boolean` | Access managed model pool |
| `priorityModels` | `boolean` | Priority (premium) models |
| `kvarkCta` | `'none' \| 'subtle' \| 'active'` | How aggressively to show the KVARK upgrade CTA |
| `stripePriceId` | `string \| null` | Stripe price id (from env at runtime) |
### 15.2 Tier × Capability matrix (verbatim from `TIER_CAPABILITIES`)
| Capability | TRIAL | FREE | PRO | TEAMS | ENTERPRISE |
|---|---|---|---|---|---|
| `connectorLimit` | -1 | **5** | -1 | -1 | -1 |
| `workspaceLimit` | -1 | **5** | -1 | -1 | -1 |
| `embeddingProviders` | all 6 | inprocess, mock, ollama | inprocess, mock, ollama, voyage, openai | all 6 | all 6 |
| `embeddingQuotaPerMonth` | -1 | -1 | -1 | -1 | -1 |
| `messageHistoryLimit` | -1 | -1 | -1 | -1 | -1 |
| `spawnAgents` | ✅ | ✅ | ✅ | ✅ | ✅ |
| `customSkills` | ✅ | ❌ | ✅ | ✅ | ✅ |
| `teamSkillLibrary` | ✅ | ❌ | ❌ | ✅ | ✅ |
| `cloudSync` | ✅ | ❌ | ❌ | ✅ | ✅ |
| `exportFormats` | txt,md,pdf,json | **txt,md** | txt,md,pdf,json | txt,md,pdf,json | txt,md,pdf,json |
| `teamMembersLimit` | -1 | **1** | **1** | -1 | -1 |
| `sharedWorkspaces` | ✅ | ❌ | ❌ | ✅ | ✅ |
| `adminPanel` | ✅ | ❌ | ❌ | ✅ | ✅ |
| `auditLog` | full | **none** | **basic** | full | full |
| `selfHosted` | ❌ | ❌ | ❌ | ✅ | ✅ |
| `managedModelPool` | ✅ | ❌ | ❌ | ✅ | ✅ |
| `priorityModels` | ✅ | ❌ | ❌ | ✅ | ✅ |
| `kvarkCta` | subtle | subtle | subtle | **active** | **none** |
| `stripePriceId` | null | null | `env STRIPE_PRICE_PRO` | `env STRIPE_PRICE_TEAMS` | null |
`embeddingProviders` "all 6" = `['inprocess','mock','ollama','voyage','openai','litellm']`.
**Key UI gating consequences (read directly off the matrix):**
- **The upgrade trigger is Skills + Connectors + Team features**, not agents/memory. FREE blocks `customSkills`, caps `connectorLimit` at 5, caps `workspaceLimit` at 5, and gives `exportFormats` only `txt,md`. `spawnAgents` is `true` everywhere.
- **PRO is a solo power tier**: unlimited connectors/workspaces, `customSkills` on, full export — but `teamMembersLimit: 1`, no `sharedWorkspaces`, no `teamSkillLibrary`, no `cloudSync`, `auditLog: 'basic'`.
- **TEAMS unlocks collaboration + governance**: `sharedWorkspaces`, `teamSkillLibrary`, `cloudSync`, `adminPanel`, `auditLog: 'full'`, `selfHosted`, `managedModelPool`, `priorityModels`, and `kvarkCta: 'active'`.
- **TRIAL mirrors TEAMS/ENTERPRISE capabilities** (max unlock) but `selfHosted: false` and is time-limited to 15 days, after which the effective tier becomes FREE.
- **`stripePriceId` is resolved from env at module load** via `readEnv()` — only PRO and TEAMS carry one; TRIAL/FREE/ENTERPRISE are `null` (ENTERPRISE is consultative/contract-billed).
### 15.3 Tier ordering & helper functions (`tiers.ts`)
Ordering (`TIER_ORDER`, higher = more capable): `FREE:0, PRO:1, TEAMS:2, ENTERPRISE:3, TRIAL:3`. **TRIAL ties ENTERPRISE at rank 3** because TRIAL has max capabilities (but is time-limited).
| Export | Signature | What the frontend uses it for |
|---|---|---|
| `TIERS` | `readonly Tier[]` | Iterate tiers in pricing UI |
| `TRIAL_DURATION_DAYS` | `15` | Trial countdown |
| `parseTier(raw)` | `(string) => Tier \| null` | Normalize a tier string; maps **legacy names** `solo→FREE, basic→PRO, business→TEAMS, enterprise→ENTERPRISE, trial→TRIAL` |
| `isTrialExpired(trialStartedAt)` | `(string\|null) => boolean` | Gate trial UI (takes ISO date string; null ⇒ expired) |
| `getEffectiveTier(tier, trialStartedAt?)` | `(Tier, string?) => Tier` | **Downgrades TRIAL→FREE when expired** — call this before gating features |
| `trialDaysRemaining(trialStartedAt)` | `(string\|null) => number` | Trial banner countdown (0 if expired/none) |
| `tierSatisfies(actual, required)` | `(Tier, Tier) => boolean` | Boolean gate using `TIER_ORDER` |
| `assertTierCapability(actual, required)` | throws `TierError` | Backend enforcement; throws `TierError(required, actual)` |
| `getCapabilities(tier)` | `(Tier) => TierCapabilities` | Fetch the whole flag set |
| `hasCapability(tier, cap, min?)` | generic `=> boolean` | Single-flag gate; `-1` numeric caps short-circuit to `true`; with a `min` value, numeric caps pass if `cap === -1 \|\| cap >= min` |
| `TierError` | `class extends Error` | Carries `.required` and `.actual` tiers; `.name = 'TierError'` |
**Critical for the frontend:** always run the user's stored tier through `getEffectiveTier(tier, trialStartedAt)` **before** reading capabilities, so an expired trial correctly collapses to FREE gating.
---
## 16. Relationship diagram
```mermaid
erDiagram
User ||--o{ TeamMember : "is"
Team ||--o{ TeamMember : "has"
User ||--o| Team : "owns (ownerId)"
User ||--o{ AgentDef : "owns (userId)"
Team ||--o{ AgentDef : "scopes (teamId)"
User ||--o{ AgentGroup : "owns"
AgentGroup ||--o{ AgentGroupMember : "contains"
AgentDef ||--o{ AgentGroupMember : "member-of"
Team ||--o{ Task : "scopes"
User ||--o{ Task : "creates (createdBy)"
Task ||--o{ Task : "parentTaskId"
Team ||--o{ WaggleMessage : "bus"
User ||--o{ WaggleMessage : "sends (senderId)"
Team ||--o{ TeamEntity : "owns"
TeamEntity ||--o{ TeamRelation : "source/target"
Team ||--o{ TeamResource : "shares"
Team ||--o{ AgentJob : "scopes"
User ||--o{ AgentJob : "runs"
Team ||--o{ CronSchedule : "schedules"
User ||--o{ ScoutFinding : "for"
Team ||--o{ ScoutFinding : "for"
ProactivePattern ||--o{ SuggestionEntry : "fires"
User ||--o{ SuggestionEntry : "receives"
User ||--o{ AuditEntry : "acts"
Team ||--o{ AuditEntry : "scopes"
User {
string id
string clerkId
string email
string mindPath
}
Team {
string id
string slug
string ownerId
}
AgentDef {
string id
string model
string_array tools
}
Task {
string id
TaskStatus status
TaskPriority priority
string parentTaskId
}
WaggleMessage {
string id
MessageType type
MessageSubtype subtype
}
```
```mermaid
flowchart LR
raw["stored tier string (maybe legacy)"] --> parseTier
parseTier --> tier["Tier"]
tier --> getEffectiveTier
trial["trialStartedAt (ISO)"] --> getEffectiveTier
getEffectiveTier -->|TRIAL expired| FREE
getEffectiveTier --> eff["effective Tier"]
eff --> getCapabilities --> caps["TierCapabilities"]
caps --> gate["UI feature gate (hasCapability / flags)"]
```

296
docs/backend-map/sections/03a-api-chat-agents.md Normal file
View File

@@ -0,0 +1,296 @@
# 03a · Chat / Agent-Execution / Session API
## Purpose
This is the contract for the **conversational core** of Waggle OS: how the frontend submits a chat turn, how tokens and tool events stream back over Server-Sent Events (SSE), how tool-execution approvals are negotiated mid-stream, how sessions are created/listed/renamed/deleted/exported, and how slash commands are run. Every endpoint here lives in the **local Fastify sidecar** (`packages/server/src/local/routes/`), mounted at base path `/api`. If you are rebuilding the frontend, this file is your source of truth for these flows — the names, paths, and JSON shapes below are quoted verbatim from the code.
> **Two distinct execution paths.** `POST /api/chat` (conversational, multi-turn message history, `runAgentLoop`) is the one the chat UI uses. `POST /api/agent/run` (one-shot structured retrieval, shape-driven, `runRetrievalAgentLoop`) is a separate research path backing a Tauri `run_agent_query` command. Both stream SSE but with **different event names**. Do not conflate them.
---
## 1. The chat turn — `POST /api/chat` (SSE)
**File:** `packages/server/src/local/routes/chat.ts` (1708 LOC — the largest route in the codebase).
This is **not** a JSON request/response endpoint. The server validates the body, then calls `reply.hijack()` and writes a raw `text/event-stream`. **All validation and auth happen BEFORE the hijack** — once hijacked, `reply.status()` is a silent no-op, so any 400/403 you get back is a normal JSON error; anything after that is SSE.
### Request body
| Field | Type | Required | Notes |
|---|---|---|---|
| `message` | `string` | **yes** | The user's turn. Max length `WAGGLE_MAX_MESSAGE_LENGTH` env (default **50000** chars) → else `400 MESSAGE_TOO_LONG`. |
| `workspace` | `string` | no | Workspace ID. `workspaceId` is accepted as a synonym (P0-4 backwards-compat). Defaults to `'default'`. |
| `workspaceId` | `string` | no | Alias for `workspace`. |
| `model` | `string` | no | Model override. Falls back to workspace model → config default → `'claude-sonnet-4-6'`. |
| `session` | `string` | no | Session ID. Defaults to `workspace`, else `'default'`. Determines the `.jsonl` file written. |
| `workspacePath` | `string` | no | Explicit working dir. Path-traversal guarded: must resolve inside `dataDir`, else `400 PATH_TRAVERSAL`. |
| `persona` | `string` | no | Per-window persona override (takes precedence over workspace default for THIS request only). |
| `autonomy` | `{ level: 'normal' \| 'trusted' \| 'yolo', expiresAt?: number }` | no | Relaxes the tool-confirmation gate. Expired (`expiresAt < Date.now()`) falls back to `'normal'`. |
### Pre-stream rejections (regular JSON, HTTP error codes)
| Condition | Status | Body |
|---|---|---|
| Missing `message` | 400 | `{ error: 'message is required' }` |
| Message too long | 400 | `{ error: 'Message too long (...)', code: 'MESSAGE_TOO_LONG' }` |
| Injection score ≥ 0.7 | 400 | `{ error: 'Message blocked by security scanner', code: 'INJECTION_DETECTED' }` (flags NOT leaked) |
| Viewer in team workspace | 403 | `{ error: 'Viewers cannot send messages...', code: 'VIEWER_READ_ONLY' }` |
| `workspacePath` escapes dataDir | 400 | `{ error: 'Invalid workspace path', code: 'PATH_TRAVERSAL' }` |
| Unsafe `workspace`/`session` segment | 400 | thrown by `assertSafeSegment` → Fastify default `{ statusCode: 400 }` |
### SSE response headers
```
Content-Type: text/event-stream
Cache-Control: no-cache
Connection: keep-alive
X-Accel-Buffering: no
Access-Control-Allow-Origin: <validated origin>
```
Each event is written as `event: <name>\ndata: <json>\n\n`.
### SSE event catalogue (`/api/chat`)
These are the exact event names emitted via `sendEvent(event, data)`. The frontend MUST handle all of them.
| `event:` | `data` shape | Meaning |
|---|---|---|
| `token` | `{ content: string }` | One streamed chunk of the assistant's text. Concatenate in order. |
| `step` | `{ content: string }` | Human-readable progress line ("Recalling relevant memories...", "✔ tool approved", budget/compression notices, model-switch notes). |
| `tool` | `{ name: string, input: object }` | Agent is invoking a tool. Pair with the `step` line from `describeToolUse`. |
| `tool_result` | `{ name, result: string, duration?: number, isError: boolean }` | Tool finished. `isError` true when result starts with `Error:`/`Error `. |
| `file_created` | `{ filePath: string, fileAction: 'write' \| 'edit' \| 'generate' }` | Emitted after `write_file` / `edit_file` / `generate_docx` succeed. |
| `approval_required` | `{ requestId, toolName, input, sourceWorkspaceId, ...trustMeta }` | **Blocking.** Agent is paused; client must POST to `/api/approval/:requestId` (see §4). For `install_capability`, `trustMeta` adds `riskLevel`, `approvalClass`, `trustSource`, `assessmentMode`, `explanation`, `permissions`. |
| `gepa_choices` | `{ original, expanded, clarifyingQuestions: string[], intent }` | GEPA optimizer expanded a vague first message; offers ask-first clarification choices. |
| `model_switch` | `{ model, reason, primary }` | Active model changed mid-turn (budget cap, retry/fallback, credential exhaustion). |
| `notification` | `{ type: 'workflow_captured', title, message, pattern }` | Auto-skill capture suggested a repeatable workflow. |
| `done` | see below | **Terminal success.** Full final content + usage + cost. |
| `error` | `{ message: string }` | **Terminal failure.** User-friendly message only (raw traces/context never leaked). |
`done` data:
```jsonc
{
"content": "<final assistant text, may include appended disclaimers/notes>",
"usage": { "inputTokens": 0, "outputTokens": 0 }, // AgentResponse.usage
"toolsUsed": ["save_memory", "..."],
"model": "claude-sonnet-4-6",
"cost": 0.001234, // present only when usage known; rounded to 1e-6
"tokens": { "input": 0, "output": 0 } // present only when cost present
}
```
> Note: in echo mode and command-only paths, `done` carries `usage: { prompt_tokens, completion_tokens, total_tokens }` (all 0) and `toolsUsed: []` — a **different usage shape** than the agent-loop `done` (`inputTokens`/`outputTokens`). The frontend should tolerate both.
### What happens inside one chat turn (server-side, in order)
1. Validate body, scan for injection, RBAC + path guards (all pre-hijack).
2. `reply.hijack()`, write SSE headers, wire `AbortController` to client disconnect (`request.raw.on('close')`).
3. Resolve model with fallback chain: explicit → workspace → config default → `claude-sonnet-4-6`; apply budget-model and smart-routing (`routeMessage`) overrides.
4. Load/create session history (RAM cache `sessionHistories`, else `loadSessionMessages` from disk); push user message; `persistMessage` to `.jsonl`.
5. Create a per-session `Orchestrator` scoped to the workspace mind (`sessionManager.getOrCreate`), else fall back to the shared singleton.
6. Probe LiteLLM availability → choose **agent-loop**, **echo mode**, or **slash-command** path.
7. **Slash command?** → run via `commandRegistry.execute` (works even in echo mode). Result either streams as `token`s + `done`, or — if prefixed `AGENT_LOOP_REROUTE::` — falls through to the agent loop with a rewritten message.
8. Agent-loop path: auto-recall memory (`auto_recall` tool events), GEPA expand (first message only), ambiguity guard, build system prompt (persona + profile + skills + workspace-now + behavioral spec), filter tools by persona/availability, register the `pre:tool` confirmation hook, compress context, then call `runAgentLoop` with `stream: true` and `onToken`/`onToolUse`/`onToolResult` callbacks that emit the SSE events above.
9. Credential-pool key rotation + model fallback on retryable errors.
10. Post-processing: cost tracking, trace finalize, auto-save memory, skill distillation, KG entity extraction, correction detection, regulated-persona disclaimers, grounding hedge notes.
11. Push assistant message to history, `persistMessage`, emit `done`.
12. `finally`: unregister the `pre:tool` hook, finalize any pending trace as `abandoned`, `raw.end()`.
### `DELETE /api/chat/history`
Clears a session's in-RAM state. Querystring `?session=<id>` (default `'default'`). Evicts the session from `sessionHistories`, `systemPromptCache`, `compressionSummaries`, and `sessionToolSequences`. Returns `{ ok: true, cleared: <sessionId> }`. **Does not delete the on-disk `.jsonl`** — use `DELETE /api/sessions/:sessionId` for that.
---
## 2. Conversation history — `GET /api/history`
**File:** `agent.ts`. Loads a session's messages, RAM-first then disk (`dataDir/workspaces/<workspace>/sessions/<session>.jsonl`).
Querystring: `session?` (defaults to `workspace` then `'default'`), `workspace?` (default `'default'`).
Response:
```jsonc
{
"sessionId": "default",
"messages": [
{ "id": "hist-0", "role": "user", "content": "...", "timestamp": "ISO-8601" }
],
"count": 1
}
```
---
## 3. Agent status / cost / model — `agent.ts`
| Method | Path | Purpose | Response |
|---|---|---|---|
| GET | `/api/agent/status` | Agent + cost snapshot | `{ running: true, model, tokensUsed, estimatedCost, turns, usage }` |
| GET | `/api/agent/cost` | Detailed cost breakdown | `{ summary: <formatted string>, ...stats }` |
| POST | `/api/agent/cost/reset` | (No-op) cost reset | `{ ok: true, message: 'Cost tracking resets on server restart' }` |
| GET | `/api/agent/model` | Current model | `{ model }` |
| PUT | `/api/agent/model` | Switch model | body `{ model }` → `{ ok: true, model }`; missing model → `400 { error }` |
| GET | `/api/agents/active` | Sub-agent orchestrator state | `{ workers: [...], active: [...] }` (empty arrays when no workflow running) |
---
## 4. Tool-execution approvals — `approval.ts`
When the agent wants to run a gated tool, the chat SSE stream emits `approval_required` and **pauses** (awaiting a `Promise` registered in `pendingApprovals`). The frontend resolves it via a separate HTTP call. Auto-denies after **5 minutes** (fail-safe).
| Method | Path | Body / Params | Purpose |
|---|---|---|---|
| POST | `/api/approval/:requestId` | `{ approved: boolean, always?: boolean, reason?: string, sourceWorkspaceId?: string\|null }` | Approve/deny the paused tool. `always: true` persists a grant so future identical (tool + target) calls auto-pass. `404` if no such pending request. Returns `{ ok: true, requestId, approved, always }`. |
| GET | `/api/approval/pending` | — | List paused approvals (for reconnect/recovery): `{ pending: [{ requestId, toolName, input, timestamp }], count }`. |
| GET | `/api/approval/grants` | — | List persistent "always allow" grants: `{ grants, count }`. |
| DELETE | `/api/approval/grants/:id` | — | Revoke one grant. `404` if not found. |
| POST | `/api/approval/grants/clear` | — | Wipe all grants. `{ ok: true }`. |
> The `requestId` you POST back is the exact `requestId` from the `approval_required` SSE event. Echo `sourceWorkspaceId` back verbatim so the grant is scoped correctly.
---
## 5. Slash commands — `commands.ts` + `command-registry.ts`
| Method | Path | Body | Purpose |
|---|---|---|---|
| POST | `/api/commands/execute` | `{ command: string, workspaceId?: string }` | Run a slash command out-of-band (not through chat SSE). Missing `command` → `400`. Returns `{ result: string, command: string }`. |
- A command is any input matching `/^\/\w/` (`commandRegistry.isCommand`). Parsed as `/<name> <args>`; aliases supported.
- This route wires a **subset** of `CommandContext`: `searchMemory`, `getWorkspaceState`, `listSkills`. It **omits** `runWorkflow` and `spawnAgent` (those need the full agent loop), so workflow commands like `/research`, `/plan`, `/spawn` return their "not available in this context" fallback. To run those, send the command **through `POST /api/chat`** instead — chat detects the slash command and, when the LLM is available, can reroute it through the agent loop via the `AGENT_LOOP_REROUTE::` prefix.
- There is **no** `GET /api/commands` listing endpoint. The registry has `list()`/`search()` in code but they are not exposed over HTTP, so the frontend cannot fetch the command catalogue from the server — it must hardcode/derive autocomplete client-side.
---
## 6. Sessions CRUD — `sessions.ts` + `session-utils.ts`
Sessions are **JSONL files on disk**: `dataDir/workspaces/<workspaceId>/sessions/<sessionId>.jsonl`. Line 0 is a `{ type: "meta", title, summary?, created, distilled?, outcome? }` record; subsequent lines are `{ role, content, timestamp }`. All path segments pass `assertSafeSegment`.
| Method | Path | Params / Query / Body | Purpose & response |
|---|---|---|---|
| GET | `/api/workspaces/:workspaceId/sessions` | query `?hideEmpty=true` | List sessions (sorted by `lastActive` desc). Returns `SessionInfo[]`. Unknown workspace → `[]`. |
| GET | `/api/workspaces/:workspaceId/sessions/search` | query `?q=<≥2 chars>&limit=<≤50>` | Full-text search across session content/summary. `<2` chars → `400`. Unknown ws → `404`. Returns `SessionSearchResult[]`. |
| GET | `/api/workspaces/:workspaceId/sessions/:sessionId/export` | — | Export one session as Markdown (`Content-Type: text/markdown`). `404` if missing. |
| GET | `/api/workspaces/:workspaceId/sessions/:sessionId/timeline` | — | Tool-event timeline (`TimelineEvent[]`, heuristically reconstructed from assistant content patterns; `spawn_agent` nests children). `404` if missing. |
| POST | `/api/workspaces/:workspaceId/sessions` | body `{ title? }` | Create a session (writes meta line). `404` if workspace missing. `201` + `SessionInfo`. |
| PATCH | `/api/sessions/:sessionId` | body `{ title }`, query `?workspace=` | Rename. Searches all workspaces if `workspace` omitted. Missing title → `400`; not found → `404`. Returns `{ id, title }`. |
| DELETE | `/api/sessions/:sessionId` | query `?workspace=` | Delete the `.jsonl`. Searches all workspaces if omitted. `404` if not found. Returns `{ deleted: true }`. |
| GET | `/api/sessions/:sessionId/summary` | query `?workspace=` | Structured post-session summary (counts user/assistant/tool/memory/doc). `404` if not found. |
### `SessionInfo` (list/create shape)
```ts
{ id: string; title: string; summary: string | null; messageCount: number; lastActive: string; created: string }
```
### `/api/sessions/:sessionId/summary` shape
```jsonc
{
"sessionId", "title", "messageCount",
"userMessages", "assistantMessages",
"toolsUsed", "memoriesSaved", "documentsCreated",
"summary", "lastActive", "created"
}
```
### Other types from `session-utils.ts` the frontend may render
- `SessionSearchResult`: `{ sessionId, title, summary, matchCount, snippets: [{ text, role }], lastActive }`
- `TimelineEvent`: `{ id, timestamp, toolName, status: 'success'|'error', durationMs, inputPreview, outputPreview, fullInput, fullOutput, children? }`
- `ThreadInfo`: `{ title, lastActive, freshness: 'fresh'|'aging'|'stale', messageCount, sessionId }` (fresh `<2d`, aging `<7d`, else stale — timestamp-based, NOT importance)
- `ProgressItem` / `OpenQuestion` / `SessionOutcome` / `DistillableSession` — heuristic extractions used by `/catchup`-style features (not directly exposed as REST here).
> Summaries are **lazily generated** (`generateSessionSummary`) for sessions with ≥4 messages and persisted back into the meta line — no LLM needed.
---
## 7. Agent groups — `agent-groups.ts`
Multi-agent group configs stored in `dataDir/agent-groups.json`. A group has a `strategy` (`parallel` | `sequential` | `coordinator`) and ordered `members`.
| Method | Path | Body | Purpose |
|---|---|---|---|
| GET | `/api/agent-groups` | — | List all groups (`AgentGroup[]`). |
| POST | `/api/agent-groups` | `{ name, description?, strategy, members: AgentGroupMember[] }` | Create. Missing name → `400`. `201` + group. |
| PATCH | `/api/agent-groups/:id` | partial of the above | Update fields. `404` if not found. |
| DELETE | `/api/agent-groups/:id` | — | Delete. `404` if not found. Returns `{ deleted: true }`. |
| POST | `/api/agent-groups/:id/run` | `{ task, teamId? }` | **Placeholder** — does NOT execute. Returns a queued stub: `{ jobId, groupId, groupName, strategy, memberCount, task, status: 'queued' }`. Missing task → `400`. |
`AgentGroup`: `{ id, name, description?, strategy, members, createdAt }`.
`AgentGroupMember`: `{ agentId, roleInGroup: 'lead'|'worker'|string, executionOrder }`.
---
## 8. One-shot structured retrieval — `POST /api/agent/run` (SSE)
**File:** `agent-run.ts`. Backs the Tauri `run_agent_query` command. Shape-aware research flow via `runRetrievalAgentLoop` — **distinct from `/api/chat`**. Requires `multiMind.personal` + an embedding provider, else `503`.
### Request body (`AgentRunBody`)
| Field | Type | Notes |
|---|---|---|
| `question` | `string` | **required**; non-string → `400 { error: 'question is required' }`. |
| `shape` | `string?` | Prompt-shape name. Validated against `listShapes()`; unknown → warn + fall back to model-default (NOT a 400). |
| `model` | `string?` | Default `'claude-sonnet-4-6'`. |
| `persona` | `string?` | Default `'general-purpose'`. |
| `workspace` / `workspaceId` | `string?` | Target mind; `'personal'` or absent → personal mind. |
| `maxSteps` | `number?` | Default `5`. |
| `maxRetrievalsPerStep` | `number?` | Default `8`. |
### SSE events (`/api/agent/run`) — **different from `/api/chat`**
| `event:` | `data` |
|---|---|
| `started` | `{ shape, shapeRequested, shapeRecognized, model }` |
| `progress` | per-step `AgentRunProgressEvent` (from the loop's `onProgress`) |
| `finalized` | `{ rawResponse, normalizedResponse, promptShapeName, stepsTaken, retrievalCalls, loopExhausted, totalTokensIn, totalTokensOut, totalCostUsd, totalLatencyMs }` |
| `error` | `{ error: string }` |
| `done` | `{ ok: true }` (always last) |
---
## 9. Persistence & context model (mental model)
- **Session file** = JSONL, append-only. Meta line first, then `{ role, content, timestamp }` per message. Written by `persistMessage`, read by `loadSessionMessages`.
- **History cache** = `server.agentState.sessionHistories: Map<sessionId, {role,content}[]>` — RAM mirror, lazily hydrated from disk.
- **Context window** = `MAX_CONTEXT_MESSAGES = 50`. When a budget model exists, chat uses intelligent `compressConversation` (LLM-summarize the middle); otherwise the simple `applyContextWindow` sliding window with a prepended `[Context summary — N earlier messages compressed]` system message.
- **Governance** (`chat-governance.ts`): for team workspaces, blocked-tool policies are fetched directly from the team server (5-min cache), no HTTP loopback.
---
## 10. Flow diagram — submitting a chat turn with a gated tool
```mermaid
sequenceDiagram
participant UI as Frontend
participant Chat as POST /api/chat (SSE)
participant Loop as runAgentLoop
participant Appr as POST /api/approval/:id
participant Disk as session .jsonl
UI->>Chat: { message, workspace, session, model?, persona?, autonomy? }
Note over Chat: validate · injection scan · RBAC · path guard (pre-hijack)
alt rejected
Chat-->>UI: 400/403 JSON error
else accepted
Chat->>Disk: persist user message
Chat-->>UI: event: step "Recalling relevant memories..."
Chat-->>UI: event: tool / tool_result (auto_recall)
Chat->>Loop: stream=true, onToken/onToolUse/onToolResult
Loop-->>UI: event: token (xN, assistant text)
Loop-->>UI: event: tool { name, input }
Note over Loop: gated tool hits pre:tool hook
Loop-->>UI: event: approval_required { requestId, toolName, input }
UI->>Appr: { approved: true, always? }
Appr-->>Loop: resolve(true) (or auto-deny after 5 min)
Loop-->>UI: event: tool_result { name, result, isError }
Loop-->>UI: event: token (xN, more text)
Chat->>Disk: persist assistant message
Chat-->>UI: event: done { content, usage, toolsUsed, model, cost? }
end
Note over Chat: on failure → event: error { message } (+ raw turn still persisted)
```

430
docs/backend-map/sections/03b-api-memory.md Normal file
View File

@@ -0,0 +1,430 @@
# 03b — Memory / Knowledge / Wiki / Harvest / Import / Identity API
## Purpose
This section is the frontend rebuild contract for Waggle OS's **memory subsystem** HTTP API: recalling and saving memory frames, browsing the knowledge graph, compiling and reading the personal wiki, harvesting external AI exports (ChatGPT / Claude / Gemini / Claude Code) into memory, importing conversation exports, reading/writing the agent identity record, tracking document versions, and the GDPR "right to erasure" flow. Every endpoint below is grounded in actual route files under `packages/server/src/local/routes/`. All routes are served by the **local sidecar** (Fastify), registered in `packages/server/src/local/index.ts`.
> **Scope note.** The recall/save/search/CRUD endpoints under `/api/memory/frames` and `/api/memory/search` live in `routes/memory.ts` (a sibling of the assigned files). They are the "recall/save memory" surface the rebuild needs, so they are documented here. The knowledge-graph read (`/api/memory/graph`) is in `routes/knowledge.ts`. Where a route lives in a specific file, the file is named in the per-section heading.
---
## Mental model
```mermaid
flowchart TD
UI[Frontend / Lovable rebuild]
subgraph Memory["Memory frames (routes/memory.ts)"]
SEARCH["/api/memory/search"]
FRAMES["/api/memory/frames (GET/POST/PUT/DELETE/PATCH)"]
STATS["/api/memory/stats"]
end
subgraph KG["Knowledge graph (routes/knowledge.ts)"]
GRAPH["/api/memory/graph"]
end
subgraph Wiki["Wiki compiler (routes/wiki.ts)"]
WPAGES["/api/wiki/pages*"]
WCOMPILE["/api/wiki/compile"]
WEXPORT["/api/wiki/export/{obsidian,notion}"]
end
subgraph Harvest["Harvest (routes/harvest.ts)"]
HPREV["/api/harvest/preview"]
HCOMMIT["/api/harvest/commit"]
HSRC["/api/harvest/sources"]
HRUNS["/api/harvest/runs*"]
HPROG["/api/harvest/progress (SSE)"]
end
subgraph Import["Legacy import (routes/import.ts)"]
IPREV["/api/import/preview"]
ICOMMIT["/api/import/commit"]
end
subgraph Identity["Identity (routes/identity.ts + mind.ts)"]
IDGET["/api/identity (GET/POST)"]
MIND["/api/mind/{identity,awareness,skills}"]
end
DOCS["/api/workspaces/:id/documents (routes/documents.ts)"]
ERASE["/api/data/erase (routes/data-erase.ts)"]
UI --> Memory & KG & Wiki & Harvest & Import & Identity & DOCS & ERASE
PERSONAL[(personal.mind\nSQLite)]
WS[(workspace minds\nSQLite)]
Memory --> PERSONAL & WS
KG --> PERSONAL & WS
Wiki --> PERSONAL
Harvest --> PERSONAL
Import --> PERSONAL
Identity --> PERSONAL & WS
```
**Two memory scopes everywhere.** Almost every read/write accepts an optional workspace selector. The canonical query/body param is **`workspace`**, but **`workspaceId` is also accepted** as an alias (do not rely on only one). Omitting it (or passing `personal`) targets the **personal mind**. Workspace minds are separate SQLite DBs resolved via `server.agentState.getWorkspaceMindDb(id)`.
**Frame field naming.** SQLite stores `snake_case` (`created_at`, `frame_type`, `gop_id`, `access_count`). The memory routes run results through `normalizeFrame()` which emits a **camelCase UI shape** (see below). The knowledge-graph and wiki routes return **raw DB rows** (snake_case), not normalized.
---
## 1. Memory frames — recall & save (`routes/memory.ts`)
Base entity is a **memory frame**. The UI-normalized frame shape returned by search/list/edit endpoints:
| Field (UI) | Type | Source DB column | Notes |
|---|---|---|---|
| `id` | number | `id` | |
| `content` | string | `content` | |
| `source` | string | `source` | provenance: `user_stated` \| `tool_verified` \| `agent_inferred` \| `import` \| `system` |
| `source_mind` | string | (computed) | `personal` \| `workspace` — which mind it came from |
| `mind` | string | (computed) | legacy alias of `source_mind` |
| `frameType` | string | `frame_type` | `'I'` (independent) or `'P'` (predicate); defaults `'I'` |
| `importance` | string | `importance` | `critical` \| `important` \| `normal` \| `temporary` \| `deprecated` |
| `timestamp` | string (ISO) | `created_at` | |
| `score` | number? | `score` | present on search results |
| `gop` | string | `gop_id` | session/group id |
| `accessCount` | number | `access_count` | |
| `authorId` / `authorName` | string? | `author_id` / `author_name` | only on team-synced frames |
| `workspaceName` | string? | `_workspace_name` | only on global cross-workspace search |
### Endpoints
| Method | Path | Purpose |
|---|---|---|
| GET | `/api/memory/search` | Full-text search frames across personal + workspace minds |
| GET | `/api/memory/frames` | List recent frames (no query) for Memory tab initial load |
| POST | `/api/memory/frames` | Direct memory **write** (save a frame); optional entity extraction |
| PUT | `/api/memory/frames/:id` | Edit a frame's content and/or importance |
| PATCH | `/api/memory/frames/:id/access` | Atomically increment `access_count` |
| DELETE | `/api/memory/frames/:id` | Delete a frame by id |
| GET | `/api/memory/stats` | Frame/entity/relation counts (personal + optional workspace + totals) |
**`GET /api/memory/search`** — query params: `q` (required; 400 if missing), `scope` (`personal` \| `workspace` \| `all` \| `global`, default `all`), `limit` (default 20), `workspace`/`workspaceId`, `since`/`until` (ISO date strings for temporal filtering). Response: `{ results: Frame[], count: number }`. `scope=global` searches personal + every workspace mind and adds `workspaceName` to each result.
**`GET /api/memory/frames`** — params: `workspace`/`workspaceId`, `limit` (default 50), `since`/`until`. Returns `{ results: Frame[], count: number }` sorted newest-first.
**`POST /api/memory/frames`** — body `{ content (required), workspace?, importance?, source? }`, query `?extract=true|false` (default true; runs entity extraction + co-occurrence relations after save). `content` is XSS-sanitized server-side. Validates `importance` (default `normal`) and `source` (default `import`; 400 on invalid). Dedupes identical content. Response on success: `{ saved: true, frameId, mind, importance, source, extraction?: { entitiesExtracted, relationsCreated } }`. On duplicate: `{ saved: false, duplicate: true, frameId, mind, message }`. Emits a `memory_write` audit event.
**`PUT /api/memory/frames/:id`** — body `{ content (required), importance? }`, query `workspace?`. 400 on missing content / invalid importance, 404 if frame not found. Returns the normalized frame plus `updated: true`.
**`PATCH /api/memory/frames/:id/access`** — query `workspace?`. Returns `{ frameId, accessed: true, accessCount, mind }`; 404 if not found.
**`DELETE /api/memory/frames/:id`** — query `workspace?`. Returns `{ deleted: true, frameId }`; 404 if not found. Emits `memory_delete` audit event.
**`GET /api/memory/stats`** — query `workspace?`. Returns:
```json
{
"personal": { "frameCount": 0, "entityCount": 0, "relationCount": 0 },
"workspace": { "frameCount": 0, "entityCount": 0, "relationCount": 0 } /* or null */,
"total": { "frameCount": 0, "entityCount": 0, "relationCount": 0 }
}
```
---
## 2. Knowledge graph read (`routes/knowledge.ts`)
| Method | Path | Purpose |
|---|---|---|
| GET | `/api/memory/graph` | Read entities + relations from a mind (or merged across all minds) |
**Query params:** `workspace`/`workspaceId`, `scope` (`all` \| `personal` \| `current`).
- `scope=personal` (also the default when no workspace + no scope): returns the personal mind's graph.
- `scope=all`: merges the **personal mind plus every workspace mind**, tagging each row with a `_source` field (`'personal'` or the workspace name) and re-offsetting ids/relation endpoints to avoid collisions.
- With a `workspace` id (and not `scope=all`): returns that workspace's graph; **404** `{ error: 'Workspace not found' }` if the mind can't be resolved. The workspace id is validated via `assertSafeSegment`.
**Response shape:** `{ entities: KGRow[], relations: KGRow[] }`. Rows are **raw DB rows** from the live (non-expired) graph:
```sql
SELECT * FROM knowledge_entities WHERE valid_to IS NULL ORDER BY name
SELECT * FROM knowledge_relations WHERE valid_to IS NULL ORDER BY id
```
Known `KGRow` fields (other columns pass through untyped): entities have `id`, `name`, `type`; relations have `id`, `source_id`, `target_id`, `type`. With `scope=all`, each row also carries `_source`.
> **No write/CRUD endpoints for entities/relations exist in these route files.** Entity/relation creation happens implicitly: (a) inside `POST /api/memory/frames?extract=true` (entity upsert + `co_occurs_with` relations), and (b) during the harvest **cognify** step (see §4). For direct graph mutation the rebuild would use the MCP tools (`save_entity`, `create_relation`) not an HTTP route. `KnowledgeGraph` is imported but no `POST /api/memory/graph` is registered.
---
## 3. Wiki compiler (`routes/wiki.ts`)
All wiki routes operate on the **personal mind only** (`server.multiMind.personal`). Pages are persisted in the `wiki_pages` SQLite table; metadata/state is managed by `CompilationState` from `@waggle/wiki-compiler`.
| Method | Path | Purpose |
|---|---|---|
| GET | `/api/wiki/pages` | List all compiled page metadata |
| GET | `/api/wiki/pages/:slug` | Get a single page's metadata (404 if missing) |
| GET | `/api/wiki/pages/:slug/content` | Get full markdown content of a page |
| POST | `/api/wiki/compile` | Trigger wiki compilation (incremental or full) |
| GET | `/api/wiki/health` | Compilation health report (gaps / data quality) |
| GET | `/api/wiki/watermark` | Current compilation watermark/state |
| POST | `/api/wiki/export/obsidian` | Write all pages to a local dir (Obsidian-vault layout) |
| POST | `/api/wiki/export/notion` | Push pages as child pages under a Notion root |
**`GET /api/wiki/pages`** → array of page metadata objects (`state.getAllPages()`).
**`GET /api/wiki/pages/:slug`** → single metadata object, or 404 `{ error: 'Page not found' }`.
**`GET /api/wiki/pages/:slug/content`** → `{ slug, markdown }`, or 404. Reads `SELECT markdown FROM wiki_pages WHERE slug = ?`.
**`POST /api/wiki/compile`** — body `{ mode?: 'incremental' | 'full', concepts?: string[] }` (default `incremental`). Response merges the compiler result with `{ llmProvider, llmModel }`.
**Critical degraded state:** if there is **no real embedding provider** (`server.embeddingProvider` absent or active provider is `'mock'`), returns **503** `{ error, skippedReason: 'no_real_embedder' }`. The UI must surface an "add a Voyage/OpenAI key (or Ollama embedding model)" prompt rather than treating this as a normal failure. The same 503 guard applies to `GET /api/wiki/health`.
**`GET /api/wiki/watermark`** → current `CompilationState` watermark.
**`POST /api/wiki/export/obsidian`** — body `{ outDir }`. Validation: `outDir` required (400) and **must be an absolute path** (400). 409 if there are no compiled pages yet ("Run compile first"). 500 on write failure. Output layout: `{outDir}/{pageType}/{slug}.md` plus a top-level `_index.md`.
**`POST /api/wiki/export/notion`** — body `{ rootPageUrl }` (a notion.so URL or raw page id). 400 if missing or unparseable. Requires a Vault secret named **`notion-wiki-token`** — **503** with setup instructions if absent. 409 if no compiled pages. Returns the Notion writer stats. Delta-tracking (Notion page ids per slug, content hashes) is delegated to `CompilationState`.
---
## 4. Harvest — external AI export ingestion (`routes/harvest.ts`)
Harvest ingests AI tool exports into the **personal mind**, then runs **cognify** (entity/relation extraction) and an incremental **wiki recompile**. Supported `source` values (`ImportSourceType`): `chatgpt`, `claude`, `claude-desktop`, `claude-code`, `gemini`, `google-ai-studio`, and anything else falls through to a `UniversalAdapter`.
| Method | Path | Purpose |
|---|---|---|
| POST | `/api/harvest/preview` | Parse an export and show what would be imported (no save) |
| POST | `/api/harvest/commit` | Run full pipeline: save frames → cognify → wiki recompile |
| GET | `/api/harvest/sources` | List registered harvest sources |
| POST | `/api/harvest/sources` | Register or update a source |
| DELETE | `/api/harvest/sources/:source` | Remove a registered source |
| PATCH | `/api/harvest/sources/:source` | Toggle auto-sync / interval for a source |
| GET | `/api/harvest/progress` | **SSE** stream of import progress events |
| GET | `/api/harvest/runs` | List recent harvest runs (debug/history) |
| GET | `/api/harvest/runs/latest-interrupted` | Latest resumable run (drives "Resume?" banner) |
| POST | `/api/harvest/runs/:id/abandon` | Discard an interrupted run + its cached input |
| POST | `/api/harvest/extract-identity` | LLM-extract identity facts from recent harvest frames |
| POST | `/api/harvest/scan-claude-code` | Scan local `~/.claude` dir for Claude Code history |
**`POST /api/harvest/preview`** — body `{ data, source }` (400 if either missing). Response:
```json
{
"source": "claude",
"itemCount": 42,
"types": { "conversation": 40, "decision": 2 },
"preview": [ { "id": "...", "title": "...", "type": "...", "source": "..." } ] /* first 10 */
}
```
**`POST /api/harvest/commit`** — body `{ data?, source?, resumeFromRun? }`. Two entry modes:
- **Fresh run:** `data` + `source` required (400 otherwise). To trigger a **local filesystem scan** (Claude Code), send `data: { scanLocal: true }` with `source: 'claude-code'` — the adapter scans `~/.claude`. 400 if the source has no filesystem adapter or no default dir.
- **Resume:** `resumeFromRun: <runId>` replays a prior interrupted run's cached input. 404 if run not found; 409 if already `completed`/`abandoned`; 410 if its cached input is gone/corrupt.
Behavior: caps each item's content at **10,000 chars**, preserves the original `item.timestamp` (strict ISO-8601 validation; falls back to ingest wall-clock with a logged warning otherwise), skips work if the content hash matches the last sync (`skipped: true`), then runs cognify and an incremental wiki compile (both fail-soft and both skip with a reason if there's no real embedder). Success response:
```json
{
"source": "claude",
"itemCount": 42,
"saved": 42,
"cognified": 42,
"cognifySkippedReason": null, /* or "no_real_embedder" */
"entitiesExtracted": 18,
"relationsCreated": 7,
"wikiCompiled": { "pagesCreated": 3, "pagesUpdated": 1, "pagesUnchanged": 12 },
"wikiSkippedReason": null, /* or "no_real_embedder" */
"runId": 17,
"message": "Imported 42 items from claude, cognified 42 frames, wiki 4 pages updated"
}
```
Other commit responses: nothing found → `{ saved: 0, message }`; unchanged since last sync → `{ saved: 0, skipped: true, message }`; personal mind unavailable → **503**.
**`GET /api/harvest/sources`** → `{ sources: HarvestSource[] }`. A source row carries (from `HarvestSourceStore`) `source`, display name, `sourcePath`, sync tracking incl. `lastContentHash`, and auto-sync config.
**`POST /api/harvest/sources`** — body `{ source, displayName (both required → 400), sourcePath?, autoSync?, syncIntervalHours? }`. → `{ source: HarvestSource }`.
**`DELETE /api/harvest/sources/:source`** → `{ ok: true }`.
**`PATCH /api/harvest/sources/:source`** — body `{ autoSync?, syncIntervalHours? }` → `{ source: HarvestSource }`.
**`GET /api/harvest/progress`** — Server-Sent Events. Each event is `data: <json>` where json is `{ phase, current, total, source }`. Phases observed: `saving`, `cognifying`, `wiki-compile`. The UI subscribes on mount.
**`GET /api/harvest/runs`** — query `limit` (default 50, max 500) → `{ runs: HarvestRun[] }`.
**`GET /api/harvest/runs/latest-interrupted`** → `{ run: HarvestRun | null }` (null if no resumable run or its cache file no longer exists). Run states: `running`, `failed`, `completed`, `abandoned`.
**`POST /api/harvest/runs/:id/abandon`** — 400 invalid id, 404 not found; returns the (now abandoned) run.
**`POST /api/harvest/extract-identity`** — scans the most recent 50 `harvest` frames, sandboxes them, and calls the internal Haiku proxy (`claude-haiku-4-5`) to extract `{ name, role, company, industry, bio }` suggestions. Server-side gate: only fields with `confidence >= 0.5` survive (`MIN_SUGGESTION_CONFIDENCE`). Persists to `profile.identitySuggestions`. Response: `{ suggestions: IdentitySuggestion[] }` (with `note: 'no_anthropic_key'` if no Anthropic key in Vault). 503 if personal mind unavailable. `IdentitySuggestion = { field, value, confidence, sourceHint, extractedAt }`.
**`POST /api/harvest/scan-claude-code`** — scans `~/.claude`. Response `{ found, path, itemCount, types?, preview? }` (`preview` = first 20 items with `id`, `title`, `type`, `metadata`).
---
## 5. Legacy import (`routes/import.ts`)
A simpler, older import path for **ChatGPT and Claude only** (`ImportSource = 'chatgpt' | 'claude'`). Distinct from harvest: no run-store, no cognify, no wiki recompile, no SSE.
| Method | Path | Purpose |
|---|---|---|
| POST | `/api/import/preview` | Parse a ChatGPT/Claude export, show extraction (no save) |
| POST | `/api/import/commit` | Parse + extract + save knowledge items to personal memory |
Both take body `{ data, source }`. 400 if missing, or if `source` is not exactly `"chatgpt"` or `"claude"`. Both return the `ImportResult` shape:
```json
{
"source": "chatgpt",
"conversationsFound": 10,
"conversationsParsed": 10,
"knowledgeExtracted": [
{ "type": "decision|fact|preference|topic",
"content": "...",
"source": "chatgpt|claude",
"conversationTitle": "...",
"importance": "important|normal" }
],
"errors": []
}
```
`/api/import/commit` additionally writes each extracted item as an `import`-source frame and adds `{ saved: <n>, message }`. If nothing was extracted: `{ ...result, saved: 0, message: 'No knowledge items found to import' }`. 503 if personal mind unavailable; 500 on save failure.
---
## 6. Identity (`routes/identity.ts`) + Mind context (`routes/mind.ts`)
### Identity record (`routes/identity.ts`)
Single-row-per-mind identity table (id = 1), so create + update collapse into upsert. Defaults to the personal mind; `?workspace=<id>` selects a workspace-scoped identity (each mind owns its own identity row).
| Method | Path | Purpose |
|---|---|---|
| GET | `/api/identity` | Read the identity record (placeholder shape if unconfigured) |
| POST | `/api/identity` | Create or update the identity record (upsert) |
**`GET /api/identity?workspace=<id>`** always returns 200 with the same `IdentityResponseShape`:
```json
{
"configured": true,
"name": null, "role": null, "department": null,
"personality": null, "capabilities": null, "system_prompt": null,
"created_at": null, "updated_at": null,
"_note": "identity not configured for this mind" /* present only on placeholder */
}
```
When unconfigured/uninitialized, `configured: false` and all fields `null` with a `_note` explaining why. The UI sees **one consistent shape** regardless of state.
**`POST /api/identity`** — body `{ workspace?, name?, role?, department?, personality?, capabilities?, system_prompt? }` (all optional, default to empty string). Returns the saved `IdentityResponseShape` (`configured: true` + populated `created_at`/`updated_at`). 503 if multi-mind not initialized; 500 on write failure.
### Mind context (`routes/mind.ts`)
Read-only context the CLI also accesses directly via the Orchestrator.
| Method | Path | Purpose |
|---|---|---|
| GET | `/api/mind/identity` | Agent identity **context string** → `{ identity }` |
| GET | `/api/mind/awareness` | Current awareness state context → `{ awareness }` |
| GET | `/api/mind/skills` | Loaded skills list → `{ skills: [{ name, length }], count }` |
> Note: `/api/mind/identity` returns the orchestrator's rendered identity **context** (`identity.toContext()`), not the structured record — that structured record is `/api/identity`.
---
## 7. Document version registry (`routes/documents.ts`)
Tracks document **versions** per workspace. Stored as JSON on disk at `~/.waggle/workspaces/{id}/documents.json` — **not** in SQLite. Part of Wave 7. All path segments validated via `assertSafeSegment`.
| Method | Path | Purpose |
|---|---|---|
| GET | `/api/workspaces/:id/documents` | List tracked documents (name + version count + latest) |
| POST | `/api/workspaces/:id/documents` | Register a new document version (auto-increments version) |
| GET | `/api/workspaces/:id/documents/:name/versions` | List all versions of one document |
**`GET .../documents`** → `{ documents: [{ name, versionCount, latestVersion: DocumentVersion | null }] }`.
**`POST .../documents`** — body `{ name (required), path (required), sizeBytes? }`. 400 if name/path missing. Auto-assigns `version` = lastVersion+1 (starts at 1). Returns **201** `{ document: <name>, version: DocumentVersion }`.
**`GET .../:name/versions`** → `{ name, versions: DocumentVersion[] }`; 404 if document not found.
`DocumentVersion = { version: number, path: string, createdAt: string (ISO), sizeBytes: number }`.
---
## 8. GDPR data erasure (`routes/data-erase.ts`)
Right-to-erasure (GDPR Art. 17). **The route does NOT delete anything** — it validates, snapshots, and writes a marker file. The destructive wipe runs at next service startup, before any DB is opened.
| Method | Path | Purpose |
|---|---|---|
| POST | `/api/data/erase` | Schedule full erasure of this install's data dir at next startup |
**Double confirmation gate (both required):**
1. HTTP header **`X-Confirm-Erase: yes`** (exact, case-sensitive).
2. Body field **`{ "confirmation": "I UNDERSTAND THIS IS PERMANENT" }`** (exact phrase).
Failure responses:
- Missing/wrong confirmation → **400** `{ error: 'ERASE_NOT_CONFIRMED', message, requirements: { header, bodyField } }` (the `requirements` object literally tells the UI the exact header value and body phrase needed).
- Data dir doesn't look like a Waggle dir → **400** `{ error: 'ERASE_REFUSED_UNSAFE_PATH', message }`.
- Marker write fails → **500** `{ error: 'ERASE_MARKER_WRITE_FAILED', message }`.
Success → **200**:
```json
{
"requestedAt": "2026-06-06T...Z",
"markerPath": "...",
"dataDirSnapshot": { "fileCount": 0, "totalBytes": 0, "topLevelEntries": [] },
"instruction": "Quit Waggle and relaunch — erasure completes during startup. ..."
}
```
Emits a `data_erase_requested` audit event **before** writing the marker (audit survives even if the marker write fails).
---
## Frontend rebuild cheat-sheet
- **Workspace param:** send `workspace=<id>` (alias `workspaceId` also works). Omit / `personal` → personal mind.
- **Frame shape differs by route:** memory routes return **camelCase normalized** frames; knowledge-graph + wiki routes return **raw snake_case** rows.
- **No real embedder = 503 with `skippedReason: 'no_real_embedder'`** on `/api/wiki/compile` and `/api/wiki/health`, and `*SkippedReason` flags inside the harvest commit response. Build UI affordances to prompt for an embedding key.
- **Harvest vs Import:** harvest (`/api/harvest/*`) is the rich, resumable, multi-source pipeline with SSE + cognify + wiki recompile; import (`/api/import/*`) is the legacy ChatGPT/Claude-only path. Prefer harvest.
- **Two identity surfaces:** `/api/identity` = structured editable record (GET/POST upsert, always one shape); `/api/mind/identity` = read-only rendered context string. Don't confuse them.
- **Erasure needs both** the `X-Confirm-Erase: yes` header **and** the exact body phrase — and the 400 response hands you the exact requirements.
- **Personal-mind-not-available → 503** is a common guard across harvest/import/identity write routes; handle it as a transient backend-not-ready state.
```mermaid
sequenceDiagram
participant UI
participant API as Sidecar
participant DB as personal.mind
UI->>API: POST /api/harvest/preview {data,source}
API-->>UI: {itemCount, types, preview}
UI->>API: GET /api/harvest/progress (SSE open)
UI->>API: POST /api/harvest/commit {data,source}
API->>DB: save frames (cap 10k chars)
API-->>UI: SSE {phase:"saving",...}
API->>DB: cognify (entities+relations)
API-->>UI: SSE {phase:"cognifying",...}
API->>DB: incremental wiki compile
API-->>UI: SSE {phase:"wiki-compile",...}
API-->>UI: {saved, cognified, wikiCompiled, runId}
```
---
## File Ingestion (`ingest.ts`) — `POST /api/ingest`
> Added to close audit gap #1. This is the endpoint the UI's file-upload / drag-drop surface calls to turn user files into LLM-ready text and (optionally) memory frames. Distinct from `/api/harvest/*` (conversation exports) and `/api/import` (memory import).
| Method | Path | Request body | Response | Streaming? |
|---|---|---|---|---|
| `POST` | `/api/ingest` | `{ files: { name: string, content: string }[], workspaceId?: string }` — `content` is **base64**. Route `bodyLimit` = **15 MB** (allows base64 overhead) | `{ files: IngestFileResult[] }` | No |
**`IngestFileResult`** = `{ name: string; type: FileType; summary: string; content?: string }`
where `type` ∈ `image | document | spreadsheet | csv | text | archive | unsupported`.
- For **images**, `content` is a `data:<mime>;base64,...` data URI (passed straight to a vision model).
- For everything else, `content` is the **extracted plain text** (omitted for `unsupported` / failed extraction).
**Validation & error codes** (per-file, fail-fast):
| Condition | Code |
|---|---|
| `files` missing / not an array / empty | `400 { error: "files array is required" }` |
| file entry missing `name` or non-string `content` | `400 { error: "Invalid file entry: <name>" }` |
| `content` not well-formed base64 (`/^[A-Za-z0-9+/]*={0,2}$/`) | `400 { error: "Invalid base64 content for file: <name>" }` |
| decoded size > **10 MB** (`MAX_FILE_SIZE`, est. `content.length * 0.75`) | `413 { error: "File <name> exceeds 10 MB limit" }` |
| `workspaceId` fails `assertSafeSegment` (path-traversal guard) | throws (4xx) before any fs touch |
**Supported extraction** (extension → handler):
- **Images** png/jpg/jpeg/gif/webp/svg/bmp/ico/tiff → data URI.
- **Documents** pdf (`pdf-parse`), docx (`mammoth`), pptx (`adm-zip`, reads `ppt/slides/slideN.xml`). Missing optional dep or scanned/encrypted file → graceful `summary` with no `content`.
- **Spreadsheets** xlsx/xls (`exceljs`, each sheet → CSV-ish text).
- **csv** (RFC-4180 line parse, reports columns/rows).
- **Text/code** ~50 extensions (md/txt/json/yaml/ts/js/py/rs/go/sql/dockerfile/...).
- **Archives** zip → lists up to 50 entry names (no extraction).
- Unknown extension → `type: "unsupported"`, skipped from side effects.
**Side effects** (only when `workspaceId` is set and ≠ `"default"`, both non-blocking — ingest still 200s if they fail):
1. Appends each non-unsupported result to the workspace **file registry** `workspaces/<id>/files.jsonl` (`{ name, type, summary, sizeBytes, ingestedAt }` — this is what `GET /api/files` reads, see 04-feature-map).
2. Saves a **memory frame** per file via `orchestrator.autoSaveFromExchange("User uploaded file: <name>", "File ingested: ... + 500-char preview")` so uploads persist across sessions.

461
docs/backend-map/sections/03c-api-workspace-team.md Normal file
View File

@@ -0,0 +1,461 @@
# 03c · Workspace, Team, Persona, Settings, Profile & Pins API
## Purpose
This section is the contract for the management-plane of Waggle OS: how the frontend creates and inspects **workspaces** (and their pre-configured **templates**), manages **teams** and members, lists/creates **personas**, reads/writes **settings** (models, budgets, autonomy, tiers), maintains the **user profile** (identity, writing-style, brand), and pins/favorites chat messages. Every endpoint below is served by the **local Fastify sidecar** (the Node process bundled into the Tauri app, default loopback port `3333`); the frontend talks to it over HTTP with **no auth header** (loopback-trust model). All persistence is local — SQLite (`teams.db`, per-workspace `*.mind`) or JSON files under the data dir (`~/.waggle` by default).
> Source files read for this section:
> `packages/server/src/local/routes/{workspaces.ts, workspace-templates.ts, team.ts, personas.ts, settings.ts, profile.ts, pins.ts}` and the prompt-builder helpers `packages/server/src/local/{workspace-sessions.ts, workspace-state.ts}` plus `packages/server/src/local/routes/workspace-context.ts`.
>
> **Important framing for the rebuild:** `workspace-context.ts` and `workspace-state.ts` are **NOT HTTP routes**. They are internal prompt-builder modules (`buildWorkspaceNowBlock`, `buildWorkspaceState`, `formatWorkspaceNowPrompt`) consumed by `chat.ts`, `commands.ts`, and `workspaces.ts` to assemble the agent's system prompt. Their output reaches the frontend only as a sub-object on `GET /api/workspaces/:id/context` (the `workspaceState` field) — see [Workspace "Now"/state model](#the-workspace-now--state-model-internal). The frontend never calls them directly.
---
## 1. Endpoint Reference (every route)
All paths are relative to the sidecar base URL (e.g. `http://127.0.0.1:3333`). Responses are JSON unless noted. Tier gating uses the `requireTier(...)` preHandler from `middleware/assert-tier.ts`.
### 1.1 Workspaces — lifecycle (`workspaces.ts`)
| Method | Path | Purpose |
|---|---|---|
| GET | `/api/workspaces` | List all workspaces. Optional `?group=` and `?teamId=` filters. Returns a bare array. |
| POST | `/api/workspaces` | Create a workspace (enforces tier `workspaceLimit`). Returns the created workspace (201). |
| GET | `/api/workspaces/:id` | Get one workspace by id (404 if missing). |
| GET | `/api/workspaces/:id/context` | **The "Workspace Now" catch-up block** — summary, recent threads, suggested prompts, stats, greeting, structured state. Used when opening a workspace. |
| GET | `/api/workspaces/:id/files` | List ingested/registered files for the workspace (newest first). |
| PUT | `/api/workspaces/:id` | Update workspace (full-ish). `personaId: null` is ignored on PUT (kept). Validates `model`. |
| PATCH | `/api/workspaces/:id` | Partial update. `personaId: null` **clears** the persona on PATCH. |
| DELETE | `/api/workspaces/:id` | Delete workspace + its mind DB (204). |
| GET | `/api/workspaces/:id/export` | Export workspace. `?format=briefing` → markdown; default → JSON dump (memories, pins, sessions). |
| GET | `/api/workspaces/:id/cost` | Per-workspace spend vs budget, status (`ok`/`warning`/`exceeded`), 7-day history. |
| GET | `/api/workspaces/:id/storage` | Virtual/linked storage stats for the workspace. |
| GET | `/api/workspaces/:id/storage/files` | List files in workspace storage (optional `?dir=`). |
| GET | `/api/workspaces/:id/storage/read` | Read a file. `?path=` required; `?raw=true` returns raw bytes, else a JSON wrapper. |
| POST | `/api/workspaces/:id/storage/write` | Write a file. `?path=` required, body `{ content }`. Returns 201. |
| DELETE | `/api/workspaces/:id/storage/delete` | Delete a file. `?path=` required. Returns 204. |
> **Persona switching** is done through these workspace update routes: set `personaId` on the workspace (PUT/PATCH). There is no dedicated `/api/personas/active` endpoint. A per-request persona override is also accepted by the chat route (`persona` field in the chat body, out of scope here). The workspace's `personaId` is the persistent default.
### 1.2 Workspace Templates (`workspace-templates.ts`)
| Method | Path | Purpose |
|---|---|---|
| GET | `/api/workspace-templates` | List all templates: 15 built-in + user-created. Returns `{ templates, count }`. |
| POST | `/api/workspace-templates` | Create a custom template (validated). Returns the template (with generated `id`). |
| PUT | `/api/workspace-templates/:id` | Update a custom template. 403 if `id` is built-in, 404 if not found. |
| DELETE | `/api/workspace-templates/:id` | Delete a custom template. 403 if built-in, 404 if not found. Returns `{ ok: true }`. |
| POST | `/api/workspace-templates/generate` | AI-generate a template config from a prompt (needs Anthropic key in `settings.json`). |
### 1.3 Team (`team.ts`)
Two sub-families: **team-server connection** (proxy to a remote Teams server) and **local team CRUD** (works solo with a local user id, stored in `teams.db`).
| Method | Path | Purpose |
|---|---|---|
| POST | `/api/team/connect` | Connect to a remote team server (validate token via its `/health`). **Tier-gated: `TEAMS`.** Stores config; never echoes token. |
| POST | `/api/team/disconnect` | Clear stored team-server config. Returns `{ disconnected: true }`. |
| GET | `/api/team/status` | Current connection status `{ connected, serverUrl?, userId?, displayName? }`. |
| GET | `/api/team/teams` | List teams from the remote server (proxied). 401 if not connected. |
| GET | `/api/team/members` | List members from remote server, or local fallback `[{ id:'local', name:'You', status:'online' }]`. |
| GET | `/api/team/presence` | Presence (`?workspaceId=`). Remote proxy, else self-as-online fallback. Emits `presence_update` on the event bus. |
| GET | `/api/team/activity` | Recent activity (`?workspaceId=&limit=`, max 50). Maps remote `memory_frame` entities to activity items. Empty if disconnected. |
| GET | `/api/team/messages` | Recent WaggleDance messages (`?workspaceId=&limit=`, max 50). Emits a `message` notification if any. |
| GET | `/api/team/governance/permissions` | Effective capability permissions (`?workspaceId=`). **Tier-gated: `ENTERPRISE`.** 5-min in-memory cache; returns stale on fetch failure. |
| GET | `/api/team/memory/search` | Search team memory frames (`?q=&limit=`, max 50). 400 if not connected or `q` missing. Client-side keyword filter. |
| POST | `/api/teams` | Create a local team. Body `{ name, description? }`. Auto-adds creator as `owner`. Returns 201 with members. |
| GET | `/api/teams` | List teams the local user belongs to `{ teams: [...] }`. |
| GET | `/api/teams/:id` | Team detail + members + linked workspaces. 404 if missing. |
| PUT | `/api/teams/:id` | Update team name/description. Requires `owner` or `admin`. |
| DELETE | `/api/teams/:id` | Delete team (owner only). Unlinks workspaces. Returns 204. |
| POST | `/api/teams/:id/members` | Add/invite member. Body `{ userId?, email?, displayName?, role? }`. Requires owner/admin. 409 if already member. |
| PUT | `/api/teams/:id/members/:userId` | Change member role. **Owner only.** |
| PATCH | `/api/teams/:id/members/:userId` | Change member role (alias for PUT). **Owner or admin.** |
| DELETE | `/api/teams/:id/members/:userId` | Remove member. Owner/admin can remove anyone; a member can remove self. Cannot remove the owner. |
| GET | `/api/teams/:id/activity` | Aggregated audit events across the team's workspaces (`?limit=` max 200, `?from=` ISO; defaults to last 7 days). |
> Note: a **second, different** `teamRoutes` exists at `packages/server/src/routes/teams.ts` registered by the *non-local* server `index.ts`. This section documents the **local** sidecar version (`local/routes/team.ts`), which is what the desktop frontend hits.
### 1.4 Personas (`personas.ts`)
| Method | Path | Purpose |
|---|---|---|
| GET | `/api/personas` | List persona catalog (no system prompts). Returns `{ personas: [...] }`. |
| POST | `/api/personas` | Create a custom persona. **Tier-gated: `PRO`.** Requires `name` + `systemPrompt`. 409 if id collides with built-in. Returns 201. |
| PATCH | `/api/personas/:id` | Update a custom persona (merge). 403 for built-ins, 404 if custom not found. |
| POST | `/api/personas/generate` | AI-generate a persona from a prompt. **Tier-gated: `PRO`.** 503 if LLM unavailable. |
| DELETE | `/api/personas/:id` | Delete a custom persona. 403 for built-ins, 404 if not found. Returns `{ deleted: true, id }`. |
### 1.5 Settings, Tier, Budget, Cloud Sync, Admin (`settings.ts`)
| Method | Path | Purpose |
|---|---|---|
| GET | `/api/settings` | Read config: models, budgets, providers (API keys **masked**), `mindPath`, `dataDir`, `litellmUrl`, `onboardingCompleted`. |
| PUT | `/api/settings` | Update models/budgets/providers. Secrets go to the encrypted vault; non-secrets to `config.json`. |
| PATCH | `/api/settings` | Partial merge for non-provider settings (e.g. `onboardingCompleted`). |
| POST | `/api/settings/test-key` | Validate an API key's **format** (no network call). Body `{ provider, apiKey }`. |
| GET | `/api/settings/permissions` | Read `{ defaultAutonomy, externalGates, workspaceOverrides }`. |
| PUT | `/api/settings/permissions` | Save permission settings. Accepts new `defaultAutonomy` enum or legacy `yoloMode` boolean. |
| GET | `/api/tier` | **Authoritative tier source** for the frontend (effective tier, trial days, capabilities, usage, legacy `limits`). |
| PATCH | `/api/tier` | Dev/test tier override. **403 unless `WAGGLE_ALLOW_TIER_OVERRIDE=1`** (fail-closed). |
| POST | `/api/tier/start-trial` | Atomically start the 15-day TRIAL. **409 if already started** (idempotent — one trial per install). |
| GET | `/api/cloud-sync` | Cloud-sync availability/enabled/connected status. |
| POST | `/api/cloud-sync/toggle` | Enable/disable cloud sync. **Tier-gated: `TEAMS`.** Body `{ enabled }`. |
| GET | `/api/admin/overview` | Admin dashboard data (usage, workspaces, plugins). **Tier-gated: `TEAMS`.** |
| GET | `/api/admin/audit-export` | Export audit log. **Tier-gated: `TEAMS`.** `?format=json|csv`, `?from=&to=` ISO. |
### 1.6 User Profile (`profile.ts`)
| Method | Path | Purpose |
|---|---|---|
| GET | `/api/profile` | Full profile (identity, writingStyle, brand, interests, meta). |
| PUT | `/api/profile` | Partial-merge update. Also mirrors identity into personal memory frames. |
| POST | `/api/profile/analyze-style` | LLM-analyze a writing sample (min 50 chars) → fills `writingStyle`. |
| POST | `/api/profile/analyze-brand` | LLM-extract brand colors/fonts from a description → fills `brand`. |
| GET | `/api/profile/style` | Writing-style summary (for agent injection). |
| GET | `/api/profile/brand` | Brand profile (for document-generation tools). |
| POST | `/api/profile/research` | LLM-research the user/company → writes a bio. 400 if no name/company set. |
### 1.7 Pins (`pins.ts`)
| Method | Path | Purpose |
|---|---|---|
| GET | `/api/workspaces/:id/pins` | List pinned messages for a workspace `{ pins }`. |
| POST | `/api/workspaces/:id/pins` | Add a pin. Body `{ messageContent, messageRole, label? }`. Returns 201 `{ pin }`. |
| PATCH | `/api/workspaces/:id/pins/:pinId` | Update a pin's `status` (`draft`/`final`) or `label`. 404 if not found. |
| DELETE | `/api/workspaces/:id/pins/:pinId` | Remove a pin. 404 if not found. Returns `{ ok: true }`. |
---
## 2. Data Shapes (real field names & types)
### 2.1 Workspace object (returned by GET/POST/PUT `/api/workspaces`)
The workspace object is produced by `server.workspaceManager`. Observed/used fields:
| Field | Type | Notes |
|---|---|---|
| `id` | string | Workspace id. Used as a path segment (validated by `assertSafeSegment`). |
| `name` | string | Required on create. |
| `group` | string | Required on create. Filterable via `?group=`. |
| `icon` | string? | |
| `model` | string? | Validated against `/^[a-zA-Z0-9][\w./-]*$/`. |
| `personaId` | string? | The workspace's default persona. `PATCH ... { personaId: null }` clears it. |
| `agentGroupId` | string? | |
| `directory` | string? | Linked local filesystem dir (if any). |
| `tone` | `'professional'\|'casual'\|'technical'\|'legal'\|'marketing'`? | |
| `templateId` | string? | Set from `template`/`templateId` on create. |
| `storageType` | `'virtual'\|'local'\|'team'`? | |
| `storagePath` / `storageConfig` | string? / object? | For local/team storage. |
| `teamId` / `team` | string? | Link to a team (`team` is a legacy alias; both are checked). |
| `teamServerUrl` / `teamRole` / `teamUserId` | string? / role? / string? | Team-server linkage. |
| `budget` | number? | Per-workspace dollar budget (used by `/cost`). |
| `created` | string (ISO) | |
**POST `/api/workspaces` request body** accepts all of: `name`(req), `group`(req), `icon`, `model`, `personaId`, `directory`, `template`, `templateId`, `tone`, `storageType`, `storagePath`, `storageConfig`, `teamId`, `teamServerUrl`, `teamRole`, `teamUserId`.
**Create side effects (frontend should expect these to "just happen"):** auto-create file dir structure; auto-install starter skills (first time); seed `template.starterMemory` as visible memory frames (unless `blank`); install a template capability-pack (best-effort); register on team server if `teamId`+`teamServerUrl`; emit `workspace_create` audit event + telemetry.
### 2.2 `GET /api/workspaces/:id/context` response (the catch-up block)
```jsonc
{
"workspace": { "id", "name", "group", "model", "directory", "templateId", "personaId" },
"summary": "string", // narrative; falls back to a generic line for empty ws
"recentThreads": [ { "id", "title", "lastActive" } ], // top 5 by mtime
"recentDecisions": [ { "content", "date" } ], // up to 5
"suggestedPrompts": ["string"], // contextual; onboarding prompts if brand-new
"recentMemories": [ { "content", "importance", "date" } ],
"progressItems": [ { "type", "content", ... } ], // up to 10
"stats": { "memoryCount", "sessionCount", "fileCount" },
"lastActive": "ISO",
"greeting": "string", // time-aware + inactivity-aware
"pendingTasks": ["string"], // up to 5 (tasks + blockers)
"upcomingSchedules": ["string"], // next 3 cron schedules, "name at <time>"
"welcomeMessage": "string|undefined", // template-specific first-time copy
"teamContext": { "isTeam", "teamId", "tasks": [...] } | undefined,
"workspaceState": { /* WorkspaceState — see §4 */ } | null,
"crossWorkspaceHints": undefined // intentionally disabled (privacy); always absent
}
```
### 2.3 `WorkspaceTemplate` (`workspace-templates.ts`)
```ts
type TemplateCategory = 'sales'|'research'|'engineering'|'marketing'|'operations'|'legal'|'design'|'custom';
interface WorkspaceTemplate {
id: string;
name: string;
description: string;
persona: string; // persona id, e.g. 'sales-rep'
connectors: string[]; // e.g. ['github','email','slack']
suggestedCommands: string[];// e.g. ['/research','/draft']
starterMemory: string[]; // seeded into the new workspace's mind
builtIn: boolean; // true = cannot edit/delete
category?: TemplateCategory;
}
```
**The 15 built-in template ids** (frontend can rely on these): `sales-pipeline`, `research-project`, `code-review`, `marketing-campaign`, `product-launch`, `legal-review`, `agency-consulting`, `customer-support`, `finance-accounting`, `hr-people`, `operations-center`, `data-analytics`, `recruiting-pipeline`, `design-studio`, `blank`.
POST/PUT validation requires: `name`, `description`, `persona` (all non-empty strings) and `connectors`, `suggestedCommands`, `starterMemory` (all arrays). Custom ids are generated as `custom-<timestamp>-<rand>`.
`POST /api/workspace-templates/generate` body: `{ prompt, availableConnectors, availableCommands, availablePersonas }` → returns `{ name, description, persona, connectors, suggestedCommands, starterMemory }`. Requires an Anthropic `apiKey` in `settings.json` (400 if absent), uses model `claude-sonnet-4-6` by default.
### 2.4 Team & Member (local CRUD — `teams.db`)
`teams.db` schema (created on first use):
```sql
teams(id TEXT PK, name, description DEFAULT '', owner_id, created, updated)
team_members(team_id, user_id, role CHECK(role IN ('owner','admin','member','viewer')) DEFAULT 'member',
display_name DEFAULT '', email, joined,
PRIMARY KEY(team_id,user_id), FOREIGN KEY(team_id)->teams(id) ON DELETE CASCADE)
```
Normalized API shapes (camelCase):
| Team field | From row | | Member field | From row |
|---|---|---|---|---|
| `id` | `id` | | `userId` | `user_id` |
| `name` | `name` | | `role` | `role` (`owner`/`admin`/`member`/`viewer`) |
| `description` | `description` | | `displayName` | `display_name` |
| `ownerId` | `owner_id` | | `email` | `email` (nullable) |
| `created` | `created` | | `joined` | `joined` |
| `updated` | `updated` | | | |
`GET /api/teams/:id` also returns `members: Member[]` and `workspaces: Workspace[]` (those whose `teamId`/`team` matches).
**Role enforcement matrix (local CRUD):**
- Update team → owner or admin
- Delete team → owner only
- Add member → owner or admin
- Change role → owner only (`PUT`) / owner or admin (`PATCH`)
- Remove member → owner/admin (anyone) or self; **never the owner**
### 2.5 Team-server connection config (`TeamServerConfig`, in `@waggle/core`)
```ts
interface TeamServerConfig { url: string; token?: string; userId?: string; displayName?: string; }
```
Stored via `WaggleConfig.setTeamServer/clearTeamServer/getTeamServer`. Tokens are **never** returned to the client (`/connect` echoes `token: '***'`).
### 2.6 Persona (catalog item from `GET /api/personas`)
System prompts and `failurePatterns` are **intentionally omitted** from GET (large/sensitive). The catalog item is:
| Field | Type | Notes |
|---|---|---|
| `id` | string | |
| `name` | string | |
| `description` | string | |
| `icon` | string | emoji |
| `workspaceAffinity` | string[] | which workspace types it fits |
| `suggestedCommands` | string[] | |
| `tagline` | string? | powers PersonaSwitcher hover card |
| `bestFor` | string[]? | hover card |
| `wontDo` | string? | hover card hard-boundary |
| `isReadOnly` | boolean? | true = no write tools ever |
Full `AgentPersona` (used by POST/PATCH) adds: `systemPrompt` (required on create), `modelPreference` (default `claude-sonnet-4-6`), `tools: string[]`, `defaultWorkflow`. Generated personas (`/generate`) return `{ name, description, icon, systemPrompt, tools }`.
### 2.7 Settings response (`GET /api/settings`)
```jsonc
{
"defaultModel": "string",
"fallbackModel": "string|null",
"budgetModel": "string|null",
"budgetThreshold": 0.8, // 0..1 fraction
"providers": { // keyed by provider name
"<name>": { "apiKey": "sk-xxx...yyyy" /* MASKED */, "models": ["..."], "baseUrl": "..." }
},
"mindPath": "string",
"dataDir": "string",
"litellmUrl": "string",
"dailyBudget": 0, // dollars; null = no limit
"budgetHardCap": false, // true = block, false = warn
"onboardingCompleted": false
}
```
`PUT /api/settings` accepts `{ defaultModel?, providers?, dailyBudget?, budgetHardCap?, fallbackModel?, budgetModel?, budgetThreshold? }`. API keys in `providers` are written to the **encrypted vault**; non-secret fields also mirrored to `config.json`. Keys are always masked on the way out via `maskApiKey` (`first7 + '...' + last4`).
### 2.8 Permissions (`GET/PUT /api/settings/permissions`)
```ts
type AutonomyLevel = 'normal' | 'trusted' | 'yolo';
interface PermissionsData {
defaultAutonomy: AutonomyLevel; // normal=gate writes; trusted=auto writes; yolo=auto all but blacklist
externalGates: string[];
workspaceOverrides: Record<string, string[]>;
}
```
Legacy `yoloMode: boolean` is accepted on PUT and migrated (`true`→`yolo`, else `normal`). Invalid `defaultAutonomy` → 400.
### 2.9 Tier response (`GET /api/tier`) — authoritative
```jsonc
{
"tier": "FREE|PRO|TEAMS|ENTERPRISE|TRIAL", // EFFECTIVE tier (trial expiry applied)
"rawTier": "TRIAL", // stored tier before expiry resolution
"trialStartedAt": "ISO|null",
"trialDaysRemaining": 0,
"trialExpired": false, // rawTier==='TRIAL' && effective==='FREE'
"capabilities": { /* TierCapabilities from @waggle/shared */ },
"teamsServerUrl": "http://127.0.0.1:3101|null",
"teamsServerAvailable": false,
"usage": { "workspaceCount": 0 },
"limits": { // LEGACY shape — kept for compat
"maxWorkspaces": -1, "maxSessions": 3, "maxMembers": -1,
"features": { "teams", "marketplace", "budgetControls", "kvark", "governance", "customModels" }
}
}
```
Tier defaults to `FREE` if `config.json` has none. `parseTier` auto-migrates legacy lowercase names. Capabilities (`workspaceLimit`, `teamMembersLimit`, `sharedWorkspaces`, `adminPanel`, `cloudSync`, etc.) come from `@waggle/shared`.
### 2.10 UserProfile (`profile.json`)
```ts
interface UserProfile {
// Identity
name; role; company; industry; bio; avatarUrl: string;
identitySuggestions: IdentitySuggestion[]; // harvest-extracted, awaiting review
// Writing Style
writingStyle: { tone; sentenceLength; vocabulary; structure: string; samples: string[]; analyzed: boolean };
// Brand & Visual Identity
brand: {
companyName; primaryColor; secondaryColor; accentColor; fontHeading; fontBody; logoDescription: string;
styles: {
docx: { margins; headerStyle; notes };
pptx: { layout; colorScheme; notes };
pdf: { coverPage; reportStyle; notes };
xlsx: { headerFormat; chartColors; notes };
};
analyzed: boolean;
};
// Prefs & Meta
interests: string[]; communicationStyle; language; timezone: string;
questionnaireCompleted: boolean; createdAt; updatedAt: string;
}
interface IdentitySuggestion {
field: 'name'|'role'|'company'|'industry'|'bio';
value: string; confidence: number /*0..1*/; sourceHint: string; extractedAt: string;
}
```
Defaults seed `brand.primaryColor='#D4A84B'`, fonts `Inter`, `language='en'`, `timezone` from the runtime, `writingStyle.tone='professional'`. PUT does a deep merge (nested `writingStyle` / `brand.styles` merged, not replaced). Analyze/research endpoints call the local Anthropic proxy (`/v1/chat/completions`) with `claude-haiku-4-5`; they 503 if no `anthropic` key in the vault.
### 2.11 PinnedItem (`pins.json`)
```ts
interface PinnedItem {
id: string; // crypto.randomUUID()
workspaceId: string;
messageContent: string;
messageRole: 'assistant' | 'user';
pinnedAt: string; // ISO
label?: string;
status?: 'draft' | 'final'; // W7.4
}
```
Stored at `~/.waggle/workspaces/{id}/pins.json` (note: pins use `os.homedir()/.waggle`, not the configurable `dataDir`).
---
## 3. Persistence map (where each thing lives)
| Subsystem | Storage | Path |
|---|---|---|
| Workspaces (list/meta) | `server.workspaceManager` (JSON) | under data dir |
| Per-workspace memory | SQLite `MindDB` | `getMindPath(id)` (`*.mind`) |
| Workspace sessions | JSONL files | `{dataDir}/workspaces/{id}/sessions/*.jsonl` |
| User templates | JSON | `{dataDir}/workspace-templates.json` |
| Teams + members | SQLite | `{dataDir}/teams.db` |
| Team-server connection | `config.json` (`teamServer`) | via `WaggleConfig` |
| Custom personas | JSON (via `@waggle/agent`) | data dir |
| Settings / models / budgets / tier | `config.json` | data dir |
| API keys (secrets) | encrypted **vault** | `server.vault` |
| Permissions | JSON | `{dataDir}/permissions.json` |
| Profile | JSON | `{dataDir}/profile.json` |
| Pins | JSON | `~/.waggle/workspaces/{id}/pins.json` |
---
## 4. The Workspace "Now" / state model (internal)
These are prompt builders, surfaced to the frontend only inside `GET /api/workspaces/:id/context` → `workspaceState`.
`buildWorkspaceState()` (`workspace-state.ts`) assembles a `WorkspaceState` from three sources (memory frames, session JSONL, awareness layer), each item classified by **freshness** (computed from timestamps, not item type):
```ts
type Freshness = 'fresh'|'aging'|'stale'; // <2d / <7d / >=7d
type StateSource = 'memory'|'session'|'awareness';
interface StateItem { content; freshness; source; sourceId?; dateLastTouched: string; }
interface WorkspaceState {
active; openQuestions; pending; blocked; completed; stale; recentDecisions: StateItem[];
nextActions: string[]; // priority cascade: blockers → questions → pending → stale → fallback
}
```
`buildWorkspaceNowBlock()` (`workspace-context.ts`) wraps that into a `WorkspaceNowBlock` (used by chat/commands for system-prompt injection) and adds a `greeting` via `buildTimeAwareGreeting()`:
1. **Fresh-state** (`frameCount===0`) → "Welcome — anything you discuss here will be remembered."
2. **Inactivity** (>24h) → "You've been away N days. Here's what happened:"
3. **Time-of-day** fallback (morning/afternoon/evening/late-night).
`WorkspaceSessionManager` (`workspace-sessions.ts`) is the in-memory runtime registry — `Map<workspaceId, WorkspaceSession>` with a per-session orchestrator, mind DB, tools, abort controller, persona id, and token counter. Concurrency cap defaults to **3** (tier-raised: Solo 3 / Basic 5 / Teams 10 / Enterprise unbounded). It is purely server-side; the frontend never touches it directly but its cap is why "max concurrent sessions reached" errors can surface from the chat path.
---
## 5. Cross-cutting rules the frontend must know
- **No auth header.** Loopback-trust. The dangerous `PATCH /api/tier` is fail-closed behind `WAGGLE_ALLOW_TIER_OVERRIDE=1` precisely because anything local could otherwise self-upgrade.
- **`/api/tier` is the single source of truth** for gating UI. Use its `capabilities` (and the legacy `limits`) — don't infer tier locally.
- **Tier-gated endpoints return the middleware's rejection** when below tier: `team/connect` & `cloud-sync/toggle` & `admin/*` need `TEAMS`; custom/generated personas need `PRO`; governance permissions need `ENTERPRISE`.
- **API keys are always masked** on read. To set a key, PUT it inside `providers`; it lands in the vault.
- **Built-ins are immutable.** Built-in templates (15) and built-in personas reject edit/delete with 403.
- **Persona switching = workspace update.** PUT/PATCH `personaId` on the workspace. `PATCH ...{personaId:null}` clears; `PUT ...{personaId:null}` is ignored.
- **`crossWorkspaceHints` is permanently empty** in `/context` (disabled for privacy) — don't build UI expecting it.
- **Team endpoints are dual-mode.** `/api/teams*` (local CRUD, always works) vs `/api/team/*` (remote-server proxy, returns local fallbacks when disconnected). Don't conflate the two prefixes.
---
## 6. Diagram — request → storage routing
```mermaid
flowchart TD
FE["Frontend (apps/web)"] -->|HTTP loopback :3333| SC["Fastify sidecar (local/index.ts)"]
SC --> WS["/api/workspaces*<br/>workspaces.ts"]
SC --> TPL["/api/workspace-templates*<br/>workspace-templates.ts"]
SC --> TEAM["/api/team* + /api/teams*<br/>team.ts"]
SC --> PER["/api/personas*<br/>personas.ts"]
SC --> SET["/api/settings* /api/tier* /api/cloud-sync* /api/admin*<br/>settings.ts"]
SC --> PROF["/api/profile*<br/>profile.ts"]
SC --> PIN["/api/workspaces/:id/pins*<br/>pins.ts"]
WS --> WSM["workspaceManager (JSON)"]
WS --> MIND["MindDB per workspace (*.mind)"]
WS --> CTX["buildWorkspaceState / NowBlock<br/>(workspace-state.ts + workspace-context.ts)"]
CTX --> MIND
CTX --> SESSJSONL["sessions/*.jsonl"]
TPL --> TPLJSON["workspace-templates.json"]
TPL -->|generate| ANTH["Anthropic SDK"]
TEAM -->|local CRUD| TDB["teams.db (SQLite)"]
TEAM -->|connect/proxy| RMT["Remote Teams server<br/>(TeamServerConfig in config.json)"]
TEAM -->|TEAMS / ENTERPRISE| GATE["requireTier middleware"]
PER --> PERJSON["custom personas (disk)"]
PER -->|PRO + generate| GATE
SET --> CFG["config.json"]
SET --> VAULT["encrypted vault (API keys)"]
SET --> PERMJSON["permissions.json"]
PROF --> PROFJSON["profile.json"]
PROF -->|analyze/research| PROXY["local Anthropic proxy /v1/chat/completions"]
PIN --> PINJSON["~/.waggle/workspaces/:id/pins.json"]
```

531
docs/backend-map/sections/03d-api-marketplace-skills.md Normal file
View File

@@ -0,0 +1,531 @@
# 03d · API — Marketplace, Skills, Connectors, Tools, OAuth, Vault & Providers
## Purpose
This subsystem is the **capability layer** of the Waggle OS sidecar: how the frontend
browses/installs marketplace packages, manages local **Skills** and **Plugins**, connects to
third-party **Connectors**, launches external **AI tools** (AI-OS), runs **OAuth** flows, stores
secrets in the **Vault**, and reads the canonical **LLM/search provider** catalog. Every route
here is served by the local Fastify sidecar (loopback-bound, base path `/api/...`). This document
is the contract: exact paths, request/response shapes, and the real identifiers from the code.
All routes live in `packages/server/src/local/routes/` in the files named per section. Names,
field keys, and paths below are quoted verbatim from source — do not rename them.
---
## 1. Marketplace (`marketplace.ts` — production `/api/marketplace/*`)
The marketplace is backed by `MarketplaceDB` (from `@waggle/marketplace`), decorated onto Fastify
as `fastify.marketplace`. If that decoration is missing, **every route returns `503`** with
`{ error: 'Marketplace not available', hint: 'marketplace.db was not found or failed to load' }`.
The frontend must handle 503 as "marketplace disabled" everywhere.
### 1.1 Endpoints
| Method | Path | Tier gate | Purpose |
|---|---|---|---|
| GET | `/api/marketplace/search` | — | FTS5 + faceted catalog search; annotates each pkg with `installed` + scan status |
| GET | `/api/marketplace/packs` | — | List all capability packs |
| GET | `/api/marketplace/packs/:slug` | — | Pack detail + its packages (`404` if slug unknown) |
| GET | `/api/marketplace/enterprise-packs` | **ENTERPRISE** | KVARK-gated enterprise packs (empty unless KVARK configured) |
| POST | `/api/marketplace/install` | **PRO** | Install a package; runs `SecurityGate` pre-scan with severity gating |
| POST | `/api/marketplace/uninstall` | — | Uninstall an installed package |
| GET | `/api/marketplace/installed` | — | List installed packages |
| POST | `/api/marketplace/security-check` | — | Scan a package by ID **without** installing |
| GET | `/api/marketplace/sources` | — | List marketplace sources with package counts |
| POST | `/api/marketplace/sources` | — | Add a user source + immediate sync (`201`) |
| DELETE | `/api/marketplace/sources/:id` | — | Remove a **user-added** source (`403` for built-in) |
| GET | `/api/marketplace/categories` | — | Category taxonomy (`PACKAGE_CATEGORIES`) |
| POST | `/api/marketplace/sync` | — | Manual sync from configured sources |
| GET | `/api/marketplace/security-status` | — | Cisco scanner availability + aggregate scan counts |
| POST | `/api/marketplace/publish` | **PRO** | Publish a local skill from `~/.waggle/skills/` to the catalog (`201`) |
### 1.2 `GET /api/marketplace/search`
**Query params** (all optional, all strings): `query`, `type`, `category`, `pack`, `source`,
`sort`, `limit` (default `20`), `offset` (default `0`).
`sort` must be one of `relevance | popular | recent | name` (invalid → `undefined`).
**Response**: spreads the `db.search()` result and overrides `packages` + adds `categories`:
```jsonc
{
"total": 123,
"packages": [
{
// ...all MarketplacePackage fields (id, name, display_name, description,
// author, package_type, waggle_install_type, version, category,
// downloads, stars, rating, rating_count, platforms, dependencies, packs ...)
"installed": true, // db.isInstalled(pkg.id)
"scanStatus": "passed", // 'passed' | 'failed' | 'not_scanned' | 'unavailable'
"scanScore": 80 // number, omitted when null/negative
}
],
"categories": [ /* PACKAGE_CATEGORIES */ ]
}
```
**`scanStatus` derivation** (from DB `security_status` column): `clean`/`low`/`medium` → `passed`;
`critical`/`high` → `failed`; `unscanned`/null → `not_scanned`.
### 1.3 `POST /api/marketplace/install` (tier: PRO)
**Body**: `{ packageId: number, installPath?: string, settings?: Record<string,string>,
force?: boolean, forceInsecure?: boolean }`. `packageId` required (`400` if missing); `404` if the
package ID is not found.
**SecurityGate gating** (heuristics-only; Cisco/GenTrust/MCP-Guardian disabled):
| Severity | Behavior |
|---|---|
| `CRITICAL` (score 0) | **`403`** always blocked — `{ blocked:true, severity, score, findings, message }` |
| `HIGH` (score 25) | **`403`** unless `force:true`. With `force:true`, override is audit-logged and install proceeds |
| `MEDIUM` | Install proceeds; `findings` returned as `warnings` |
| `LOW` | Install proceeds; logged to audit trail |
| `CLEAN` | Install proceeds immediately |
**Success response** (`200` on success, `422` on installer failure): the installer result spread,
plus a `security` block:
```jsonc
{
"success": true,
/* ...MarketplaceInstaller result fields... */
"security": {
"severity": "MEDIUM",
"score": 60,
"findingsCount": 2,
"findings": [ /* ScanFinding[] */ ],
"warnings": ["[medium] <finding title>", ...] // only present when MEDIUM
}
}
```
After a successful install the DB `packages.security_status` / `security_score` columns are updated.
### 1.4 Other marketplace shapes
- **`POST /api/marketplace/uninstall`** — Body `{ packageId: number }`; `200`/`422` with installer result.
- **`POST /api/marketplace/security-check`** — Body `{ packageId: number }`. Returns
`{ packageId, severity, score, blocked, enginesUsed, findingsCount, findings, durationMs, contentHash }`.
- **`GET /api/marketplace/installed`** — `{ installations: [...], total }`.
- **`GET /api/marketplace/sources`** — `{ sources: [...], total }` (each source includes a package count).
- **`POST /api/marketplace/sources`** — Body `{ name, url, displayName? }`. `400` if name/url missing
or URL invalid; `409` if name exists. `source_type` auto-detected: `github.com` URL → `community_repo`,
else `aggregator`. Returns `201 { source, syncResult }`.
- **`DELETE /api/marketplace/sources/:id`** — `400` invalid id; `404` not found; **`403`** if
`!source.is_custom` (built-in sources can't be deleted). Success: `{ deleted:true, sourceId, name }`.
- **`GET /api/marketplace/categories`** — `{ categories: PACKAGE_CATEGORIES, total }`.
- **`POST /api/marketplace/sync`** — Body `{ sources?: string[] }`. Returns
`{ sourcesChecked, packagesAdded, packagesUpdated, errors: string[], details: [...] }`.
Emits a notification (`category:'agent'`, `actionUrl:'/capabilities'`) when new packages appear.
- **`GET /api/marketplace/security-status`** — `{ ciscoScannerAvailable, jsSecurityGateVersion:'1.0',
totalScanned, totalPassed, totalFailed, hint? }`. `hint` suggests
`pip install cisco-ai-skill-scanner` when the Cisco scanner is missing.
- **`POST /api/marketplace/publish`** (PRO) — Body `{ skillName: string }`. Reads
`~/.waggle/skills/<skillName>.md`, validates frontmatter via `validateSkillMd` (`422` on failure),
runs `SecurityGate` (`403` if `blocked`), upserts into a `user-published` source. Returns
`201 { success:true, packageId, skillName, metadata, security }`. Path-traversal in `skillName`
→ `400`.
### 1.5 Marketplace dev routes (`marketplace-dev.ts`)
**Gated behind env `WAGGLE_DEV_MARKETPLACE=1`** — these routes do **not** register otherwise and are
NOT a production contract. Prefixed `/_dev/marketplace/`. The frontend should not depend on them.
| Method | Path | Purpose |
|---|---|---|
| GET | `/_dev/marketplace/search` | Catalog seam probe (returns `_dev:true` envelope) |
| GET | `/_dev/marketplace/security-check` | SecurityGate seam probe on a sample skill |
| GET | `/_dev/marketplace/packs` | Pack reconciliation seam |
| GET | `/_dev/marketplace/health` | DB seed verification: `{ status, dbPath, dbExists, dbSizeBytes, packageCount, sourceCount, packCount }` |
---
## 2. Skills, Capability Packs, Plugins & Hooks (`skills.ts`)
Skills are **markdown files** in `~/.waggle/skills/` that extend the system prompt. Plugins are
structured packages in `~/.waggle/plugins/` with a `plugin.json` manifest. On first run, starter
skills auto-install (when the skills dir is empty and `.starter-installed` marker is absent). Every
mutation reloads `server.agentState.skills` and (where relevant) records an audit-trail entry and a
content hash via `server.skillHashStore`. **Skill content written through the API is passed through
`redactSkillContent()`** (strips secrets + user paths).
### 2.1 Endpoints
| Method | Path | Purpose |
|---|---|---|
| POST | `/api/skills/starter-pack` | Install all starter skills; reloads agent state |
| GET | `/api/skills/starter-pack/catalog` | Browse starter skills with per-skill `state` + families |
| POST | `/api/skills/starter-pack/:id` | Install ONE starter skill (`409` if installed, `404` if unknown) |
| GET | `/api/skills/capability-packs/catalog` | List packs with per-skill states + `packState` |
| POST | `/api/skills/capability-packs/:id` | Install all skills in a pack |
| GET | `/api/skills` | List installed skills (name, length, 200-char preview) |
| GET | `/api/skills/suggestions` | Contextual recommendations (`?context=...&topN=3`) |
| GET | `/api/skills/:name` | Full skill content `{ name, content }` |
| POST | `/api/skills` | Create skill from raw `{ name, content }` |
| POST | `/api/skills/create` | Create skill from structured template `{ name, description, steps[], tools?, category? }` |
| PUT | `/api/skills/:name` | Update skill content (`404` if missing) |
| DELETE | `/api/skills/:name` | Delete skill |
| GET | `/api/skills/hash-status` | Which skills changed on disk vs. recorded hash |
| POST | `/api/skills/test` | Sandbox/dry-run: shows what a skill would inject into the prompt |
| GET | `/api/audit/installs` | Recent install audit trail (`?limit=`, max 100) |
| GET | `/api/plugins` | List installed plugins |
| POST | `/api/plugins/install` | Install a plugin from a local dir (`{ sourceDir }` or `{ path }`) |
| DELETE | `/api/plugins/:name` | Uninstall a plugin |
| GET | `/api/plugins/:name/tools` | List a plugin's declared tools + impl status |
| GET | `/api/plugins/:name/tools/:toolName` | Get one tool's impl file (returns template if absent) |
| PUT | `/api/plugins/:name/tools/:toolName` | Write a tool impl file (must export `execute()`) |
| DELETE | `/api/plugins/:name/tools/:toolName` | Delete a tool impl file |
| POST | `/api/plugins/:name/tools` | Declare a new tool in the plugin manifest |
| GET | `/api/hooks` | List `pre:tool` deny rules |
| POST | `/api/hooks` | Add a deny rule `{ type:'deny', tools:string[], pattern }` |
| DELETE | `/api/hooks/:index` | Remove a rule by index |
> **Path-traversal guard everywhere**: any `name`/`id` containing `..`, `/`, or `\` (and for
> `POST /api/skills` also a space) → `400 Invalid …`.
### 2.2 Skill `state` model (used by catalog endpoints)
A skill is in one of three states, computed against on-disk files and loaded `agentState.skills`:
| state | meaning |
|---|---|
| `active` | loaded in `agentState.skills` (in effect now) |
| `installed` | file exists on disk but not loaded |
| `available` | exists only in the starter pack, not installed |
`GET /api/skills/starter-pack/catalog` returns:
```jsonc
{
"skills": [
{
"id": "draft-memo",
"name": "Draft Memo",
"description": "...",
"family": "writing",
"familyLabel": "Writing & Docs",
"state": "available", // active | installed | available
"isWorkflow": false // true for research-team / review-pair / plan-execute
}
],
"families": [ { "id": "writing", "label": "Writing & Docs" }, ... ]
}
```
**Skill families** (`SKILL_FAMILIES` map, ordered): `writing`, `research`, `decision`, `planning`,
`communication`, `code`, `creative`. **Workflow skills** (`WORKFLOW_SKILLS`): `research-team`,
`review-pair`, `plan-execute`.
### 2.3 Capability packs
`GET /api/skills/capability-packs/catalog` returns `{ packs: [...] }`; each pack entry spreads the
pack manifest and adds `skillStates` (`{ id, state }[]`), `packState`
(`available | complete | incomplete`), `installedCount`, `totalCount`.
`POST /api/skills/capability-packs/:id` returns
`{ ok, pack:{id,name}, installed:string[], skipped:string[], errors? }`.
### 2.4 Skill creation responses
- **`POST /api/skills`** (raw) → `{ ok:true, name, path }`.
- **`POST /api/skills/create`** (structured) → name is kebab-cased; generates SKILL.md via
`generateSkillMarkdown`. Returns `{ success:true, path, registered:true, skill:{ name, description,
steps, tools, category } }`. Requires `name`, `description`, and non-empty `steps[]` (`400` otherwise).
- **`POST /api/skills/test`** (sandbox) → `{ skill:{ name, displayName, description, permissions[],
family, familyLabel, isWorkflow, contentLength }, wouldInject, wouldInjectLength, testPreview? }`.
`testPreview` only appears when `testInput` is supplied.
### 2.5 Audit trail (`GET /api/audit/installs`)
```jsonc
{ "entries": [ {
"id", "timestamp", "capabilityName", "capabilityType", "source",
"riskLevel", "trustSource", "approvalClass", "action", "initiator", "detail"
} ] }
```
### 2.6 Plugin tool files
`GET /api/plugins/:name/tools` →
`{ pluginName, tools:[{ name, description, parameters, hasImplementation, implPath, content }], toolsDir }`.
`PUT .../tools/:toolName` requires the body `content` to include both `export` and `execute`
(`400` otherwise) and hot-reloads the plugin runtime. `POST .../tools` declares a tool in
`plugin.json` (`409` if the tool name already exists; name must match `^[a-zA-Z0-9_-]+$`).
---
## 3. Connectors (`connectors.ts` — `/api/connectors/*`)
Connectors are backed by `fastify.connectorRegistry`. Credentials live in the **Vault** under
`connector:<id>` (plus sub-keys like `connector:<id>:email`).
| Method | Path | Purpose |
|---|---|---|
| GET | `/api/connectors` | List all connector definitions (`registry.getDefinitions()`) |
| GET | `/api/connectors/:id/health` | Live health probe (`404` unknown, `502` on probe throw) |
| POST | `/api/connectors/:id/connect` | Store credentials in vault + re-init the connector |
| POST | `/api/connectors/:id/disconnect` | Remove credential + all sub-keys |
- **`GET /api/connectors`** → `{ connectors: ConnectorDefinition[] }` (or `{ connectors: [] }` if no
registry). A `ConnectorDefinition` (from `@waggle/shared`) has: `id, name, description, service,
authType ('api_key'|'oauth2'|'bearer'|'basic'), status, capabilities (('read'|'write'|'search')[]),
substrate ('waggle'|'kvark'), tools:string[], config?, actions?, logoUrl?, category?, setupGuide?`.
- **`GET /api/connectors/:id/health`** → `ConnectorHealth` =
`{ id, name, status, lastChecked, error?, tokenExpiresAt? }`. On a thrown probe the error is
sanitized to `error:'Health check failed'` (never leaks raw detail) and returned with `502`.
- **`POST /api/connectors/:id/connect`** — Body `{ token?, apiKey?, refreshToken?, expiresAt?,
scopes?, email? }`. `value = token ?? apiKey` (`400` if neither). `404` if the connector is unknown.
`503` if vault unavailable. `authType` defaults to the connector's own or `'bearer'`. Returns
`{ connected:true, connectorId }`.
- **`POST /api/connectors/:id/disconnect`** → `{ disconnected, connectorId, cleanedKeys }`.
---
## 4. AI-OS Tool Launcher (`tools.ts` — `/api/tools/*`)
These routes detect external AI tools on the user's machine, launch them with workspace-context env
injection, manage hive-mind hook installation, and track spawned processes. Tool IDs are validated
against `SUPPORTED_TOOLS` from `@waggle/shared`:
**`claude-code`, `claude-desktop`, `cursor`, `codex`, `codex-desktop`, `hermes`, `openclaw`**.
Note: `launchTool()` only actually launches the **launch cohort** (claude-code, cursor,
claude-desktop) — others return `ok:false` with a reason.
| Method | Path | Status | Purpose |
|---|---|---|---|
| GET | `/api/tools/detect` | 200/500 | Scan machine for supported AI tools |
| POST | `/api/tools/launch` | 202/400 | Spawn a tool detached w/ workspace env; registers PID |
| GET | `/api/tools/processes` | 200 | List tracked running processes |
| POST | `/api/tools/kill` | 200/404/500 | Kill a **tracked** PID (SIGTERM → SIGKILL after 3s) |
| POST | `/api/tools/hooks` | 200/400/500 | Run `npx @waggle/hive-mind-hooks-<id> <action>` |
### 4.1 `GET /api/tools/detect`
Returns `ToolDetectionResult`:
```jsonc
{
"platform": "win32",
"detectedAt": "2026-06-06T...Z",
"tools": [
{
"id": "claude-code",
"displayName": "Claude Code",
"installed": true,
"installedPath": "/abs/path/to/binary", // or null
"version": "1.2.3", // or null
"hooksInstalled": false,
"hookPointerPath": null,
"diagnostic": "optional reason string"
}
// ... one DetectedTool per SUPPORTED_TOOLS entry
]
}
```
### 4.2 `POST /api/tools/launch`
**Body (Zod-validated)**: `{ id: ToolId, installedPath: string(1..1024), workspaceId?: string,
cwd?: string, args?: string[]≤50 }`. Validation failure → `400 { error:'Validation failed', details }`.
**Response** = `LaunchResult` `{ ok, pid: number|null, executed:{ binary, args, cwd? }, error? }`.
On `ok:false` → `400`; on success → **`202`** and the PID is registered in the tracker.
### 4.3 `GET /api/tools/processes` & `POST /api/tools/kill`
- `processes` → `{ processes: TrackedProcess[], total }` where
`TrackedProcess = { pid, toolId, startedAt, workspaceId? }`. **In-memory only — PID state does NOT
survive a sidecar restart.**
- `kill` → Body `{ pid: number }` (positive int). Only previously-tracked PIDs can be killed
(`not-tracked` → `404`; SIGTERM→SIGKILL failure → `500`).
### 4.4 `POST /api/tools/hooks`
**Body**: `{ id: ToolId, action: 'install'|'verify'|'uninstall', cliPath?: string }`. Returns
`HookCommandResult { ok, action, packageName, stdout, stderr, exitCode, ... }`. `ok:false` → `400`.
---
## 5. OAuth (`oauth.ts` — `/api/oauth/*`)
Browser-redirect OAuth flows for 5 providers. App credentials (`client_id` / `client_secret`) must
already be in the Vault under provider-specific keys; the callback stores the resulting token as
`<provider>_oauth_token` (and `<provider>_oauth_refresh_token` if present). CSRF is protected via an
in-memory `state` map (10-min expiry).
### 5.1 Configured providers (`OAUTH_PROVIDERS`)
| provider | clientIdKey | clientSecretKey | scopes |
|---|---|---|---|
| `github` | `GITHUB_OAUTH_CLIENT_ID` | `GITHUB_OAUTH_CLIENT_SECRET` | `repo`, `user`, `read:org` |
| `slack` | `SLACK_OAUTH_CLIENT_ID` | `SLACK_OAUTH_CLIENT_SECRET` | `chat:write`, `channels:read`, `users:read` |
| `google` | `GOOGLE_OAUTH_CLIENT_ID` | `GOOGLE_OAUTH_CLIENT_SECRET` | calendar + drive.readonly |
| `notion` | `NOTION_OAUTH_CLIENT_ID` | `NOTION_OAUTH_CLIENT_SECRET` | (none; Basic-auth token exchange) |
| `jira` | `JIRA_OAUTH_CLIENT_ID` | `JIRA_OAUTH_CLIENT_SECRET` | `read:jira-work`, `write:jira-work`, `read:jira-user` |
### 5.2 Endpoints
| Method | Path | Purpose |
|---|---|---|
| GET | `/api/oauth/providers` | List providers + credential/token status |
| GET | `/api/oauth/:provider/authorize` | Build OAuth URL and **redirect** to provider |
| GET | `/api/oauth/:provider/callback` | Exchange code → token, store in vault, return HTML page |
- **`GET /api/oauth/providers`** → `{ providers: [{ provider, hasCredentials, clientIdKey,
clientSecretKey, scopes, hasToken }] }`. The frontend uses `hasCredentials` to know whether the
"Connect" button can start the flow, and `hasToken` to show "connected".
- **`GET /api/oauth/:provider/authorize`** — `400` if provider unknown (returns
`availableProviders`) or credentials missing; `503` if vault unavailable. On success → HTTP
**redirect** to the provider's authorize URL. The callback URI is
`http://127.0.0.1:<port>/api/oauth/<provider>/callback`. Google adds `access_type=offline` +
`prompt=consent`; Jira adds `audience` + `prompt=consent`; Notion adds `owner=user`.
- **`GET /api/oauth/:provider/callback`** — returns an **HTML page** (not JSON) for success/error;
the success page auto-closes the tab after 2s. Token stored as `<provider>_oauth_token`
(`credentialType:'oauth2'`).
---
## 6. Vault (`vault.ts` — `/api/vault/*`)
Encrypted secret storage (`fastify.vault`). All routes return `503` if the vault is unavailable.
**Listing never returns values** — only `/reveal` does, and it is **same-origin-enforced**.
| Method | Path | Purpose |
|---|---|---|
| GET | `/api/vault` | List secrets (names, types, dates — NO values) + suggestions |
| POST | `/api/vault` | Add or update a secret |
| DELETE | `/api/vault/:name` | Delete a secret (`404` if not found) |
| POST | `/api/vault/:name/reveal` | Decrypt + return full value (**`403`** if external origin) |
- **`GET /api/vault`** →
`{ secrets: [{ name, type, updatedAt, isCommon }], suggestedKeys: string[], suggestedSecrets:
[{ category, items: [{ name, type, label }] }] }`. `suggestedKeys`/`suggestedSecrets` exclude
already-stored keys. The suggestion catalog (`SUGGESTED_SECRETS`) groups well-known keys by
category: **LLM Providers** (`anthropic`, `openai`, `google`, `mistral`, `deepseek`, `xai`,
`alibaba`, `minimax`, `zhipu`, `openrouter`), **Embedding Providers** (`voyage-api-key`),
**Search & Tools** (`perplexity`, `moonshot`, `TAVILY_API_KEY`, `BRAVE_API_KEY`,
`COMPOSIO_API_KEY`), **Code & DevOps** (`GITHUB_TOKEN`, `GITLAB_TOKEN`, `BITBUCKET_TOKEN`),
**Communication**, **Productivity**, **CRM & Sales**, **Cloud & Storage**, **User Credentials**.
- **`POST /api/vault`** — Body `{ name, value, type? }` (name + value required, `400` otherwise).
Returns `{ success:true, name }`.
- **`POST /api/vault/:name/reveal`** — Rejects non-local requests with `403`
(`isLocalRequest` guard). Returns `{ name, value, type }`.
> **Frontend note:** the LLM provider key naming convention is the **bare provider id** for native
> providers (`anthropic`, `openai`, `google`, ...) — this is the same key `GET /api/providers`
> checks (`vault.get(p.id)`). Connector creds use `connector:<id>`; OAuth uses `<provider>_oauth_token`.
---
## 7. LLM & Search Providers (`providers.ts` — `/api/providers`)
Single source of truth for the model/provider picker (Settings, Onboarding, workspace selector,
Spawn dialog, Agents). One route.
| Method | Path | Purpose |
|---|---|---|
| GET | `/api/providers` | All LLM providers + models + key status, search providers, active search |
**Response**:
```jsonc
{
"providers": [
{
"id": "anthropic", "name": "Anthropic",
"keyPrefix": "sk-ant-", "keyUrl": "https://...", "badge": null,
"requiresKey": true, "hasKey": true,
"models": [
{ "id": "claude-opus-4-7", "name": "Claude Opus 4.7", "cost": "$$$", "speed": "slow" }
]
}
],
"search": [
{ "id": "perplexity", "name": "Perplexity", "vaultKey": "perplexity",
"priority": 1, "requiresKey": true, "hasKey": false }
],
"activeSearch": "duckduckgo" // highest-priority search provider that has a key
}
```
- **`cost`** is one of `$ | $$ | $$$`; **`speed`** is `fast | medium | slow`.
- `hasKey` is computed from the vault: native providers check `vault.get(provider.id)`; Ollama is
always keyless (`requiresKey:false`).
- **Static LLM providers** (`LLM_PROVIDERS`): `anthropic`, `openai`, `google`, `deepseek`, `xai`,
`mistral`, `alibaba` (Qwen), `minimax`, `zhipu` (GLM), `moonshot` (Kimi), `perplexity`,
`openrouter`, `ollama`.
- **Live enrichment**: `ollama` is enriched from `http://localhost:11434/api/tags` (or `OLLAMA_HOST`)
— model ids get an `ollama/` prefix, `source:'local'|'cloud'`, optional `sizeMB`; `badge` becomes
`"N installed"` / `"Running"` / `"Not running"`. `openrouter` (only when its key is present) is
enriched with up to 20 free models from `https://openrouter.ai/api/v1/models` (1h cache),
`source:'cloud'`.
- **Search providers** (`SEARCH_PROVIDERS`, priority order): `perplexity` (1), `tavily` (2,
`TAVILY_API_KEY`), `brave` (3, `BRAVE_API_KEY`), `duckduckgo` (4, keyless fallback).
---
## 8. How the pieces connect
```mermaid
flowchart TD
subgraph Frontend["Frontend (Lovable rebuild)"]
UI_MKT["Marketplace / Capabilities view"]
UI_SKILL["Skills & Plugins (Install Center)"]
UI_CONN["Connectors view"]
UI_LAUNCH["AI-OS Launcher dock"]
UI_SET["Settings / Onboarding (providers + keys)"]
end
subgraph Sidecar["Fastify sidecar (/api/*)"]
MKT["marketplace.ts"]
SKILL["skills.ts"]
CONN["connectors.ts"]
TOOLS["tools.ts"]
OAUTH["oauth.ts"]
VAULT["vault.ts"]
PROV["providers.ts"]
end
subgraph Backing["Backing stores / services"]
MDB[("MarketplaceDB\n(marketplace.db)")]
DISK[("~/.waggle/skills + plugins")]
REG["connectorRegistry"]
V[("Vault (encrypted secrets)")]
AGENTST["agentState.skills"]
AUDIT["auditStore"]
PROC["ToolProcessTracker"]
end
UI_MKT --> MKT --> MDB
MKT -->|SecurityGate scan| AUDIT
UI_SKILL --> SKILL --> DISK
SKILL --> AGENTST
SKILL --> AUDIT
MKT -->|publish reads| DISK
UI_CONN --> CONN --> REG
CONN -->|store creds connector:id| V
UI_LAUNCH --> TOOLS --> PROC
UI_SET --> PROV
PROV -->|hasKey lookups| V
UI_SET --> VAULT --> V
UI_CONN -->|OAuth start| OAUTH
OAUTH -->|token <provider>_oauth_token| V
PROV -->|live| OLLAMA["Ollama :11434"]
PROV -->|live free models| OR["OpenRouter API"]
```
### Cross-cutting facts the frontend must internalize
1. **Vault is the credential hub.** LLM keys = bare provider id; connectors = `connector:<id>`;
OAuth tokens = `<provider>_oauth_token`. Provider/connector "connected" status is derived from
vault presence, never sent as a value except via `/api/vault/:name/reveal` (local-origin only).
2. **Tier gates return `403 TIER_INSUFFICIENT`** with `{ error, message, required, actual, upgradeUrl }`.
Marketplace install + publish require **PRO**; enterprise-packs require **ENTERPRISE**.
3. **Marketplace 503** means `fastify.marketplace` (the DB) is absent — treat as "marketplace off".
4. **Security gating** can hard-block installs (`403` for CRITICAL/HIGH) — surface `findings` and the
`force` override path (HIGH only) in the UI.
5. **Skill mutations reload agent state** and redact secrets; skill `state` is
`active | installed | available`.
6. **Tool launch returns `202` + a PID** that is tracked in-memory only (lost on sidecar restart);
only tracked PIDs can be killed.

457
docs/backend-map/sections/03e-api-evolution-governance.md Normal file
View File

@@ -0,0 +1,457 @@
# 03e — Evolution & Governance API
**Purpose.** This subsystem groups the Waggle OS sidecar's "machine-improves-itself + you-stay-in-control" surfaces: self-evolution runs (propose/accept/reject/run prompt mutations), user feedback capture, local-only telemetry, EU AI Act compliance reporting, agent cost tracking, capability/plugin status, and the approvals inbox (pending tool approvals + persistent grants). Every endpoint is served by the local Fastify sidecar under `/api/...` and returns JSON unless noted (one route returns a PDF binary, one streams SSE). For a frontend rebuild, treat the tables below as the literal contract — paths, methods, request bodies, and response shapes are quoted from the route source.
> Source files: `packages/server/src/local/routes/{evolution,feedback,telemetry,compliance,cost,capabilities,approval,validate}.ts` and `packages/server/src/local/services/{evolution-service,optimizer-service}.ts`. Data shapes are grounded in `packages/hive-mind-core/src/mind/evolution-runs.ts`, `packages/core/src/compliance/types.ts`, and `packages/core/src/telemetry.ts`.
---
## 1. Endpoint Index (every route in this subsystem)
| Method | Path | Purpose |
|---|---|---|
| GET | `/api/evolution/runs` | List evolution proposals/runs (newest first; filterable) |
| GET | `/api/evolution/runs/:uuid` | Single run detail with parsed JSON blobs |
| POST | `/api/evolution/runs/:uuid/accept` | Accept + deploy a proposed run |
| POST | `/api/evolution/runs/:uuid/reject` | Reject a proposed run |
| GET | `/api/evolution/targets` | Enumerate evolvable targets (personas + spec sections) |
| GET | `/api/evolution/baseline` | Fetch current baseline text for a target |
| POST | `/api/evolution/run` | Trigger a real evolution run (JSON or SSE stream) |
| GET | `/api/evolution/status` | Aggregate status counts for the dashboard |
| POST | `/api/feedback` | Record thumbs up/down feedback on an agent message |
| GET | `/api/feedback/stats` | Improvement stats + trend |
| GET | `/api/telemetry/summary` | Local telemetry summary object |
| GET | `/api/telemetry/events` | Query telemetry events (filterable) |
| DELETE | `/api/telemetry/events` | Clear all telemetry events (right to delete) |
| GET | `/api/telemetry/status` | Telemetry enabled flag + total event count |
| POST | `/api/telemetry/toggle` | Enable/disable telemetry |
| POST | `/api/telemetry/track` | Record a single telemetry event (frontend) |
| GET | `/api/compliance/status` | EU AI Act compliance status (per-article) |
| POST | `/api/compliance/export` | Generate audit report (JSON) |
| POST | `/api/compliance/export-pdf` | Generate audit report as a PDF binary |
| GET | `/api/compliance/interactions` | List recorded AI interactions |
| POST | `/api/compliance/interactions` | Record an AI interaction |
| GET | `/api/compliance/models` | Model inventory for a date range |
| GET | `/api/compliance/templates` | List saved compliance report templates |
| GET | `/api/compliance/templates/:id` | Get one template by numeric id |
| POST | `/api/compliance/templates` | Create a compliance template |
| PATCH | `/api/compliance/templates/:id` | Update a compliance template |
| DELETE | `/api/compliance/templates/:id` | Delete a compliance template |
| GET | `/api/cost/summary` | Cost dashboard: today/week/all-time + daily breakdown + budget |
| GET | `/api/cost/by-workspace` | Per-workspace cost breakdown (TEAMS tier gated) |
| GET | `/api/costs` | Alias → `/api/cost/summary` (internal re-route, 200) |
| GET | `/api/capabilities/status` | Plugins/MCP/skills/tools/commands/hooks/workflows status |
| POST | `/api/capabilities/plugins/:name/enable` | Enable a plugin |
| POST | `/api/capabilities/plugins/:name/disable` | Disable a plugin |
| POST | `/api/approval/:requestId` | Approve/deny a pending tool execution |
| GET | `/api/approval/pending` | List pending approvals (for reconnection) |
| GET | `/api/approval/grants` | List all persistent approval grants |
| DELETE | `/api/approval/grants/:id` | Revoke a single grant |
| POST | `/api/approval/grants/clear` | Wipe all grants |
> `validate.ts` exposes **no routes** — it is a helper module (`isSafeSegment`, `assertSafeSegment`) for rejecting path-traversal in route params. `optimizer-service.ts` is an internal service (prompt classify/expand via Haiku) consumed by the chat loop, **not** an HTTP route.
---
## 2. Evolution (self-improvement loop)
The evolution loop proposes mutations to either a **persona system prompt** or a **behavioral-spec section**, gates them, and stores `proposed` runs. The user reviews and accepts/rejects from the Memory → Evolution UI. Accepting deploys the new text to disk and fires a cache-invalidation event; the loop **never auto-deploys**.
### 2.1 The `EvolutionRun` entity (full shape)
Returned by list/detail/accept/reject. From `packages/hive-mind-core/src/mind/evolution-runs.ts`:
| Field | Type | Notes |
|---|---|---|
| `id` | number | Autoincrement row id |
| `run_uuid` | string | Stable id used in all `:uuid` routes |
| `target_kind` | `'persona-system-prompt' \| 'behavioral-spec-section' \| 'tool-description' \| 'skill-body' \| 'generic'` | Only the first two can currently be **deployed** |
| `target_name` | string \| null | Persona id (e.g. `coder`) or spec section id |
| `baseline_text` | string | The pre-evolution instruction text |
| `winner_text` | string | The evolved/winning instruction text |
| `winner_schema_json` | string \| null | JSON-encoded `Schema` (DSPy signature) when structure evolved |
| `delta_accuracy` | number | Score improvement over baseline |
| `gate_verdict` | `'pass' \| 'fail'` | Constraint-gate verdict |
| `gate_reasons_json` | string | JSON array of `{gate, verdict, reason}` |
| `status` | `'proposed' \| 'accepted' \| 'rejected' \| 'deployed' \| 'failed'` | Lifecycle state |
| `artifacts_json` | string \| null | Per-generation history / scores / Pareto front |
| `user_note` | string \| null | Note attached on accept |
| `failure_reason` | string \| null | Set when status is `failed` |
| `created_at` | string | ISO timestamp |
| `decided_at` | string \| null | Set on accept/reject |
| `deployed_at` | string \| null | Set on successful deploy |
### 2.2 `GET /api/evolution/runs`
List runs, newest first. All query params optional.
| Query param | Type | Meaning |
|---|---|---|
| `status` | string or string[] (repeatable) | Filter by one or more statuses |
| `targetKind` | string | Filter by target kind |
| `targetName` | string | Filter by target name (e.g. `coder`) |
| `since` | ISO string | Only runs after this time |
| `limit` | string→int | Default 50, clamped to `1..500` |
**Response:** `{ runs: EvolutionRun[], count: number }`.
### 2.3 `GET /api/evolution/runs/:uuid`
Single run. **404** `{ error: 'Run not found' }` if missing. On success returns the full `EvolutionRun` **plus** parsed convenience fields:
```jsonc
{
...EvolutionRun,
"winnerSchema": object | null, // parsed from winner_schema_json
"artifacts": object | null, // parsed from artifacts_json
"gateReasons": Array<{gate, verdict, reason}> // parsed from gate_reasons_json, defaults []
}
```
### 2.4 `POST /api/evolution/runs/:uuid/accept`
- **Body:** `{ note?: string }`
- **Guards:** 404 if not found; **409** if `status !== 'proposed'` (`error: 'Run is in status "<x>" — only proposed runs can be accepted'`).
- **Effect:** marks accepted → runs the deploy dispatcher → moves to `deployed` (success) or `failed` (throw). Deploy is only implemented for `persona-system-prompt` (writes a persona override) and `behavioral-spec-section` (writes a spec-section override); `tool-description`/`skill-body`/`generic` throw "not yet implemented" and end as `failed`.
- **Side effect:** emits `persona:reloaded` or `behavioral-spec:reloaded` on the server event bus so the chat route drops its cached system prompt.
- **Response:** the updated `EvolutionRun` (200). 500 if accept returned no record.
### 2.5 `POST /api/evolution/runs/:uuid/reject`
- **Body:** `{ reason?: string }`
- **Guards:** 404 if not found; **409** if not `proposed`.
- **Response:** the updated `EvolutionRun` (200).
### 2.6 `GET /api/evolution/targets`
Populates the "Run" form dropdowns. **Response:**
```jsonc
{
"personas": Array<{ id, name, description, icon }>, // from listPersonas()
"sections": string[], // BEHAVIORAL_SPEC_SECTIONS
"defaultSchema": Schema // generic default DSPy signature
}
```
### 2.7 `GET /api/evolution/baseline?kind=X&name=Y`
Returns the current live baseline for one target so the Run form can pre-fill it.
- **400** if `kind` or `name` missing, or `kind` not one of the two supported.
- **404** for unknown persona / unknown section.
- `kind=persona-system-prompt` → returns the persona's live `systemPrompt`.
- `kind=behavioral-spec-section` → returns the **active** section text (deployed overrides applied, falling back to compile-time `BEHAVIORAL_SPEC`).
- **Response:** `{ baseline: string, schemaBaseline: Schema }`.
### 2.8 `POST /api/evolution/run` (trigger a real run — JSON or SSE)
Synchronously runs a GEPA + EvolveSchema composition using the vault's Anthropic key (Haiku-backed judge/mutate/execute). Persists a `proposed` run if one wins.
**Body:**
| Field | Type | Notes |
|---|---|---|
| `targetKind` | EvolutionTarget | Required; must be in the 5-value union |
| `targetName` | string | Required; non-empty |
| `baseline` | string | Required; non-empty current instruction text |
| `schemaBaseline` | `Schema` | Required object `{ name: string, fields: array, version }` |
| `minDelta` | number | Optional; default 0.02 |
| `gepa` | object | `{ populationSize?, generations?, miniEvalSize?, anchorEvalSize?, seed?, concurrency? }` (concurrency default 4) |
| `schema` | object | `{ populationSize?, generations?, evalSize?, anchorEvalSize?, seed? }` |
| `gateOptions` | `GateOptions` | Optional constraint-gate config |
**Status codes:** `200` ran (see `body.outcome`), `400` validation error, `422` no Anthropic key in vault (`'No Anthropic API key configured. Add one in Settings → Vault.'`), `503` `@ax-llm/ax` unavailable, `500` run threw.
**JSON response payload:**
```jsonc
{
"outcome": string, // e.g. "proposed" / "skipped-*"
"reason": string,
"run": EvolutionRun | null,
"gateResults": ...,
"composeSummary": {
"combinedDelta", "fullyImproved",
"schemaImproved", "schemaDelta",
"instructionImproved", "instructionDelta",
"winnerId"
} | null
}
```
**SSE mode (frontend should prefer this):** send header `Accept: text/event-stream`. The route streams `event:` frames — `open` (`{targetKind, targetName}`), repeated `progress` (the `GEPAProgress` object: `{phase, generation, populationSize, best, message?}`), then either `done` (the JSON payload above) or `error` (`{error}`). The run completes server-side even if the client disconnects (it does not cancel in-flight LLM spend).
### 2.9 `GET /api/evolution/status`
- **Query:** `targetKind?`, `targetName?`, `since?`
- **Response:** `{ counts: Record<EvolutionRunStatus, number>, pendingCount: number }` where `pendingCount === counts.proposed`.
### 2.10 Background autonomy (no HTTP surface)
`EvolutionService` (in `services/evolution-service.ts`) is an **opt-in** `setInterval` daemon (env `WAGGLE_EVOLUTION_AUTO_ENABLED=1`, default off; tick interval 6h, min 60s). Each tick picks one target whose new-trace count clears `minTracesPerTarget` (default 20; eligible outcomes `success`/`corrected`/`verified`) and produces a `proposed` run via the same orchestrator as `/api/evolution/run`. It **never auto-accepts** — proposals still flow through the manual accept/reject routes above. The frontend does not call this directly; it just sees new `proposed` runs appear.
---
## 3. Feedback
Feedback writes to a `feedback_entries` table in the personal `.mind` DB (auto-created). Negative feedback with a reason is cross-recorded as a `correction` improvement signal feeding the self-improvement loop.
### 3.1 `POST /api/feedback`
**Body (`FeedbackBody`):**
| Field | Type | Required | Notes |
|---|---|---|---|
| `sessionId` | string | yes | 400 if missing/non-string |
| `messageIndex` | number | yes | Must be ≥ 0 |
| `rating` | `'up' \| 'down'` | yes | 400 if not in set |
| `reason` | `'wrong_answer' \| 'too_verbose' \| 'wrong_tool' \| 'too_slow' \| 'other'` | no | Validated if present |
| `detail` | string | no | Free text, defaults `''` |
**Response:** `{ ok: true }`. 500 on DB failure.
### 3.2 `GET /api/feedback/stats`
**Response:**
```jsonc
{
"totalFeedback": number,
"positiveRate": number, // 0..1, 2 decimals
"topIssues": string[], // up to 5 negative-feedback reasons by frequency
"correctionsThisWeek": number, // from improvement_signals (last 7d)
"improvementTrend": string // e.g. "+12%" / "-5%" / "0%"
}
```
---
## 4. Telemetry (local-only — no cloud reporting)
`TelemetryEvent = { id, event, properties: Record<string, unknown>, created_at }`. `TelemetrySummary = { enabled, totalEvents, firstEvent, lastEvent, onboardingCompleted, totalSessions, embeddingProvider, templatesUsed, ... }`.
| Endpoint | Request | Response |
|---|---|---|
| `GET /api/telemetry/summary` | — | `TelemetrySummary` |
| `GET /api/telemetry/events` | query `event?, since?, until?, limit?` (limit default 100) | `TelemetryEvent[]` |
| `DELETE /api/telemetry/events` | — | result of `telemetry.clear()` (right-to-delete) |
| `GET /api/telemetry/status` | — | `{ enabled: boolean, totalEvents: number }` |
| `POST /api/telemetry/toggle` | `{ enabled: boolean }` | `{ enabled }` (also persists to `WaggleConfig`) |
| `POST /api/telemetry/track` | `{ event: string, properties?: object }` | `{ ok: true }`; 400 if `event` missing |
---
## 5. Compliance (EU AI Act)
All compliance routes require the personal mind; they return **503** `{ error: 'Personal mind not available' }` if it isn't ready. Interactions live in the `ai_interactions` table; templates in their own table on the same personal DB.
### 5.1 `GET /api/compliance/status?workspaceId=`
Returns the per-article `ComplianceStatus`:
| Field | Shape |
|---|---|
| `overall` | `'compliant' \| 'warning' \| 'non-compliant'` |
| `art12Logging` | `ArticleStatus & { totalInteractions }` |
| `art14Oversight` | `ArticleStatus & { humanActions, approvalRate }` |
| `art19Retention` | `ArticleStatus & { oldestLogDate, retentionDays }` |
| `art26Monitoring` | `ArticleStatus & { activeMonitors: string[] }` |
| `art50Transparency` | `ArticleStatus & { modelsDisclosed: boolean }` |
`ArticleStatus = { status: 'compliant'|'warning'|'non-compliant', detail: string }`.
### 5.2 `POST /api/compliance/export` and `POST /api/compliance/export-pdf`
Both take the same `AuditReportRequest` body:
```jsonc
{
"workspaceId": string?,
"from": string, // required ISO date — 400 if missing
"to": string, // required ISO date
"format": "json" | "pdf" | "both",
"include": {
"interactions": boolean, "oversight": boolean, "models": boolean,
"provenance": boolean, "riskAssessment": boolean, "fria": boolean
}
}
```
- `/export` returns the `AuditReport` JSON object: `{ report:{version,generatedAt,period,generatedBy}, workspace:{id,name,riskLevel,riskClassifiedAt}|null, complianceStatus, modelInventory[], humanOversightLog[], harvestProvenance[], interactionCount }`.
- `/export-pdf` returns **`application/pdf`** binary with `Content-Disposition: attachment; filename="ai-act-compliance-<from>-to-<to>.pdf"`. It additionally accepts three optional template-override fields on the body: `templateOrgName`, `templateFooterText`, `templateRiskClassification` (an `AIActRiskLevel`). 500 on PDF render failure.
### 5.3 `GET /api/compliance/interactions` and `POST`
- **GET** query `limit?` (default 20, max 100), `workspaceId?`. Response `{ interactions: AIInteraction[] }` (by workspace if `workspaceId` given, else recent N).
- **POST** body `RecordInteractionInput` (requires `model` + `provider`, else 400). Returns the stored `AIInteraction`.
`AIInteraction` fields: `id, timestamp, workspaceId, sessionId, model, provider, inputTokens, outputTokens, costUsd, toolsCalled[], humanAction('approved'|'denied'|'modified'|'none'), riskContext, importedFrom, persona, inputText, outputText`.
### 5.4 `GET /api/compliance/models?from=&to=&workspaceId=`
Returns `{ models: ModelInventoryEntry[] }`, each `{ model, provider, calls, inputTokens, outputTokens, costUsd }`.
### 5.5 Compliance templates (M-03 CRUD)
`ComplianceTemplate = { id, name, description, sections: ComplianceTemplateSections, riskClassification: AIActRiskLevel|null, orgName, footerText, createdAt, updatedAt }`, where `sections` mirrors the six `include` booleans.
| Endpoint | Body | Notes |
|---|---|---|
| `GET /api/compliance/templates` | — | `{ templates: ComplianceTemplate[] }` |
| `GET /api/compliance/templates/:id` | — | 400 invalid id, 404 not found, else `{ template }` |
| `POST /api/compliance/templates` | Zod-validated `CreateComplianceTemplateInput` | 201 `{ template }`; 400 on invalid body (`detail` = Zod issues) |
| `PATCH /api/compliance/templates/:id` | Zod-validated `UpdateComplianceTemplateInput` | 404 if not found |
| `DELETE /api/compliance/templates/:id` | — | `{ deleted: true }`; 404 if not found |
Zod schemas: `sections` is all six booleans required; `riskClassification` ∈ `{minimal, limited, high-risk, unacceptable}`. Sections **merge (union)** with the runtime `include` flags in the UI before POSTing to `/export` — the export routes stay template-agnostic.
---
## 6. Cost dashboard
Data source is the **in-memory** `CostTracker` (populated by the chat route per agent turn). All costs are estimates from published model pricing; fallback pricing is Sonnet (`$0.003`/1K in, `$0.015`/1K out).
### 6.1 `GET /api/cost/summary?days=`
`days` default 7, max 90. **Response:**
```jsonc
{
"today": { inputTokens, outputTokens, estimatedCost, turns },
"allTime": { inputTokens, outputTokens, estimatedCost, turns, byModel },
"week": { inputTokens, outputTokens, estimatedCost, turns },
"daily": [ { date, inputTokens, outputTokens, cost, turns } ], // one per day in range
"budget": { dailyBudget: number|null, todayCost, budgetStatus: 'ok'|'warning'|'exceeded', budgetPercent }
}
```
`budget.dailyBudget` is read from `/api/settings`; `budgetStatus` is `warning` at ≥80% and `exceeded` at ≥100% of `dailyBudget`.
### 6.2 `GET /api/cost/by-workspace` (TEAMS-gated)
Guarded by `requireTier('TEAMS')`. Returns `{ workspaces: Array<{ workspaceId, workspaceName, inputTokens, outputTokens, estimatedCost, turns, percentOfTotal }>, totalCost }`, sorted by cost descending.
### 6.3 `GET /api/costs`
Discoverability alias. Internally re-routes to `/api/cost/summary` (passing `days` through) and returns the same 200 body. Free for all tiers (usage info is not gated).
---
## 7. Capabilities (read-only status + plugin toggles)
### 7.1 `GET /api/capabilities/status`
One aggregated snapshot (500 with `{error}` on failure):
```jsonc
{
"plugins": [ { name, state, tools, skills } ],
"mcpServers": [ { name, state, healthy, tools } ],
"skills": [ { name, length } ],
"tools": { count, native, plugin, mcp },
"commands": [ { name, description, usage } ],
"hooks": { registered: 10, recentActivity: [ { event, timestamp, cancelled, reason } ] },
"workflows": [ { name, description, steps } ]
}
```
### 7.2 Plugin toggles
| Endpoint | Effect | Response |
|---|---|---|
| `POST /api/capabilities/plugins/:name/enable` | `pluginRuntimeManager.enable(name)` | `{ ok: true, name, state: 'active' }`; 503 if no runtime; 400 on error |
| `POST /api/capabilities/plugins/:name/disable` | `pluginRuntimeManager.disable(name)` | `{ ok: true, name, state: 'disabled' }`; 503/400 as above |
---
## 8. Approvals inbox + grants
The agent loop registers a **pending approval** when a tool needs human sign-off; the request hangs on a promise until the user resolves it via the API. "Always allow" persists a **grant** so future identical `(toolName, input, sourceWorkspaceId)` requests resolve silently.
| Endpoint | Body / Params | Behavior |
|---|---|---|
| `POST /api/approval/:requestId` | `{ approved: boolean, always?: boolean, reason?: string, sourceWorkspaceId?: string\|null }` | 404 if no pending request; if `approved && always` persists a grant first; resolves the pending promise and removes it. Returns `{ ok, requestId, approved, always }` |
| `GET /api/approval/pending` | — | `{ pending: Array<{ requestId, toolName, input, timestamp }>, count }` |
| `GET /api/approval/grants` | — | `{ grants: [...], count }` |
| `DELETE /api/approval/grants/:id` | param `id` | `{ ok: true, id }`; 404 if grant not found |
| `POST /api/approval/grants/clear` | — | `{ ok: true }` — wipes every grant |
For the frontend: poll `GET /api/approval/pending` on reconnect to rebuild the inbox; the live push of new approval requests arrives via the chat/SSE stream (out of scope here).
---
## 9. Cross-cutting notes for the rebuild
- **No auth headers documented here** — these are local sidecar routes. Only `/api/cost/by-workspace` is tier-gated (`requireTier('TEAMS')` preHandler).
- **Compliance routes degrade with 503** when the personal mind isn't loaded — handle that as an empty/loading state, not an error toast.
- **Two non-JSON responses:** `/api/compliance/export-pdf` (PDF binary, trigger a download) and `/api/evolution/run` with `Accept: text/event-stream` (SSE; render a progress bar from `GEPAProgress`).
- **State machine:** evolution runs only leave `proposed` via accept/reject; the UI must disable accept/reject buttons for any non-`proposed` run (the server returns 409 otherwise).
- **Numeric template ids:** compliance template routes use a numeric `:id` and 400 on non-finite values; everything else keyed by string `uuid`/`requestId`/grant `id`/plugin `name`.
---
## 10. Subsystem map
```mermaid
flowchart TD
subgraph UI[Frontend - Lovable rebuild]
EvoUI[Memory: Evolution tab]
FbUI[Feedback thumbs]
CompUI[Compliance app]
CostUI[Cost dashboard]
CapUI[Capabilities status]
AprUI[Approvals inbox]
end
subgraph API[Fastify sidecar /api]
Evo[/evolution/*/]
Fb[/feedback/*/]
Tel[/telemetry/*/]
Comp[/compliance/*/]
Cost[/cost/* and /costs/]
Cap[/capabilities/*/]
Apr[/approval/*/]
end
subgraph Stores[Persistence and runtime state]
RunStore[(EvolutionRunStore<br/>evolution_runs)]
TraceStore[(ExecutionTraceStore)]
FeedTbl[(feedback_entries +<br/>improvement_signals)]
TelTbl[(telemetry_events)]
Inter[(ai_interactions +<br/>compliance_templates)]
CostMem[CostTracker in-memory]
Pending[pendingApprovals map]
Grants[(approvalGrantStore)]
end
EvoUI --> Evo --> RunStore
Evo -->|runOnce| TraceStore
Evo -->|accept deploy| Disk[persona / spec override files]
Evo -.persona:reloaded / behavioral-spec:reloaded.-> Bus[server eventBus]
EvoSvc[EvolutionService daemon<br/>opt-in setInterval] --> RunStore
FbUI --> Fb --> FeedTbl
Tel --> TelTbl
CompUI --> Comp --> Inter
Comp -->|export-pdf| PDF[renderComplianceReportPdf]
CostUI --> Cost --> CostMem
CapUI --> Cap --> Runtime[plugin / mcp / hook registries]
AprUI --> Apr
Apr --> Pending
Apr --> Grants
```
---
## Workflow Templates CRUD (`workflows.ts`) — `/api/workflows`
> Added to close audit gap #2. These manage **multi-step agent workflow templates** (sequences of agent steps with an aggregation strategy). Built-in templates from `WORKFLOW_TEMPLATES` are read-only; custom ones are persisted as JSON under `~/.waggle/workflows/` (the sidecar `dataDir`). Execution of a workflow happens through the agent-run surface (see 03a / 05e); these endpoints only manage the template definitions.
| Method | Path | Request body | Response | Notes |
|---|---|---|---|---|
| `GET` | `/api/workflows` | — | `{ workflows: (WorkflowTemplate & { builtIn: boolean })[], builtInCount: number, customCount: number }` | Returns built-ins first, then custom (`builtIn:false`) |
| `POST` | `/api/workflows` | `Partial<WorkflowTemplate>` — **requires** `name` + non-empty `steps[]` | `201 WorkflowTemplate` | `description` defaults `""`, `aggregation` defaults `"concatenate"`; `400 { error }` if `name`/`steps` missing |
| `DELETE` | `/api/workflows/:name` | — | `{ deleted: true, name }` | `404 { error: "Workflow not found" }` if the custom workflow doesn't exist (built-ins can't be deleted) |
**`WorkflowTemplate`** = `{ name: string; description: string; steps: WorkflowStep[]; aggregation: 'concatenate' | ... }` (from `@waggle/agent`). The frontend's workflow-composer UI reads `GET` to populate the template list and `POST`/`DELETE` to manage user-authored ones.

347
docs/backend-map/sections/03f-api-realtime-ops.md Normal file
View File

@@ -0,0 +1,347 @@
# 03f — Real-Time & Ops API Surface
**Purpose.** This section is the frontend rebuild contract for Waggle OS's *real-time and operations* backend: the two WaggleDance signal systems (the UI-facing `/api/waggle/*` stream plus the v2 protocol bus `/api/waggle-dance/*`), the audit `/api/events` stream, cron schedules, persisted notifications, offline message queue, encrypted backup/restore, the agent fleet, the LiteLLM / local-inference / Anthropic LLM proxies, filesystem browse, the browser extension health check, and Telegram outbound push. Four of these endpoints are **Server-Sent Events (SSE)** streams the UI subscribes to with `EventSource`; everything else is plain JSON REST.
All routes are served by the **Node.js Fastify sidecar** bundled into the Tauri desktop binary. Every path below was read directly from `packages/server/src/local/routes/` and `packages/server/src/local/`.
---
## 1. SSE / streaming endpoints (read these first)
Four endpoints hold the connection open and push `text/event-stream` frames. The frontend consumes them with `new EventSource(url)`. Each sends an initial connect frame, a periodic heartbeat/keepalive comment (`: ...`), and named events.
| Endpoint | Initial frame | Named events emitted | Heartbeat | Source |
|---|---|---|---|---|
| `GET /api/waggle/stream` | `event: connected\ndata: {}` | `signal` (full `WaggleSignal` JSON) | `: heartbeat` every 30s | `routes/waggle-signals.ts` |
| `GET /api/events/stream` | `data: {"type":"connected"}` | `audit` (full `AuditEvent` JSON) | `: keepalive` every 30s | `routes/events.ts` |
| `GET /api/notifications/stream` | `data: {"type":"connected"}` | `notification` (default unnamed `data:` frame), `subagent_status`, `workflow_suggestion` | `: heartbeat` every 30s | `routes/notifications.ts` |
| `POST /v1/chat/completions` (when `stream:true`) | none | streamed OpenAI-format `data:` chunks, terminated by `data: [DONE]` | none | `routes/anthropic-proxy.ts` |
**Notes for the frontend:**
- On `/api/notifications/stream`, `notification` events arrive as **unnamed** `data:` frames (use `eventSource.onmessage`); `subagent_status` and `workflow_suggestion` arrive as **named** events (use `addEventListener('subagent_status', ...)`).
- The SSE streams set `Access-Control-Allow-Origin` only when the request `Origin` passes the same exact-match allowlist the CORS plugin uses (`corsOriginAllowed` / `validateOrigin`). A cross-origin page cannot read these streams.
- The connection is held open server-side via `await new Promise(() => {})` (waggle/stream) or by hijacking the raw reply; closing the `EventSource` triggers `request.raw.on('close')` cleanup that removes the listener and clears the heartbeat.
---
## 2. WaggleDance signals — UI stream (`/api/waggle/*`)
Source: `routes/waggle-signals.ts`. In-memory store of the last **500** signals (newest-first `unshift`, capped). This is the legacy/high-level signal shape the existing `WaggleDanceApp` UI renders.
### `WaggleSignal` shape (the JSON the UI receives)
| Field | Type | Notes |
|---|---|---|
| `id` | `string` | `sig-<epochMs>-<rand4>` |
| `type` | `string` | e.g. `agent:started`, `tool:called`, `memory:saved`, `agent:completed`, `agent:spawned`, `agent:error`, or bridged `waggle-dance:<category>` |
| `workspaceId` | `string` | defaults to `'global'` if unset |
| `content` | `string` | human-readable primary text |
| `metadata` | `Record<string,unknown>` (optional) | arbitrary provenance |
| `timestamp` | `string` | ISO-8601 |
| `acknowledged` | `boolean` | toggled by the ack PATCH |
### Endpoints
| Method | Path | Purpose |
|---|---|---|
| `GET` | `/api/waggle/signals` | List recent signals. Query: `limit` (default 50, max 200), `unacked=1` (only unacknowledged). Returns `{ signals, total }`. |
| `POST` | `/api/waggle/signals` | Publish a signal. Body `{ type, content, workspaceId?, metadata? }` (`type` + `content` required). Returns 201 with the full signal. |
| `PATCH` | `/api/waggle/signals/:id/ack` | Mark a signal acknowledged. Returns `{ acknowledged: true, id }`. |
| `GET` | `/api/waggle/stream` | **SSE.** Stream of new signals (see §1). |
`emitWaggleSignal(...)` is the internal publish helper called from the chat loop and `fleet.ts` spawn flow.
---
## 3. WaggleDance v2 protocol bus (`/api/waggle-dance/*`)
Source: `routes/waggle-dance.ts` + `signal-bus.ts` + `waggle-dance-bridge.ts`. This is the AI-OS Phase 1B cross-tool activity bus. It uses the **protocol** message shape (`WaggleMessage`), not the UI signal shape. A `SignalBus` ring buffer (default capacity **500**, drops oldest) backs it, decorated onto the Fastify instance as `server.signalBus`.
### `WaggleMessage` shape (`packages/shared/src/types.ts`)
| Field | Type | Notes |
|---|---|---|
| `id` | `string` | auto-filled `randomUUID()` |
| `teamId` | `string` | defaults to `personal::<senderId>` when `teamId` omitted — this is the personal-tier moat |
| `senderId` | `string` | defaults to `'local'` |
| `type` | `'broadcast' \| 'request' \| 'response'` | |
| `subtype` | `MessageSubtype` (10 values, see below) | |
| `content` | `Record<string, unknown>` | |
| `referenceId` | `string \| null` | |
| `routing` | `Array<{ userId; reason }> \| null` | |
| `createdAt` | `Date` | auto-filled |
### Valid `type` → `subtype` combinations (`packages/waggle-dance/src/protocol.ts`)
The POST endpoint **rejects** (400) any combo not in this table:
| `type` | Allowed `subtype` values |
|---|---|
| `request` | `knowledge_check`, `task_delegation`, `skill_request`, `model_recommendation` |
| `response` | `knowledge_match`, `task_claim` |
| `broadcast` | `discovery`, `routed_share`, `skill_share`, `model_recipe` |
### Endpoints
| Method | Path | Purpose |
|---|---|---|
| `POST` | `/api/waggle-dance/signal` | Normalize + validate + dispatch a v2 signal. Body `{ type, subtype, content, senderId?, teamId?, referenceId?, routing? }`. Returns **201** `{ dispatched: true, response, message }`; **400** on validation/combo/dispatch failure. |
| `GET` | `/api/waggle-dance/signals` | Snapshot of the ring buffer, newest-first. Query: `subtype?`, `tool?`, `teamId?`, `limit?` (max 1000), `since?` (ISO). Returns `{ signals, total }`. |
### The bridge (zero-frontend-change cross-tool activity)
`installWaggleDanceBridge(bus)` subscribes the v2 bus and re-emits every message into the legacy `/api/waggle/signals` stream as a `waggle-dance:<category>` signal, so the existing UI surfaces cross-tool activity with no changes. The 10 protocol subtypes collapse to **5 UI categories**:
| v2 subtype(s) | → UI category |
|---|---|
| `discovery`, `knowledge_check`, `skill_request` | `discovery` |
| `task_delegation`, `skill_share`, `routed_share` | `handoff` |
| `knowledge_match` | `insight` |
| `task_claim`, `model_recipe`, `model_recommendation` | `coordination` |
| any with `content.priority === 'critical'` | `alert` (overrides the above) |
Bridged signal `metadata` preserves `{ subtype, senderId, tool, teamId, referenceId, routing, priority, protocolMessage }`.
---
## 4. Audit events (`/api/events`)
Source: `routes/events.ts`. Backed by a **separate `audit.db` SQLite database** in `dataDir` (WAL mode), table `audit_events`. Default retention **90 days** (cron cleanup). This is the full audit trail for tool calls, memory ops, workspace changes, approvals, and exports.
### `AuditEventType` (15 values)
`tool_call`, `tool_result`, `memory_write`, `memory_delete`, `workspace_create`, `workspace_update`, `workspace_delete`, `session_start`, `session_end`, `approval_requested`, `approval_granted`, `approval_denied`, `approval_auto`, `export`, `cron_trigger`, `data_erase_requested`.
### `AuditEvent` shape (camelCased in responses by `normalizeEvent`)
`id`, `timestamp` (ISO), `workspaceId`, `userId?`, `eventType`, `toolName?`, `input?` (parsed JSON), `output?` (parsed JSON), `model?`, `tokensUsed?`, `cost?`, `sessionId?`, `approved?` (boolean).
### Endpoints
| Method | Path | Purpose |
|---|---|---|
| `GET` | `/api/events` | Paginated, filterable listing. Query: `workspaceId`/`workspace`, `type`/`eventType`, `from`, `to`, `sessionId`, `limit` (default 100, max 1000), `offset`. Returns `{ events, total, limit, offset, hasMore, page, totalPages }`. |
| `GET` | `/api/events/stats` | Aggregates. Query: `workspaceId`/`workspace`, `days` (default 30, max 365). Returns `{ totalEvents, period:{days,since}, byType, byDay, topTools }`. |
| `GET` | `/api/events/stream` | **SSE.** Live audit events (see §1). |
---
## 5. Cron schedules (`/api/cron`)
Source: `routes/cron.ts`, backed by `server.cronStore` (Wave 1.1 Solo Cron Service). Execution runs through `server.scheduler.executeJob`.
### Schedule response shape (camelCased from DB snake_case)
`id` (number), `name`, `cronExpr`, `jobType` (`CronJobType`), `jobConfig` (object; corrupt rows degrade to `{}`), `workspaceId`, `enabled` (boolean), `lastRunAt`, `nextRunAt`, `createdAt`.
### Endpoints
| Method | Path | Purpose |
|---|---|---|
| `POST` | `/api/cron` | Create. Body `{ name, cronExpr, jobType, jobConfig?, workspaceId?, enabled? }` (first three required). `workspaceId:'global'` is normalized to `'*'`. |
| `GET` | `/api/cron` | List all. Returns `{ schedules, count }`. |
| `GET` | `/api/cron/:id` | Get one. 404 if not found, 400 if id non-numeric. |
| `PATCH` | `/api/cron/:id` | Update. Body any of `{ name, cronExpr, jobConfig, workspaceId, enabled }`. |
| `DELETE` | `/api/cron/:id` | Delete. Returns `{ ok: true, id }`. |
| `POST` | `/api/cron/:id/trigger` | Manually run now. **Auto-enables a disabled job before executing.** Returns `{ triggered, id, nextRunAt, autoEnabled, schedule }` and emits a `cron`-category notification on success. |
| `GET` | `/api/cron/:id/history` | (registered in `notifications.ts`) Execution history. Query `limit` (default 20). Returns `{ history, count }`. 503 if unavailable. |
---
## 6. Notifications (`/api/notifications`)
Source: `routes/notifications.ts`. Two surfaces: the live SSE stream and a persisted-notification REST store (`cronStore` persists them so they survive restart).
### `NotificationEvent` shape
`type:'notification'`, `title`, `body`, `category` (`'cron' \| 'approval' \| 'task' \| 'message' \| 'agent'`), `timestamp` (ISO), `actionUrl?`.
The stream also relays two other event shapes:
- **`subagent_status`** — `{ type, workspaceId, agents:[{ id, name, role, status:'pending'|'running'|'done'|'failed', task, toolsUsed, startedAt?, completedAt? }], timestamp }`.
- **`workflow_suggestion`** — `{ type, workspaceId, pattern:{ name, description, steps, tools, category }, reason, timestamp }`.
### Endpoints
| Method | Path | Purpose |
|---|---|---|
| `GET` | `/api/notifications/stream` | **SSE.** Live notifications + subagent status + workflow suggestions (see §1). |
| `GET` | `/api/notifications` | List persisted notifications. Query `since`, `limit` (default 50), `unread=true`. Returns `{ notifications, count, unread }`. |
| `POST` | `/api/notifications/:id/read` | Mark one read. Returns `{ read: true, id }`. |
| `GET` | `/api/notifications/history` | Alias of list with `limit` default 100. Returns `{ notifications, count, unread }`. |
| `PATCH` | `/api/notifications/:id/read` | Mark one read (PATCH variant). |
| `POST` | `/api/notifications/read-all` | Mark all read. Returns `{ markedRead: count }`. |
---
## 7. Offline mode (`/api/offline`)
Source: `routes/offline.ts`, backed by `server.offlineManager`. Queues user messages while the connection is down. If the manager is absent, status returns `{ offline:false, since:null, queuedMessages:0, lastCheck }` and queue ops return 503/empty.
| Method | Path | Purpose |
|---|---|---|
| `GET` | `/api/offline/status` | Current offline state `{ ...mgr.state, lastCheck }`. |
| `POST` | `/api/offline/queue` | Queue a message. Body `{ message, workspaceId?/workspace? }` (`message` required; workspace defaults `'default'`). Returns `{ queued }`. |
| `GET` | `/api/offline/queue` | List queued messages `{ messages }`. |
| `DELETE` | `/api/offline/queue/:id` | Remove one. 404 if not found. Returns `{ removed: true }`. |
| `DELETE` | `/api/offline/queue` | Clear all. Returns `{ cleared: <n> }`. |
---
## 8. Backup & restore (`/api/backup`, `/api/restore`)
Source: `routes/backup.ts`. Produces a single encrypted archive of the `~/.waggle/` data dir for machine migration. Format: gzipped JSON manifest, AES-256-GCM encrypted when a `.vault-key` exists, with magic header `WAGGLE-BACKUP-V1`. Extension `.waggle-backup`. **Max 500 MB**; excludes `node_modules`, `.git`, `models` (re-downloadable ONNX weights), and `marketplace.db*`.
| Method | Path | Purpose |
|---|---|---|
| `POST` | `/api/backup` | Build + stream the archive as `application/octet-stream` (`Content-Disposition: attachment; filename="waggle-backup-<date>.waggle-backup"`). Response headers `X-Waggle-Backup-Encrypted` and `X-Waggle-Backup-Files`. 413 if over 500 MB; 400 if no files. |
| `POST` | `/api/restore` | Restore from an archive. Body `{ backup: <base64>, preview? }`. `preview:true` returns `{ preview, backupCreatedAt, totalFiles, existingFiles, newFiles, conflicts }` without writing. Apply returns `{ restored, filesRestored, totalFiles, conflicts, errors?, backupCreatedAt }`. Path-traversal protected; `marketplace.db` is skipped (re-syncs on startup). |
| `GET` | `/api/backup/metadata` | Last backup info `{ lastBackupAt, sizeBytes, fileCount }`. 404 if none. |
---
## 9. Agent fleet (`/api/fleet`)
Source: `routes/fleet.ts`. Mission-Control surface over `server.sessionManager` workspace sessions. Spawning is free for all tiers (agents generate memory). `maxSessions` is tier-gated: FREE=3, PRO=10, TEAMS=25, ENTERPRISE/TRIAL=100.
### Session shape (in `GET /api/fleet`)
`workspaceId`, `workspaceName`, `personaId`, `model`, `status`, `lastActivity`, `durationMs`, `toolCount`, `tokensUsed`, `costEstimate`.
| Method | Path | Purpose |
|---|---|---|
| `GET` | `/api/fleet` | List active workspace sessions. Returns `{ sessions, count, maxSessions }`. |
| `POST` | `/api/fleet/spawn` | Spawn a new agent session. Body `{ task, persona?, model?, parentWorkspaceId? }` (`task` required). Creates/uses the workspace session, emits `agent:spawned` then fire-and-forget runs the agent loop emitting `agent:started`/`tool:called`/`agent:completed`/`agent:error` signals. Returns `{ id, workspaceId, sessionId, status, startedAt, task, persona, model }`. 404 if workspace has no mind; 409 on spawn failure. |
| `POST` | `/api/fleet/:workspaceId/pause` | Pause a session. 404 if not found/already paused. |
| `POST` | `/api/fleet/:workspaceId/resume` | Resume a paused session. 404 if not found/not paused. |
| `POST` | `/api/fleet/:workspaceId/kill` | Abort + close a session. 404 if not found. |
---
## 10. LiteLLM control (`/api/litellm`)
Source: `routes/litellm.ts`. Manages the optional LiteLLM router process (lifecycle helpers `getLiteLLMStatus`/`startLiteLLM`/`stopLiteLLM`) and proxies its model list.
| Method | Path | Purpose |
|---|---|---|
| `GET` | `/api/litellm/status` | `{ running, port, error? }`. |
| `POST` | `/api/litellm/restart` | Stop then start. `{ running, port, error? }`. |
| `GET` | `/api/litellm/models` | Available model IDs from LiteLLM `{ models: string[] }` (empty array on failure). |
| `GET` | `/api/litellm/pricing` | Static per-model pricing array `[{ model, inputPer1k, outputPer1k, provider }]` (claude-sonnet/haiku/opus-4-6, gpt-5.4(+mini), gemini-3.1-pro/flash). |
---
## 11. Local inference (`/api/local-inference`)
Source: `routes/local-inference.ts`. Hardware detection + local model recommendations via the `llmfit` CLI (falls back to OS/RAM basics), plus Ollama/vLLM availability and model pulls. Ollama defaults to `http://localhost:11434`, vLLM to `http://localhost:8000` (overridable via `OLLAMA_HOST`/`VLLM_HOST`).
| Method | Path | Purpose |
|---|---|---|
| `GET` | `/api/local-inference/hardware` | Detect GPU/RAM/CPU. `{ hardware:<HardwareInfo>, source:'llmfit'|'basic', llmfitAvailable }`. |
| `GET` | `/api/local-inference/models` | Recommend models that fit. Query `useCase?`, `limit?` (default 20). `{ models:<ModelRecommendation[]>, source, totalScanned }`. |
| `GET` | `/api/local-inference/status` | Ollama/vLLM availability + installed models. `{ servers, primaryServer, ollamaInstalled, ollamaUrl, vllmUrl, totalLocalModels }`. |
| `POST` | `/api/local-inference/pull` | Pull a model via Ollama. Body `{ model }`. 502 if Ollama unreachable; up to 10-min timeout. |
`HardwareInfo` includes `totalRamGb, availableRamGb, cpuCores, cpuName, platform, hasGpu, gpuName, gpuVramGb, gpuCount, gpus[], backend`. `ModelRecommendation` includes `name, provider, parameterCount, paramsB, useCase, category, fitLevel, score, scoreComponents{quality,speed,fit,context}, estimatedTps, memoryRequiredGb, memoryAvailableGb, utilizationPct, bestQuant, runMode, runtime, contextLength, isMoe, notes[]`.
---
## 12. Anthropic proxy (`/v1/chat/completions`)
Source: `routes/anthropic-proxy.ts`. A built-in **OpenAI-compatible** proxy backed by the Anthropic Messages API — replaces LiteLLM when calling Anthropic models directly. The API key is read from the **vault** first (key `anthropic`), then `ANTHROPIC_API_KEY` env, then `config.json`. Applies Anthropic prompt caching (system + rolling last-3-message window).
| Method | Path | Purpose |
|---|---|---|
| `GET` | `/v1/health/liveliness` | Always `{ status: 'healthy' }` (built-in, never down). |
| `POST` | `/v1/chat/completions` | OpenAI chat-completions body `{ model, messages, tools?, stream?, stream_options?, max_tokens?, temperature? }`. Translates to/from Anthropic. **SSE when `stream:true`** (OpenAI-format chunks, terminated by `data: [DONE]`); otherwise a single OpenAI `{ choices, usage, model }` JSON. 500 if no API key. |
Model names are normalized in `mapModel` (strips provider prefix, dots→dashes; Haiku 4.6 typos map to the valid Haiku 4.5 snapshot; Sonnet/Opus 4.6 pass through as floating aliases).
---
## 13. Filesystem browse (`/api/browse`)
Source: `routes/browse.ts`. System-level directory browsing (not workspace-scoped) for the Create-Workspace dialog. **Local-only** — both endpoints reject external origins with 403 via `isLocalRequest`. On Windows, abstract root `/` returns the full drive list.
| Method | Path | Purpose |
|---|---|---|
| `GET` | `/api/browse/local` | List directories. Query `path` (default `/`). Returns `{ entries:[{name,path,type:'directory'}], current }`. Skips hidden + non-dir entries. 403/404/400/403(EACCES). |
| `POST` | `/api/browse/local/mkdir` | Create a directory (recursive). Body `{ path }`. Returns 201 `{ name, path, type:'directory' }`. |
---
## 14. Browser extension health (`/api/browser-ext`)
Source: `routes/browser-ext.ts`. Single health check for the `apps/browser-ext` Chrome MV3 extension. Ingest/ask flows reuse `/api/memory/frames` and `/api/chat`.
| Method | Path | Purpose |
|---|---|---|
| `GET` | `/api/browser-ext/health` | `{ ok: true, version: '0.1.0', activeWorkspace }` so the extension can show "saving to: &lt;workspace&gt;". |
---
## 15. Telegram outbound push (`/api/telegram`)
Source: `routes/telegram.ts`. One-way push from Waggle to a user's Telegram (no webhook receiver). Bot token + chat_id stored in the **vault** (`telegram_bot_token`, `telegram_chat_id`). URL is hard-coded to `api.telegram.org` (no SSRF). Text capped at 4096 chars. `pushTelegramMessage(server, text)` is the internal hook cron jobs call.
| Method | Path | Purpose |
|---|---|---|
| `GET` | `/api/telegram/status` | `{ configured, hasToken, hasChatId }`. |
| `POST` | `/api/telegram/config` | Save creds. Body `{ botToken?, chatId? }`. Validates `botToken` against `<id>:<secret>` pattern and `chatId` as a signed integer string; 400 on bad format. Returns `{ ok: true }`. |
| `POST` | `/api/telegram/test` | Send a "Waggle is connected" test message. 400 if not configured; 502 on Telegram error. Returns `{ ok, messageId }`. |
| `POST` | `/api/telegram/send` | Send arbitrary text. Body `{ text, parseMode? }` (`text` required, ≤4096). 400/502. Returns `{ ok, messageId }`. |
---
## 16. How it connects (data flow)
```mermaid
flowchart TD
subgraph Agent["Agent runtime / chat loop / fleet spawn"]
AL[runAgentLoop]
end
subgraph V2["WaggleDance v2 (protocol)"]
POSTv2["POST /api/waggle-dance/signal"]
DISP[WaggleDanceDispatcher]
BUS[(SignalBus ring buffer 500)]
GETv2["GET /api/waggle-dance/signals"]
BRIDGE[installWaggleDanceBridge]
end
subgraph UI["WaggleDance UI stream (legacy)"]
EMIT[emitWaggleSignal]
STORE[(signals[] 500)]
GETs["GET /api/waggle/signals"]
SSEs["GET /api/waggle/stream (SSE)"]
end
subgraph Ops["Ops subsystems"]
AUDIT[(audit.db SQLite)]
SSEev["GET /api/events/stream (SSE)"]
EB[(eventBus EventEmitter)]
SSEnotif["GET /api/notifications/stream (SSE)"]
CRON[cronStore + scheduler]
end
Frontend["Frontend (EventSource + fetch)"]
AL -->|emitWaggleSignal| EMIT
POSTv2 --> DISP --> BUS
BUS --> GETv2
BUS -->|subscribe| BRIDGE -->|waggle-dance:category| EMIT
EMIT --> STORE --> GETs
EMIT -->|EventEmitter signal| SSEs --> Frontend
GETs --> Frontend
GETv2 --> Frontend
AL -->|emitAuditEvent| AUDIT
AUDIT -->|eventBus audit_event| EB --> SSEev --> Frontend
CRON -->|emitNotification| EB
EB -->|notification / subagent_status / workflow_suggestion| SSEnotif --> Frontend
CRON -->|saveNotification| AUDIT
```
**Key wiring facts:**
- Two parallel signal systems exist: the **legacy UI store** (`signals[]`, `EventEmitter`, `/api/waggle/*`) and the **v2 protocol bus** (`SignalBus`, `/api/waggle-dance/*`). The **bridge** is one-directional: v2 → legacy. The frontend only needs to read `/api/waggle/stream` to see both.
- `server.eventBus` (a Node `EventEmitter`, max 50 listeners) is the shared relay for audit, notification, subagent-status, and workflow-suggestion SSE.
- Persistence: audit events → `audit.db`; notifications + cron history → `cronStore`; v2 signals + UI signals are **in-memory only** (lost on sidecar restart).

294
docs/backend-map/sections/03g-api-cloud-billing-kvark.md Normal file
View File

@@ -0,0 +1,294 @@
# 03g — API: Cloud / Multi-Tenant Routes, Stripe Billing, KVARK, WebSockets
## Purpose
This section documents the **two distinct HTTP surfaces** the Waggle backend exposes, the **Stripe billing flow** (checkout → webhook/sync → tier resolution), the **KVARK enterprise client** (tier-gated, vault-credentialed), and the **WebSocket** real-time channels. Crucially, it nails down **how the frontend discovers the API root** and what auth/origin guards every request must satisfy. For a Lovable rebuild, treat the endpoint tables below as the binding contract.
---
## 1. Two Servers — Know Which One You Are Talking To
Waggle ships **two separate Fastify servers**. They do NOT share a port, an auth model, or a database. The frontend you are rebuilding almost always talks to the **Local Sidecar**, not the Cloud server.
| | **Local Sidecar** | **Cloud / Multi-Tenant Server** |
|---|---|---|
| Entry file | `packages/server/src/local/index.ts` → `buildLocalServer()` | `packages/server/src/index.ts` → `buildServer()` |
| Started by | `packages/server/src/local/start.ts` → `service.ts:startService()` | run `index.ts` directly |
| Default port | **3333** (`WAGGLE_PORT`, const `DEFAULT_PORT = 3333`) | **3100** (`PORT`) |
| Default host | loopback (`resolveBindHost()`) | `0.0.0.0` (`HOST`) |
| Database | SQLite `.mind` files in `~/.waggle` (better-sqlite3) | Postgres (`DATABASE_URL`) + Redis (`REDIS_URL`) |
| Auth model | **Session-token Bearer** (generated at boot) + same-origin/host guards | **Clerk JWT** Bearer (`fastify.authenticate`) |
| Tier source | `~/.waggle/config.json` `{ tier }` (read by `requireTier`) | Clerk-provisioned user rows |
| Serves frontend | **Yes** — `@fastify/static` from `dist/` with SPA fallback | No |
| WebSocket | `/ws` (token in query param, event-bus relay) | `/ws` (Clerk-auth message, team chat + Redis) |
| Stripe routes | **Yes** (`stripeRoutes` registered) | No |
| KVARK client | Used by agent tools via vault config | No |
> The desktop binary loads `apps/web` from the sidecar's static handler, so the **frontend origin IS the sidecar** (`http://localhost:3333` or `tauri://localhost`). API calls are same-origin to the sidecar. The cloud server (3100) is only used in SaaS/team-server deployments and is referenced by the sidecar's `team` routes via `teamServerUrl`.
---
## 2. How the Frontend Discovers the API Root (Local Sidecar)
The frontend is served by the sidecar itself, so the API root is **same-origin** (the page's own origin). The bootstrap handshake:
1. **Static shell loads token-free.** Non-API `GET` requests (the SPA shell + `/assets/*`) are auth-exempt, so the webview can load app code before it has a token (`security-middleware.ts`, `isNonApiGet`).
2. **Fetch the session token.** The app calls `GET /api/auth/session-token` once on connect. This endpoint is **auth-exempt but same-origin-gated** (`isLocalRequest`). It returns `{ token }` = `server.agentState.wsSessionToken` (a random token generated at boot).
3. **Send Bearer on everything else.** Every `/api/*` route (and any non-GET) requires `Authorization: Bearer <token>`. Missing → `401 MISSING_TOKEN`; wrong → `401 INVALID_TOKEN`.
4. **WebSocket** connects to `GET /ws?token=<sessionToken>` (token in query param, not header). Wrong token → socket closed with code `4001`.
### Request guards the frontend must satisfy (Local Sidecar, every request)
| Guard | Source | Rule | Failure |
|---|---|---|---|
| Security headers | `securityMiddleware` `onRequest` | CSP, `X-Frame-Options: DENY`, `nosniff`, etc. set on every reply | n/a (response only) |
| Host allowlist (anti-DNS-rebind) | `hostHeaderAllowed()` | When loopback-bound: `Host` (port stripped) ∈ `{127.0.0.1, localhost, ::1}` + `WAGGLE_ALLOWED_HOSTS`. **Empty Host fails closed.** | `403 BAD_HOST` |
| Bearer token | `securityMiddleware` | All non-exempt routes need `Authorization: Bearer <sessionToken>`. Exempt: `OPTIONS`, `/health`, `/api/auth/session-token`, non-API GETs. `WAGGLE_TRUST_LOCALHOST=1` restores legacy loopback trust. | `401 MISSING_TOKEN` / `401 INVALID_TOKEN` |
| CORS | `corsOriginAllowed()` | **Exact-match** allowlist (`cors-config.ts`); missing origin allowed. Includes `tauri://localhost`, `https://tauri.localhost`, `localhost:1420/3333/8080/8081/8082/5173`. | CORS error |
| Same-origin gate (sensitive routes) | `isLocalRequest()` | URL-parsed origin/referer must be local (`tauri:`, `https://tauri.localhost`, or local host). No origin+referer ⇒ treated local. | `403 Forbidden: external origin` |
| Rate limit | `RateLimiter` | Sliding window, default **100 req/min** per `clientIP:method route`. Overrides: `/api/chat` 120, `/api/vault/*/reveal` 5, `/api/backup` 2, `/api/restore` 2, `/api/browse/local/mkdir` 10. | `429` + `Retry-After` |
| Session timeout (team mode only) | `SessionTimeoutTracker` | Only active when `CLERK_SECRET_KEY` set. 30 min inactivity (`WAGGLE_SESSION_TIMEOUT_MS`). Exempt: `/health`, `/api/vault`. | `401 SESSION_TIMEOUT` |
---
## 3. Local Sidecar — Full Route-Plugin Registration
`buildLocalServer()` registers **62 route plugins** plus inline endpoints. **None are mounted with a base prefix** — each route file hardcodes its own `/api/...` paths. Registration order (from `local/index.ts` lines 1974–2031):
```
workspaceRoutes chatRoutes memoryRoutes settingsRoutes
sessionRoutes knowledgeRoutes litellmRoutes ingestRoutes
mindRoutes agentRoutes skillRoutes approvalRoutes
anthropicProxyRoutes teamRoutes taskRoutes capabilitiesRoutes
toolsRoutes waggleDanceRoutes commandRoutes cronRoutes
notificationRoutes marketplaceDevRoutes marketplaceRoutes connectorRoutes
fleetRoutes importRoutes vaultRoutes personaRoutes
feedbackRoutes workspaceTemplateRoutes evolutionRoutes exportRoutes
dataEraseRoutes costRoutes backupRoutes offlineRoutes
weaverRoutes eventRoutes workflowRoutes pinRoutes
documentRoutes fileRoutes waggleSignalRoutes providerRoutes
profileRoutes oauthRoutes browseRoutes browserExtRoutes
telegramRoutes telemetryRoutes agentGroupRoutes stripeRoutes
harvestRoutes wikiRoutes identityRoutes agentRunRoutes
localInferenceRoutes complianceRoutes
```
Plus inline endpoints on the sidecar root:
| Method | Path | Purpose | Guard |
|---|---|---|---|
| GET | `/health` | Health probe (used by `/api/debug/logs`) | none |
| GET | `/api/auth/session-token` | Bootstrap session token for the webview | same-origin (`isLocalRequest`), auth-exempt |
| GET | `/api/debug/logs` | Support bundle (health + 50 audit rows) | same-origin |
| GET | `/api/docs` | Auto-generated route/OpenAPI listing | bearer |
| GET | `/ws` | WebSocket event-bus relay (approvals, agent steps, notifications) | `?token=<sessionToken>` |
| GET | `/*` (SPA fallback) | `setNotFoundHandler` serves `index.html` for non-API/non-asset routes | auth-exempt GET |
> Static frontend dir is resolved from `WAGGLE_FRONTEND_DIR` or probed (`<root>/dist`, then legacy `app/dist`). SPA fallback explicitly does NOT intercept `/api/`, `/v1/`, `/assets/`, `/health`, `/ws`.
---
## 4. Local Sidecar — Route Files Mapped (the four requested + key contracts)
All paths below are **bearer-gated** (`Authorization: Bearer <sessionToken>`) at the middleware layer unless noted. These route files (`agents.ts`, `jobs.ts`, `scout.ts`, `suggestions.ts`) live in the **Cloud** server (`packages/server/src/routes/`) and use **Clerk** `fastify.authenticate` — they are NOT in the sidecar. The sidecar's agent surface is `routes/agent.ts` + `routes/agent-groups.ts` + `routes/agent-run.ts`. Both are documented here because the prompt named the cloud files.
### 4a. Cloud `routes/agents.ts` — sub-agent definitions & groups (Clerk-auth)
| Method | Path | Purpose | Notes |
|---|---|---|---|
| POST | `/api/agents` | Create sub-agent definition | body `createAgentSchema`; 201 |
| GET | `/api/agents` | List user's agents | scoped to `request.userId` |
| PATCH | `/api/agents/:id` | Update agent config | 403 if not owner |
| DELETE | `/api/agents/:id` | Delete agent | 403 if not owner; 204 |
| POST | `/api/agent-groups` | Create group w/ members | body `createAgentGroupSchema`; 201 |
| GET | `/api/agent-groups` | List groups | |
| GET | `/api/agent-groups/:id` | Get group + members | 404 if not found |
| PATCH | `/api/agent-groups/:id` | Update name/description/strategy/members | |
| DELETE | `/api/agent-groups/:id` | Delete group + members | 204 |
| POST | `/api/agent-groups/:id/run` | Execute group on a `{ task, teamId? }` | builds workflow; returns `202 { jobId?, workflow{name,steps,strategy,aggregation}, message }` |
### 4b. Cloud `routes/jobs.ts` — async job queue (Clerk-auth, Redis-backed)
| Method | Path | Purpose | Notes |
|---|---|---|---|
| GET | `/api/jobs?teamSlug=&limit=` | List jobs for a team | 400 if no `teamSlug`; 403 if not a member; default limit 50 |
| POST | `/api/jobs` | Queue a job | body `queueJobSchema` `{ teamId?, jobType, input }`; returns `202 { jobId, status }` |
| POST | `/api/jobs/:id/cancel` | Cancel queued/running job | 409 if status not queued/running; `{ cancelled, jobId }` |
| GET | `/api/jobs/:id` | Get job status | 404 if not found |
### 4c. Cloud `routes/scout.ts` — proactive findings (Clerk-auth)
| Method | Path | Purpose | Notes |
|---|---|---|---|
| GET | `/api/scout/findings` | List findings for user | via `ScoutAgent.listFindings` |
| PATCH | `/api/scout/findings/:id` | Adopt or dismiss | body `{ status: 'adopted' \| 'dismissed' }`; 400 otherwise; 404 if not found |
### 4d. Cloud `routes/suggestions.ts` — proactive suggestions (Clerk-auth)
| Method | Path | Purpose | Notes |
|---|---|---|---|
| GET | `/api/suggestions` | List pending suggestions | via `ProactiveService.listPending` |
| PATCH | `/api/suggestions/:id` | Accept/dismiss/snooze | body `{ status: 'accepted' \| 'dismissed' \| 'snoozed' }`; 400 otherwise; 404 if not found |
---
## 5. Stripe Billing (Local Sidecar — `stripeRoutes`)
All Stripe routes are gated behind `STRIPE_SECRET_KEY`. When unset, every route returns **`503 STRIPE_NOT_CONFIGURED`**. The Stripe SDK is lazily, dynamically required (`getStripe()`, apiVersion `2025-03-31.basil`) so `stripe` is not a hard dependency.
### Endpoints
| Method | Path | Body | Returns | Guard / Notes |
|---|---|---|---|---|
| POST | `/api/stripe/create-checkout-session` | `{ tier: 'PRO'\|'TEAMS', billingPeriod?: 'monthly'\|'annual' }` | `{ url }` | 400 `INVALID_TIER` if not PRO/TEAMS; 400 `NO_PRICE_CONFIGURED` if no price env; 502 `STRIPE_ERROR`. `mode: 'subscription'`, `allow_promotion_codes: true`, metadata carries `tier`+`billingPeriod`. success → `{origin}/payment-success?session_id={CHECKOUT_SESSION_ID}`, cancel → `{origin}/payment-cancelled`. |
| POST | `/api/stripe/webhook` | raw Stripe event (buffer) | `{ received: true }` or `{ received, duplicate: true }` | Validates `stripe-signature` (needs `STRIPE_WEBHOOK_SECRET`). 400 `MISSING_SIGNATURE`/`INVALID_SIGNATURE`. Idempotent (last 500 event IDs), serialized critical section. Uses **raw-body content-type parser**. |
| POST | `/api/stripe/sync` | `{ sessionId }` | `{ tier, customerId }` | Poll fallback for NAT'd desktop apps. 402 `PAYMENT_NOT_COMPLETED` if unpaid; 400 `TIER_NOT_RESOLVED`; 502 `STRIPE_ERROR`. |
| POST | `/api/stripe/create-portal-session` | — | `{ url }` | **`preHandler: requireTier('PRO')`**. Reads `stripe_customer_id` from `config.json`; 400 `NO_STRIPE_CUSTOMER` if none. return → `{origin}/settings`. |
### Tier ↔ Price resolution (`stripe/index.ts`)
`tierFromPriceId(priceId): Tier | null` — synchronous, **offline** (no Stripe round-trip). Resolves a price ID to a tier by matching env vars, **new 4-var contract checked first, legacy fallbacks after**:
| Tier | Env vars checked (in order) |
|---|---|
| `PRO` | `STRIPE_PRICE_PRO_MONTHLY`, `STRIPE_PRICE_PRO_ANNUAL`, `STRIPE_PRICE_PRO`, `STRIPE_PRICE_BASIC` |
| `TEAMS` | `STRIPE_PRICE_TEAMS_MONTHLY`, `STRIPE_PRICE_TEAMS_ANNUAL`, `STRIPE_PRICE_TEAMS` |
`priceIdForTier(tier, billingPeriod='monthly')` — the inverse, used by checkout. Prefers the period-specific 4-var, falls back to legacy single-var, then `TIER_CAPABILITIES[tier].stripePriceId`.
### Webhook event handling → writes `config.json` `{ tier, stripe_customer_id }`
| Stripe event | Action |
|---|---|
| `checkout.session.completed` | Grant tier from `metadata.tier` **only if `payment_status ∈ {paid, no_payment_required}`** |
| `customer.subscription.updated` | Resolve tier via `tierFromPriceId(items[0].price.id)`, update |
| `customer.subscription.deleted` | Downgrade to `FREE` |
Tier writes use `atomicWriteJson` (temp + rename) and a module-scoped promise queue (`serializeWebhook`) to prevent TOCTOU double-processing on Stripe retries. The same `config.json` is read by `requireTier`/`readTierFromRequest` and the portal route. For the frontend: after a checkout redirect, **call `POST /api/stripe/sync` with the `session_id`** to confirm payment locally (webhooks are unreliable behind NAT).
---
## 6. KVARK Client (`packages/server/src/kvark/*`) — Enterprise, Tier-Gated
KVARK is Egzakta's sovereign enterprise platform. Waggle never calls KVARK directly — **everything flows through `KvarkClient`**, which is the single boundary. KVARK tools are registered only when KVARK is configured (TEAMS/ENTERPRISE tiers, per `kvark-tools.ts`). There are **no Fastify routes** for KVARK; it is consumed internally by agent tools.
### Config & auth
- **Credentials live in the vault** as `kvark:connection` → JSON `{ baseUrl, identifier, password, timeoutMs? }` (`getKvarkConfig(vault)`; returns `null` if unset/invalid).
- **`KvarkAuth`** manages the JWT lifecycle: `POST {baseUrl}/api/auth/login` with `{ identifier, password }`, caches the Bearer token in memory, auto-relogins on 401. `KvarkLoginResponse = { success, access_token, token_type, user, error }`.
### `KvarkClient` public methods → KVARK API (FastAPI backend)
| Method | KVARK endpoint | Returns | Notes |
|---|---|---|---|
| `search(query, {limit?, offset?})` | `GET /api/search?q=&limit=&offset=` | `KvarkSearchResponse { results[], total, query }` | Waggle never re-ranks results |
| `askDocument(documentId, question)` | `POST /api/chat/ask` | `KvarkAskResponse { answer, sources[] }` | KVARK side currently stubbed 501; handled gracefully |
| `feedback(documentId, query, useful, reason?)` | `POST /api/feedback` | `KvarkFeedbackResponse { ok, data{stored, feedbackId?}, error }` | fire-and-ack |
| `action(actionType, target, payload, reason, approvalReference?, workspaceId?)` | `POST /api/actions` | `KvarkActionResponse { ok, data{status,actionId?,auditRef?,result?}, error }` | governed; requires `userApproved` |
| `ping()` | `GET /api/auth/me` | `KvarkUser` | connectivity/auth check |
### KVARK resilience contract (`kvark-client.ts request()`)
- **401** → invalidate token, re-login once, retry; second 401 → `KvarkAuthError`.
- **429** → exponential backoff (honors numeric retry hint), up to `MAX_RETRIES = 3`, then `KvarkServerError(429)`.
- **5xx** (except 501) → backoff retry up to 3 (when `retryOnServerError`, default true).
- Typed errors: `KvarkAuthError`, `KvarkNotFoundError` (404), `KvarkNotImplementedError` (501), `KvarkServerError` (403/429/5xx, carries `statusCode`), `KvarkUnavailableError` (network/timeout, default 30s).
### Key KVARK DTO shapes (the Waggle↔KVARK contract)
| Type | Fields |
|---|---|
| `KvarkUser` | `id:number, identifier, first_name\|null, last_name\|null, admin:boolean, developer:boolean, status\|null, created_at\|null` |
| `KvarkSearchResult` | `document_id:number, title, snippet, score:number, document_type\|null` |
| `KvarkChatEvent` (SSE) | discriminated union: `status \| token \| tool_call \| tool_result \| thought \| done \| error` |
| `KvarkTokenUsage` | `input_tokens, output_tokens, latency_ms` |
---
## 7. WebSocket Channels
There are **two different `/ws` implementations** — one per server.
### 7a. Local Sidecar `/ws` (`local/index.ts`) — event-bus relay
- **Auth:** `?token=<sessionToken>` query param; wrong → close code `4001`.
- **Server → client** (forwarded from the in-process `eventBus`): events `approval_required`, `step`, `tool`, `done`, `error`, `presence_update`, `notification`. Frame shape: `{ event, data }`.
- **Client → server:** `{ type: 'approve'|'deny', requestId }` resolves a `pendingApprovals` entry (the human-in-the-loop tool-gate).
- Listeners are per-connection (clean removal on close — one client disconnecting does not kill others).
### 7b. Cloud `/ws` (`ws/gateway.ts`) — team chat (Clerk + Redis)
- **Auth:** client sends `{ type: 'authenticate', token }`. Token must be **JWT-structured** (`isJwtStructure`) and verified via Clerk `verifyToken` (or test override). In **production with no Clerk key, the gateway refuses to start**; in dev/desktop it warns and rejects connections. Maps Clerk `sub` → internal user; replies `{ type: 'authenticated', userId }`.
- **Client events:** `authenticate`, `join_team` (`{ teamSlug }` → subscribes Redis channel `team:<id>:waggle`, replies `joined_team`), `send_message` (`{ messageType, subtype, content }` → persists to `messages` table + publishes to Redis).
- **Fan-out:** `ConnectionManager` (`ws/connection-manager.ts`) maps `teamId → userId → WebSocket`. Methods: `add`, `remove`, `broadcast(teamId, event, excludeUserId?)`, `sendTo`, `getConnectedUsers`, `getTeamCount`.
- **Redis bridge:** subscribes `team:*:waggle` (→ `waggle_message` broadcast) and pattern `job:*:progress` (→ `job_progress` broadcast routed by `parsed.teamId`). This is how multi-process job progress reaches clients.
---
## 8. Cloud Server Route Registration (`packages/server/src/index.ts`)
`buildServer()` registers (no prefixes; each route file hardcodes `/api/...`), in order: `cors` (origin from `CORS_ORIGIN` env, fail-closed in prod), `websocket`, `redisPlugin`, **`authPlugin`** (decorates `fastify.authenticate` = Clerk verify + auto-provision user from JWT), then route plugins: `webhookRoutes`, `teamRoutes`, `agentRoutes`, `taskRoutes`, `messageRoutes`, `knowledgeRoutes`, `resourceRoutes`, `jobRoutes`, `cronRoutes`, `suggestionRoutes`, `scoutRoutes`, `auditRoutes`, `capabilityGovernanceRoutes`, `analyticsRoutes`, `wsGateway`. Plus inline `GET /health`. Decorators: `config`, `db`, `jobService`.
`authPlugin` self-heals: on a valid Clerk JWT for an unknown user it calls `clerk.users.getUser` and `upsertFromClerk` (works even if the Clerk webhook was missed). Sets `request.userId` (internal UUID) + `request.clerkId`.
---
## 9. Connection Diagram
```mermaid
flowchart TD
subgraph Desktop["Desktop / Web (apps/web)"]
UI["Frontend SPA<br/>(served by sidecar /dist)"]
end
subgraph Sidecar["Local Sidecar :3333 (buildLocalServer)"]
SEC["securityMiddleware<br/>Host allowlist · Bearer · RateLimit · CSP"]
TOK["/api/auth/session-token<br/>(same-origin, auth-exempt)"]
WSL["/ws?token=…<br/>event-bus relay"]
STRIPE["stripeRoutes<br/>checkout · webhook · sync · portal"]
ROUTES["62 route plugins<br/>(/api/* hardcoded paths)"]
CFG[("~/.waggle/config.json<br/>{ tier, stripe_customer_id }")]
VAULT[("Vault<br/>kvark:connection, api keys")]
end
subgraph KVARK["KVARK (sovereign, FastAPI)"]
KAPI["/api/auth/login · /api/search<br/>/api/chat/ask · /api/actions · /api/feedback"]
end
subgraph Cloud["Cloud Server :3100 (buildServer)"]
AUTHP["authPlugin (Clerk JWT)"]
CROUTES["agents · jobs · scout · suggestions<br/>teams · messages · audit · analytics"]
WSG["/ws gateway (team chat)"]
PG[("Postgres")]
REDIS[("Redis pub/sub")]
end
STRIPEAPI["Stripe API"]
UI -->|"1. load shell (auth-exempt GET)"| Sidecar
UI -->|"2. GET token"| TOK
UI -->|"3. Bearer + /api/*"| SEC --> ROUTES
UI -->|"4. WS"| WSL
ROUTES --> CFG
ROUTES --> VAULT
STRIPE -->|"create session / verify"| STRIPEAPI
STRIPEAPI -->|"webhook"| STRIPE
STRIPE -->|"write tier"| CFG
VAULT -->|"KvarkClient (TEAMS/ENTERPRISE)"| KAPI
UI -. "team mode" .-> AUTHP --> CROUTES --> PG
WSG <--> REDIS
CROUTES --> REDIS
```
---
## 10. Frontend Rebuild Checklist (must-knows)
- **API root = the page's own origin** (sidecar 3333 / `tauri://localhost`). No separate host to configure.
- **Two-step auth:** fetch `/api/auth/session-token` once, then send `Authorization: Bearer <token>` on every `/api/*` call and `/ws?token=`.
- **All `/api/*` paths are flat** — no plugin prefix; the path written in each route file IS the path.
- **After Stripe checkout redirect, call `POST /api/stripe/sync { sessionId }`** — do not rely on the webhook for desktop.
- **Tier-gated UI:** PRO+ features may return `403 TIER_INSUFFICIENT` with `{ required, actual, upgradeUrl }`; KVARK features only exist on TEAMS/ENTERPRISE.
- **Stripe unconfigured** → `503 STRIPE_NOT_CONFIGURED`; render upgrade UI defensively.
- **Rate limits** are real (chat 120/min, vault reveal 5/min, backup/restore 2/min) — handle `429` + `Retry-After`.

525
docs/backend-map/sections/04-feature-map.md Normal file
View File

@@ -0,0 +1,525 @@
# 04 — Frontend Feature → API Map
**Purpose.** This section is the contract a Lovable rebuild must reproduce. It enumerates every OS app, overlay, and page in the Waggle web client (`apps/web/src/`), the single HTTP/SSE/WS client (`adapter`) they all share, the global providers/state, and — for each feature — the exact backend endpoints it calls. Everything below is grounded in the actual code; identifiers, paths, and field names are quoted verbatim from source.
---
## 1. Mental Model
Waggle's web UI is a **single-page "desktop OS"**, not a multi-page app. React Router (`apps/web/src/App.tsx`) defines only two routes:
| Path | Element | File |
|---|---|---|
| `/` | `<Index />` | `apps/web/src/pages/Index.tsx` |
| `*` | `<NotFound />` | `apps/web/src/pages/NotFound.tsx` |
`Index.tsx` renders a `BootScreen` (gated by `localStorage["waggle-booted"]`) and then `<Desktop />` (`apps/web/src/components/os/Desktop.tsx`). **`Desktop.tsx` is the real shell**: it owns the window manager, the dock, all app windows, and all overlays. There is no per-app routing — apps are opened as draggable windows by `appId`.
```mermaid
flowchart TD
App["App.tsx (BrowserRouter)"] --> Index["pages/Index.tsx"]
Index --> Boot["BootScreen"]
Index --> Desktop["components/os/Desktop.tsx<br/>(the OS shell)"]
Desktop --> WM["useWindowManager<br/>(open/close/focus windows)"]
Desktop --> Dock["Dock (dock-tiers.ts config)"]
Desktop --> AppWindows["AppWindow x N<br/>renderAppContent(win) switch on appId"]
Desktop --> Overlays["Overlays (modals/rails)"]
AppWindows --> Apps["*App.tsx components"]
Apps --> Hooks["hooks/use*.ts"]
Apps --> Adapter
Hooks --> Adapter["lib/adapter.ts<br/>(LocalAdapter singleton)"]
Adapter --> Backend["Fastify sidecar @ http://127.0.0.1:3333"]
```
Two things every Lovable rebuild MUST recreate first:
1. **The `adapter` singleton** (`apps/web/src/lib/adapter.ts`) — one `LocalAdapter` instance exported as `export const adapter = new LocalAdapter()`. Every component and hook imports this same instance. It is the only thing that talks to the backend.
2. **`ServiceProvider`** (`apps/web/src/providers/ServiceProvider.tsx`) — the only React context provider. It calls `adapter.connect()` once on mount and exposes `{ adapter, connected, connecting, error, reconnect }` via `useService()`.
---
## 2. API Client Contract (`lib/adapter.ts`)
### 2.1 Base URL & connection
| Concern | Behavior (from code) |
|---|---|
| Default server | `const DEFAULT_SERVER = 'http://127.0.0.1:3333'` |
| Base URL resolution | constructor: `serverUrl ?? localStorage.getItem('waggle:server-url') ?? DEFAULT_SERVER` |
| Change server | `adapter.setServerUrl(url)` — persists to `localStorage["waggle:server-url"]`, resets connected flags |
| Connect | `adapter.connect()` → `healthProbe()` (GET `/health`) then `fetchSessionToken()`; sets `_connected = true` |
| Auto-rediscovery | `healthProbe()` falls back to `DEFAULT_SERVER` once if the stored URL fails, and persists the working URL |
| Connection getters | `adapter.isConnected`, `adapter.hasAttemptedConnect`, `adapter.getServerUrl()` |
### 2.2 Auth / header pattern
Auth is a **bearer token fetched from a same-origin bootstrap**, not a login form:
- On `connect()`, `fetchSessionToken()` does GET `/api/auth/session-token` → `{ token }`, stored in `this.authToken`.
- Every request goes through `adapter.fetch(path, init)` which:
- Adds `Content-Type: application/json` **only when a body is present** and no content-type was supplied (a deliberate fix — bodyless POSTs must not send JSON content-type).
- Adds `Authorization: Bearer <token>` when `authToken` is set.
- On HTTP **403** with body `{ error: 'TIER_INSUFFICIENT' }`, dispatches a global `window` event `waggle:tier-insufficient` with `{ required, actual, message }` (this drives the `UpgradeModal`).
- All requests use `fetchWithTimeout` (`lib/fetch-utils.ts`, default 10s; uploads/ingest use 30s) which throws `TimeoutError` / `NetworkError`.
### 2.3 Streaming patterns
| Pattern | Method(s) | Transport |
|---|---|---|
| Chat token stream | `async *sendMessage(...)` | `POST /api/chat` returning an SSE-formatted body, parsed manually (`event:` / `data:` lines) into `StreamEvent` |
| Server-Sent Events (named/default) | `private subscribeSSE(path, onData)` | `EventSource`; used by `subscribeEvents`, `subscribeNotifications`, `subscribeWaggleDance` |
| Named SSE event | `subscribeSubagentStatus(...)` | `EventSource.addEventListener('subagent_status', ...)` on `/api/notifications/stream` |
| Harvest progress | `subscribeHarvestProgress(...)` | `EventSource` on `/api/harvest/progress`; returns `{ ready: Promise, close }` |
| WebSocket | `connectWebSocket(onMessage)` | `new WebSocket(baseUrl→ws + /ws?token=<authToken>)` |
The chat SSE event names are normalized inside `sendMessage`: `token→token`, `tool→tool_start`, `tool_result→tool_end`, `done`, `error`, `step`, `approval_request`. `useChat` additionally handles `approval_required` and `model_switch`.
### 2.4 Response normalization helpers (rebuild must mirror these)
The backend and the frontend contract disagree on several field names; the adapter normalizes on read. A Lovable rebuild that talks to the same backend must reproduce these mappings or it will crash on `undefined`:
| Helper | Maps |
|---|---|
| `unwrapArray<T>(data)` | accepts raw array OR `{ results: [...] }` / `{ key: [...] }` envelopes |
| `normalizeFrame(raw)` | `frameType` codes `I/F/E/D/T/N` → `insight/fact/event/decision/task/entity`; `importance` string `low/normal/high/critical` ↔ number `1-4` |
| `normalizeCronJob(raw)` | server `cronExpr/lastRunAt/nextRunAt` → client `schedule/lastRun/nextRun` |
| `getFleet()` | server `durationMs/tokensUsed` → client `duration/tokenUsage` |
| `getModelPricing()` | server `inputPer1k/outputPer1k` → client `inputCostPer1k/outputCostPer1k` |
| `getMemoryStats()` | server `frameCount/entityCount/relationCount` → client `frames/entities/relations`; tolerates `workspace: null` |
| `getModel()` | accepts raw string OR `{ model }` |
### 2.5 Tauri dual-path
Several memory methods branch on `isTauri()` (`lib/tauri-bindings.ts`) and use Rust IPC instead of HTTP when running inside the desktop binary: `addMemoryFrame`, `searchMemory`, `getKnowledgeGraph`, `getIdentity`. A web-only Lovable rebuild uses the HTTP path exclusively (the `else` branch of each).
---
## 3. Complete Endpoint Reference
Every adapter method below maps to a backend route. Method = HTTP verb the adapter issues. "Stream" = SSE/WS. Request/response shapes are the adapter's declared TS types (verbatim).
### 3.1 Auth / Health / System
| Method | Path | Request | Response | Stream |
|---|---|---|---|---|
| GET | `/health` | — | `SystemHealth { status, uptime, services[] }` | no |
| GET | `/api/auth/session-token` | — | `{ token? }` | no |
| GET | `/api/agent/status` | — | `AgentStatus { model, tokensUsed, costUsd, isActive }` | no |
| GET | `/api/agent/cost` | — | `{ totalCost, totalTokens }` | no |
| GET | `/api/agent/model` | — | `string \| { model }` | no |
| PUT | `/api/agent/model` | `{ model }` | — | no |
| POST | `/api/agent/abort` | `{ workspaceId }` | — | no |
### 3.2 Workspaces & Templates
| Method | Path | Request | Response |
|---|---|---|---|
| GET | `/api/workspaces` | — | `Workspace[]` (normalizes `personaId`→`persona`) |
| POST | `/api/workspaces` | `{ name, group, persona?/personaId?, agentGroupId?, templateId?, shared?, model? }` | `Workspace` |
| PUT | `/api/workspaces/:id` | `Partial<Workspace>` | `Workspace` |
| PATCH | `/api/workspaces/:id` | `Partial<{persona, agentGroupId, templateId, name, group, model}>` | `Workspace` |
| DELETE | `/api/workspaces/:id` | — | — |
| GET | `/api/workspaces/:id/context` | — | `WorkspaceContext` |
| GET | `/api/workspaces/:id/files` | — | `unknown[]` |
| GET | `/api/workspace-templates` | — | `{ templates: WorkspaceTemplate[], count }` |
| POST | `/api/workspace-templates` | `Omit<WorkspaceTemplate,'id'\|'builtIn'>` | `WorkspaceTemplate` |
| POST | `/api/workspace-templates/generate` | `{ prompt, availableConnectors[], availableCommands[], availablePersonas[] }` | template |
| PUT | `/api/workspace-templates/:id` | template | `WorkspaceTemplate` |
| DELETE | `/api/workspace-templates/:id` | — | — |
| GET | `/api/browse/local?path=` | — | `{ entries[{name,path,type}], current }` |
| POST | `/api/browse/local/mkdir` | `{ path }` | `{ name, path, type }` |
### 3.3 Files
| Method | Path | Request | Response |
|---|---|---|---|
| GET | `/api/workspaces/:id/files/list?path=` | — | `FileEntry[]` |
| POST | `/api/workspaces/:id/files/upload` | `FormData(file, path)` (30s timeout) | `FileEntry` |
| GET | `/api/workspaces/:id/files/download?path=` | — | `Blob` |
| POST | `/api/workspaces/:id/files/mkdir` | `{ path }` | `FileEntry` |
| POST | `/api/workspaces/:id/files/delete` | `{ path }` | — |
| POST | `/api/workspaces/:id/files/move` | `{ from, to }` | `FileEntry` |
| POST | `/api/workspaces/:id/files/copy` | `{ from, to }` | `FileEntry` |
| GET | `/api/workspaces/:id/documents` | — | `{ documents[{name, versions[]}] }` |
| GET | `/api/workspaces/:id/documents/:name/versions` | — | `{ versions[] }` |
### 3.4 Chat / Sessions / Pins / History / Feedback
| Method | Path | Request | Response | Stream |
|---|---|---|---|---|
| POST | `/api/chat` | `{ workspaceId, message, sessionId?, persona?, autonomy?, shape }` | SSE body of `StreamEvent`s | **yes (SSE)** |
| DELETE | `/api/chat/history?session=` | — | — | no |
| GET | `/api/history?workspace=&session=` | — | `ChatMessage[]` | no |
| GET | `/api/workspaces/:id/sessions` | — | `Session[]` | no |
| POST | `/api/workspaces/:id/sessions` | — | `Session` | no |
| PATCH | `/api/sessions/:id?workspace=` | `{ title }` | — | no |
| DELETE | `/api/sessions/:id?workspace=` | — | — | no |
| GET | `/api/workspaces/:id/sessions/search?q=` | — | `Session[]` | no |
| GET | `/api/workspaces/:id/sessions/:sid/export` | — | `string` | no |
| GET | `/api/workspaces/:id/pins` | — | `{ pins[] }` | no |
| POST | `/api/workspaces/:id/pins` | `{ messageContent, messageRole, label? }` | pin | no |
| DELETE | `/api/workspaces/:id/pins/:pinId` | — | — | no |
| POST | `/api/feedback` | `{ sessionId, messageIndex, rating, reason?, detail? }` | — (fire-and-forget) | no |
### 3.5 Memory / Knowledge Graph / Identity
| Method | Path | Request | Response |
|---|---|---|---|
| GET | `/api/memory/frames?limit=&workspace=` | — | `MemoryFrame[]` (normalized) |
| POST | `/api/memory/frames` | `Omit<MemoryFrame,'id'>` | `MemoryFrame` |
| PUT | `/api/memory/frames/:id` | `Partial<MemoryFrame>` | `MemoryFrame` |
| DELETE | `/api/memory/frames/:id` | — | — |
| PATCH | `/api/memory/frames/:id/access?workspace=` | — | `{ accessCount }` |
| GET | `/api/memory/search?q=&scope=` | — | `MemoryFrame[]` |
| GET | `/api/memory/graph?workspace=` / `?scope=all\|personal` | — | `{ nodes: KGNode[], edges: KGEdge[] }` |
| GET | `/api/memory/stats` | — | `{ personal, workspace, total }` each `{frames,entities,relations}` |
| GET | `/api/identity` | — | `IdentityResponse { configured, name, ... }` |
| GET | `/api/team/memory/search?q=&limit=` | — | `{ results[] }` |
| GET | `/api/mind/identity` · `/api/mind/awareness` · `/api/mind/skills` | — | `unknown` |
### 3.6 Local Inference (Ollama)
| Method | Path | Response |
|---|---|---|
| GET | `/api/local-inference/hardware` | `{ hardware, source }` |
| GET | `/api/local-inference/models?useCase=` | `{ models[], source }` |
| GET | `/api/local-inference/status` | `{ servers[], ollamaInstalled, totalLocalModels }` |
| POST | `/api/local-inference/pull` | `{ ok }` (body `{ model }`) |
### 3.7 Events / Timeline / Stats
| Method | Path | Response | Stream |
|---|---|---|---|
| GET | `/api/events?workspaceId=` | `AgentStep[]` | no |
| GET | `/api/events?workspaceId=&limit=&from=` | `TimelineEvent[]` | no |
| GET | `/api/events/stats` | `{ byType, total, dailyBreakdown? }` | no |
| — | `/api/events/stream` | `AgentStep` | **yes (SSE)** |
### 3.8 Skills / Capabilities / Marketplace
| Method | Path | Request | Response |
|---|---|---|---|
| GET | `/api/skills` | — | `SkillPack[]` (forced `installed:true`) |
| POST | `/api/skills/create` | `{ name, description }` | — |
| GET | `/api/skills/starter-pack/catalog` | — | `SkillPack[]` (maps `family`→category) |
| POST | `/api/skills/starter-pack/:skillId` | `{}` | — (throws with status/body on !ok) |
| GET | `/api/skills/capability-packs/catalog` | — | `SkillPack[]` |
| GET | `/api/skills/test` | — | (raw `adapter.fetch`, CapabilitiesApp) |
| GET | `/api/capabilities/status` | — | `unknown` |
| GET | `/api/marketplace/packs` | — | `SkillPack[]` |
| GET | `/api/marketplace/search?query=&limit=` | — | raw `Response` |
| GET | `/api/marketplace/installed` | — | raw `Response` |
| POST | `/api/marketplace/install` | `{ packageId }` | raw `Response` |
| POST | `/api/marketplace/uninstall` | `{ packageId }` | raw `Response` |
> Note: marketplace search/installed/install/uninstall return the raw `Response` so callers can do status-aware handling (403 → `UpgradeModal`). All four go through authenticated `adapter.fetch` (a raw `fetch` 401s before the session token bootstraps).
### 3.9 Fleet / Agents / Agent Groups / Jobs
| Method | Path | Request | Response |
|---|---|---|---|
| GET | `/api/fleet` | — | `FleetSession[]` (normalized) |
| POST | `/api/fleet/:workspaceId/(pause\|resume\|kill)` | — | — (`stop`→`kill`) |
| POST | `/api/fleet/spawn` | `{ task, persona?, model?, parentWorkspaceId? }` | `FleetSession` (throws on !ok) |
| GET | `/api/personas` | — | `Persona[]` |
| POST | `/api/personas` | `{ name, description, icon?, systemPrompt, tools? }` | `Persona` |
| PATCH | `/api/personas/:id` | persona patch | `unknown` |
| DELETE | `/api/personas/:id` | — | — |
| POST | `/api/personas/generate` | `{ prompt }` | `{ name, description, systemPrompt, tools[] }` |
| GET | `/api/agent-groups` | — | `unknown[]` |
| POST | `/api/agent-groups` | `{ name, description, strategy, members[] }` | `unknown` |
| PATCH | `/api/agent-groups/:id` | group patch | `unknown` |
| DELETE | `/api/agent-groups/:id` | — | — |
| POST | `/api/agent-groups/:id/run` | `{ task, teamId:'default' }` | `unknown` |
| GET | `/api/jobs/:jobId` | — | `{ status, startedAt?, completedAt?, output? } \| null` |
| POST | `/api/jobs/:jobId/cancel` | — | — |
### 3.10 Cron / Scheduled Jobs
| Method | Path | Request | Response |
|---|---|---|---|
| GET | `/api/cron` | — | `CronJob[]` (normalized) |
| POST | `/api/cron` | `{ name, cronExpr, jobType, jobConfig?, workspaceId?, enabled? }` | `CronJob` |
| PUT | `/api/cron/:id` | `Partial<CronJob>` | `CronJob` |
| DELETE | `/api/cron/:id` | — | — |
| POST | `/api/cron/:id/trigger` | — | `{ triggered, autoEnabled?, schedule? }` |
### 3.11 Notifications / Approvals
| Method | Path | Request | Response | Stream |
|---|---|---|---|---|
| — | `/api/notifications/stream` | — | `Notification` / `subagent_status` event | **yes (SSE)** |
| GET | `/api/notifications/history` | — | `Notification[]` | no |
| PATCH | `/api/notifications/:id/read` | — | — | no |
| POST | `/api/notifications/read-all` | — | — | no |
| GET | `/api/approval/pending` | — | `{ pending[{requestId,toolName,input,timestamp}], count }` | no |
| POST | `/api/approval/:requestId` | `{ approved, always, sourceWorkspaceId }` | — | no |
| GET | `/api/approval/grants` | — | `{ grants[], count }` | no |
| DELETE | `/api/approval/grants/:id` | — | — | no |
| POST | `/api/approval/grants/clear` | — | — | no |
### 3.12 Settings / Permissions / Providers / LiteLLM
| Method | Path | Request | Response |
|---|---|---|---|
| GET | `/api/settings` | — | `Settings` |
| PUT | `/api/settings` | `Partial<Settings>` | — |
| GET | `/api/settings/permissions` | — | `{ defaultAutonomy, externalGates[], workspaceOverrides }` |
| PUT | `/api/settings/permissions` | partial | — |
| POST | `/api/settings/test-key` | `{ provider, apiKey }` | `{ valid }` |
| GET | `/api/providers` | — | `{ providers[], search[], activeSearch }` |
| GET | `/api/litellm/models` | — | `string[]` |
| GET | `/api/litellm/status` | — | `unknown` |
| GET | `/api/litellm/pricing` | — | `ModelPricing[]` (normalized) |
| GET | `/api/debug/logs` | — | (raw, SettingsApp) |
| GET | `/api/export` · POST `/api/backup` · POST `/api/restore` | — | (raw, SettingsApp/BackupApp) |
### 3.13 Connectors / Vault / Profile
| Method | Path | Request | Response |
|---|---|---|---|
| GET | `/api/connectors` | — | `Connector[]` |
| GET | `/api/connectors/:id/health` | — | `unknown` |
| POST | `/api/connectors/:id/connect` | — | — |
| POST | `/api/connectors/:id/disconnect` | — | — |
| GET | `/api/vault` | — | `unknown` |
| POST | `/api/vault` | `{ name, value, type? }` | — |
| DELETE | `/api/vault/:id` | — | — |
| GET | `/api/profile` | — | profile |
| PUT | `/api/profile` | `Record<string,unknown>` | profile |
| POST | `/api/profile/analyze-style` | `{ text }` | analysis |
| POST | `/api/profile/analyze-brand` | `{ description }` | analysis |
| POST | `/api/profile/research` | `{}` | research |
### 3.14 Costs / Telemetry / Team / Weaver / Audit
| Method | Path | Response |
|---|---|---|
| GET | `/api/costs` · `/api/cost/by-workspace` · `/api/cost/summary` | cost objects |
| GET | `/api/telemetry/status` | `{ enabled, totalEvents }` |
| POST | `/api/telemetry/toggle` | — (`{ enabled }`) |
| DELETE | `/api/telemetry/events` | `{ deleted }` |
| POST | `/api/telemetry/track` | — (`{ event, properties }`, fire-and-forget) |
| POST | `/api/team/connect` | — (`{ serverUrl, token }`) |
| POST | `/api/team/disconnect` | — |
| GET | `/api/team/status` | `{ connected, teamName? }` |
| GET | `/api/team/members` · `/api/team/activity` · `/api/team/messages?workspaceId=` | arrays |
| GET | `/api/weaver/status` | `{ lastConsolidation?, status }` |
| POST | `/api/weaver/trigger` | — (WeaverPanel, raw) |
| GET | `/api/audit/installs` | `unknown[]` |
### 3.15 Waggle Dance (multi-agent signals)
| Method | Path | Request | Response | Stream |
|---|---|---|---|---|
| GET | `/api/waggle/signals` | — | `WaggleSignal[]` | no |
| POST | `/api/waggle/signals` | `Omit<WaggleSignal,'id'\|'timestamp'>` | `WaggleSignal` | no |
| PATCH | `/api/waggle/signals/:id/ack` | — | — | no |
| — | `/api/waggle/stream` | — | `WaggleSignal` | **yes (SSE)** |
### 3.16 AI-OS Tool Launcher
| Method | Path | Request | Response |
|---|---|---|---|
| GET | `/api/tools/detect` | — | `{ platform, detectedAt, tools[{id,displayName,installed,installedPath,version,hooksInstalled,...}] }` |
| POST | `/api/tools/launch` | `{ id, installedPath, workspaceId?, args?, cwd? }` | `{ ok, pid, error? }` |
| GET | `/api/tools/processes` | — | `{ processes[{pid,toolId,startedAt,workspaceId?}], total }` |
| POST | `/api/tools/kill` | `{ pid }` | `{ ok, pid, reason, error? }` |
| POST | `/api/tools/hooks` | `{ id, action:'install'\|'verify'\|'uninstall', cliPath? }` | `{ ok, action, stdout, stderr, code, error? }` |
### 3.17 Billing / Tier / GDPR Erase / Trial
| Method | Path | Request | Response |
|---|---|---|---|
| GET | `/api/tier` | — | `{ tier, trialDaysRemaining?, trialExpired?, capabilities, usage }` |
| POST | `/api/tier/start-trial` | — | `{ tier, rawTier, trialStartedAt, trialDaysRemaining, trialExpired, capabilities }` (409 if started) |
| POST | `/api/stripe/sync` | `{ sessionId }` | `{ tier, customerId }` |
| POST | `/api/stripe/create-checkout-session` | `{ tier:'PRO'\|'TEAMS' }` | `{ url }` |
| POST | `/api/stripe/create-portal-session` | — | `{ url }` |
| POST | `/api/data/erase` | header `X-Confirm-Erase: yes` + `{ confirmation:'I UNDERSTAND THIS IS PERMANENT' }` | `{ requestedAt, markerPath, dataDirSnapshot, instruction }` |
### 3.18 Import / Harvest
| Method | Path | Request | Response | Stream |
|---|---|---|---|---|
| POST | `/api/import/preview` | `{ data, source }` | `{ knowledgeExtracted[] }` | no |
| POST | `/api/import/commit` | `{ data, source }` | — | no |
| POST | `/api/harvest/preview` | `{ data, source }` | preview | no |
| POST | `/api/harvest/commit` | `{ data, source }` / `{ resumeFromRun }` | result | no |
| GET | `/api/harvest/sources` | — | `{ sources[] }` | no |
| DELETE | `/api/harvest/sources/:source` | — | — | no |
| PATCH | `/api/harvest/sources/:source` | `{ autoSync }` | source | no |
| POST | `/api/harvest/scan-claude-code` | (bodyless) | scan | no |
| POST | `/api/harvest/extract-identity` | — | `{ suggestions[], note? }` | no |
| GET | `/api/harvest/runs/latest-interrupted` | — | `{ run \| null }` | no |
| POST | `/api/harvest/runs/:id/abandon` | — | — | no |
| — | `/api/harvest/progress` | — | `{ phase, current, total, source }` | **yes (SSE)** |
### 3.19 Wiki / Compliance
| Method | Path | Request | Response |
|---|---|---|---|
| GET | `/api/wiki/pages` · `/api/wiki/pages/:slug` · `/api/wiki/pages/:slug/content` | — | page(s) |
| POST | `/api/wiki/compile` | `{ mode:'incremental'\|'full', concepts? }` | result |
| GET | `/api/wiki/health` · `/api/wiki/watermark` | — | objects |
| POST | `/api/wiki/export/obsidian` | `{ outDir }` | `{ outDir, filesWritten, indexPath, byType }` |
| POST | `/api/wiki/export/notion` | `{ rootPageUrl }` | `{ pagesCreated, pagesUpdated, pagesUnchanged, pagesFailed, byType, errors[] }` |
| GET | `/api/compliance/status?workspaceId=` | — | status |
| POST | `/api/compliance/export` | request | json |
| POST | `/api/compliance/export-pdf` | request | `Blob` |
| GET | `/api/compliance/interactions?limit=` · `/api/compliance/models` | — | data |
| GET | `/api/compliance/templates` | — | `{ templates[] }` |
| POST | `/api/compliance/templates` | input | `{ template }` |
| PATCH | `/api/compliance/templates/:id` | patch | `{ template }` |
| DELETE | `/api/compliance/templates/:id` | — | — |
### 3.20 Endpoints called via raw `adapter.fetch` (no dedicated method)
These are hit directly inside components, bypassing a named adapter method:
| Path(s) | Caller |
|---|---|
| `/api/audit/installs`, `/api/skills/test` | `CapabilitiesApp.tsx` |
| `/api/backup/metadata`, `/api/backup`, `/api/restore` | `BackupApp.tsx`, `SettingsApp.tsx` |
| `/api/tasks` | `DashboardApp.tsx` |
| `/api/cost/by-workspace`, `/api/events/stats` | `TelemetryApp.tsx` |
| `/api/evolution/{runs,runs/:id,run,targets,baseline,status}` | `memory/EvolutionTab.tsx` |
| `/api/weaver/trigger` | `memory/WeaverPanel.tsx` |
| `/api/settings`, `/api/settings/permissions`, `/api/providers`, `/api/export`, `/api/debug/logs` | `SettingsApp.tsx` |
| `/api/v1/models`, `/api/litellm/models` | onboarding `ModelTierStep.tsx`, `SpawnAgentDialog.tsx` |
---
## 4. Feature → UI Component → Backend Endpoints
### 4.1 Dock Apps (windows opened via `appId`)
App registration lives in two places: the dock config (`lib/dock-tiers.ts`, by `UserTier`) and the window content switch (`Desktop.tsx` `renderAppContent`). The full `AppId` union (`lib/dock-tiers.ts`):
`chat`, `dashboard`, `memory`, `events`, `capabilities`, `connectors`, `cockpit`, `mission-control`, `settings`, `vault`, `profile`, `terminal`, `calculator`, `notes`, `waggle-dance`, `files`, `agents`, `scheduled-jobs`, `marketplace`, `voice`, `room`, `approvals`, `timeline`, `backup`, `telemetry`, `governance`, `launcher`.
| Feature | UI component | Backend endpoints used |
|---|---|---|
| **Chat** (token-streaming conversation, model switch, pins, file ingest, memory recall, feedback, approvals) | `apps/ChatApp.tsx` + `ChatWindowInstance.tsx` + `chat-blocks/*` (via `useChat`) | `POST /api/chat` (SSE), `GET /api/history`, `DELETE /api/chat/history`, `POST /api/approval/:id`, `submitFeedback`, `searchMemory`, pins (`GET/POST/DELETE /api/workspaces/:id/pins`), `ingestFile` (`/api/ingest`), `getModel/setModel` (`/api/agent/model`), `getModels`, `getSettings`, `getTeamMembers`, `patchWorkspace` |
| **Dashboard / Home** | `apps/DashboardApp.tsx` | `getMemoryStats` (`/api/memory/stats`), `GET /api/tasks` (raw), `getServerUrl` |
| **Memory** (tabs: Frames, Knowledge Graph, Harvest, Wiki, Evolution) | `apps/MemoryApp.tsx` → `memory/KnowledgeGraphViewer.tsx`, `HarvestTab.tsx`, `WikiTab.tsx`, `EvolutionTab.tsx`, `WeaverPanel.tsx`, `ImportReminderBanner.tsx` (via `useMemory`, `useKnowledgeGraph`) | frames CRUD (`/api/memory/frames*`), `searchMemory`, `getMemoryStats`, `getKnowledgeGraph`; **Harvest:** `getHarvestSources`, `scanClaudeCode`, `harvestPreview/Commit`, `getLatestInterruptedHarvestRun`, `resume/abandonHarvestRun`, `extractHarvestIdentity`, `subscribeHarvestProgress`, `toggleHarvestAutoSync`, `removeHarvestSource`; **Wiki:** `getWikiPages/PageContent`, `compileWiki`, `getWikiHealth`, `exportWikiToObsidian/Notion`; **Evolution:** `/api/evolution/{runs,run,targets,baseline,status}`; **Weaver:** `getWeaverStatus`, `POST /api/weaver/trigger` |
| **Events & Logs** | `apps/EventsApp.tsx` (via `useEvents`) | `getEvents` (`/api/events`), `subscribeEvents` (`/api/events/stream` SSE) |
| **Skills & Apps** (incl. Marketplace tab) | `apps/CapabilitiesApp.tsx` | `getSkills`, `getStarterPacks`, `getCapabilityPacks`, `getMarketplacePacks`, `installPack`, `installMarketplacePack`, `GET /api/audit/installs` + `/api/skills/test` (raw) |
| **Connectors** | `apps/ConnectorsApp.tsx` + `connectors/{McpCatalog,McpServerCard,BrandTile}.tsx` | `getConnectors`, `connectConnector`, `disconnectConnector`, `addVaultSecret` |
| **Cockpit / Command Center** (incl. Compliance dashboard) | `apps/CockpitApp.tsx` + `cockpit/{ComplianceDashboard,ComplianceTemplateModal}.tsx` | `getSystemHealth`, `getAgentCost`, `getConnectors`, `getCronJobs`, `getVault`, `getCapabilitiesStatus`, `getAuditInstalls`, `getCostSummary`, `getWeaverStatus`, `getEventStats`; **Compliance:** `getComplianceStatus`, `listComplianceTemplates`, `getHarvestSources`, `exportComplianceReport(+Pdf)`, template CRUD (`/api/compliance/templates*`) |
| **Mission Control** | `apps/MissionControlApp.tsx` | `getFleet`, `getTeamMembers`, `getTeamActivity`, `detectTools`, `fleetAction` |
| **Waggle Dance** | `apps/WaggleDanceApp.tsx` (via `useWaggleDance`) | `getWaggleSignals`, `subscribeWaggleDance` (`/api/waggle/stream` SSE), `publishWaggleSignal`, `acknowledgeWaggleSignal` |
| **Personas (Agents)** | `apps/AgentsApp.tsx` + `agents/{AgentCard,AgentDetail,CreateAgentForm,CreateGroupForm,GroupCard,GroupDetail,GroupExecutionPanel}.tsx` | `getPersonas`, `getCapabilityStatus`, `getAgentGroups`, `createPersona`, `updatePersona`, `deletePersona`, `createAgentGroup`, `updateAgentGroup`, `deleteAgentGroup`, `runAgentGroup`, `generatePersona`, `getJobStatus`, `cancelJob` |
| **Files** | `apps/FilesAppTabs.tsx` → `FilesApp.tsx` + `files/{FileTree,FilePreview,FileActions,FileUploadZone,SyntaxPreview,WorkspaceRail}.tsx` | `listFiles`, `uploadFile`, `downloadFile`, `createDirectory`, `deleteFile`, `moveFile`, `copyFile`, `getDocuments`, `getDocumentVersions` |
| **Scheduled Jobs** | `apps/ScheduledJobsApp.tsx` | `getCronJobs`, `createCronJob`, `updateCronJob`, `deleteCronJob`, `triggerCronJob` |
| **Marketplace** | `apps/MarketplaceApp.tsx` | `connect`, `searchMarketplace`, `getMarketplaceInstalled`, `installMarketplacePackage`, `uninstallMarketplacePackage` |
| **AI Tools / Launcher** | `apps/LauncherApp.tsx` | `detectTools`, `launchTool`, `manageHooks`, `getToolProcesses`, `killTool` |
| **Voice** | `apps/VoiceApp.tsx` | **none** — static "Coming Soon" placeholder |
| **Room** (live sub-agent canvas) | `apps/RoomApp.tsx` (via `useRoomState`) | `subscribeSubagentStatus` (`/api/notifications/stream`, named `subagent_status` SSE event) |
| **Approvals** | `apps/ApprovalsApp.tsx` | `getPendingApprovals`, `getApprovalGrants`, `respondApproval`, `revokeApprovalGrant`, `clearApprovalGrants` |
| **Timeline** | `apps/TimelineApp.tsx` | `getTimeline` (`/api/events?...&from=`) |
| **Backup & Restore** | `apps/BackupApp.tsx` | `GET /api/backup/metadata`, `POST /api/backup`, `POST /api/restore` (raw `adapter.fetch`) |
| **Usage & Telemetry** | `apps/TelemetryApp.tsx` | `GET /api/cost/by-workspace`, `GET /api/events/stats` (raw) |
| **Team Governance** (TEAMS tier) | `apps/TeamGovernanceApp.tsx` | **no direct adapter calls** (renders governance UI; sub-components / props supply data) |
| **Settings** | `apps/SettingsApp.tsx` | `getSettings`/`saveSettings`, `getPermissions`/`savePermissions`, `getTeamStatus`, `getTelemetryStatus`/`toggleTelemetry`/`clearTelemetry`, `teamConnect`/`teamDisconnect`, `GET /api/providers`, `/api/export`, `/api/backup`, `/api/restore`, `/api/debug/logs` (raw) |
| **Vault** | `apps/VaultApp.tsx` | `getVault`, `getConnectors`, `addVaultSecret`, `deleteVaultSecret`, `connectConnector`, `disconnectConnector`, `DELETE /api/vault/:id` |
| **My Profile** | `apps/UserProfileApp.tsx` | `getProfile`, `updateProfile`, `analyzeWritingStyle`, `analyzeBrand`, `researchProfile` |
> `terminal`, `calculator`, `notes` appear in the `AppId` union but have no `*App.tsx` component or `renderAppContent` case — they are declared-but-unimplemented placeholders. `voice` is implemented but is a static placeholder.
### 4.2 Overlays (modals, dialogs, rails — rendered directly by `Desktop.tsx`)
| Overlay | UI component | Backend endpoints used |
|---|---|---|
| Onboarding wizard (8 steps) | `overlays/OnboardingWizard.tsx` + `overlays/onboarding/{Welcome,WhyWaggle,Tier,ModelTier,Import,Template,Persona,ApiKey,Ready}Step.tsx` | `connect`, `trackTelemetry`, `getVault`, `getSystemHealth`, `getProviders`, `harvestPreview/Commit`, `scanClaudeCode`, `import/*`, `createPersona`, `addVaultSecret`, `createWorkspace`, `saveSettings`, `/api/v1/models` |
| Login briefing (session-start digest) | `overlays/LoginBriefing.tsx` | `getIdentity`, `getWorkspaces`, `searchMemory`, `getMemoryStats`, `getWorkspaceContext` |
| Global search (Cmd+K) | `overlays/GlobalSearch.tsx` | `getWorkspaces`, `getSessions`, `getSkills`, `searchMemory` |
| Create workspace dialog | `overlays/CreateWorkspaceDialog.tsx` | `browseLocal`, `browseLocalMkdir`, `getWorkspaceTemplates`, `createWorkspaceTemplate`, `updateWorkspaceTemplate`, `deleteWorkspaceTemplate`, `generateTemplateFromPrompt`, `getConnectors`, `getAgentGroups` |
| Persona switcher | `overlays/PersonaSwitcher.tsx` | `getPersonas`, `getAgentGroups` |
| Spawn agent dialog | `overlays/SpawnAgentDialog.tsx` | `getModels`, `getModelPricing`, `getProviders`, `getModel`, `createWorkspace`, `spawnAgent`, `/api/litellm/models` |
| Workspace switcher | `overlays/WorkspaceSwitcher.tsx` | (props-driven; workspaces from `useWorkspaces`) |
| Notification inbox | `overlays/NotificationInbox.tsx` | (props from `useNotifications`) |
| Context rail | `overlays/ContextRail.tsx` | uses `lib/context-rail-fetch.ts` (memory/context lookups) |
| Erase data dialog (GDPR) | `overlays/EraseDataDialog.tsx` | `eraseData` (`POST /api/data/erase`) |
| Upgrade modal | `overlays/UpgradeModal.tsx` | (triggered by `waggle:tier-insufficient` event; actions call `startTrial`, `createCheckoutSession`) |
| Trial expired modal | `overlays/TrialExpiredModal.tsx` | `createCheckoutSession` |
| Keyboard shortcuts help | `overlays/KeyboardShortcutsHelp.tsx` | none |
| Onboarding tooltips / tour | `overlays/OnboardingTooltips.tsx` | none |
---
## 5. Global Providers & State
There is **no Redux / Zustand / React Query**. State is React Context + custom hooks + the singleton `adapter`.
### 5.1 Provider tree (`apps/web/src/App.tsx`)
`App.tsx` wraps the router in `<ServiceProvider>` plus shadcn `<TooltipProvider>` and a `<Toaster>` (toast system in `hooks/use-toast.ts`). The only domain provider is `ServiceProvider`.
| Provider | File | Provides |
|---|---|---|
| `ServiceProvider` / `useService()` | `providers/ServiceProvider.tsx` | `{ adapter, connected, connecting, error, reconnect }`; calls `adapter.connect()` once on mount |
### 5.2 Domain hooks (the de-facto "store")
Each hook wraps adapter calls + local `useState`. `Desktop.tsx` composes them.
| Hook | File | Owns / returns |
|---|---|---|
| `useWorkspaces` | `hooks/useWorkspaces.ts` | workspaces list, active workspace, `selectWorkspace`, `createWorkspace`, `patchWorkspace`, `deleteWorkspace`, `refresh` |
| `useChat` | `hooks/useChat.ts` | `messages`, `isLoading`, `sendMessage`, `clearHistory`, `pendingApproval`, `approveAction`; parses SSE stream into `ContentBlock[]` |
| `useSessions` | `hooks/useSessions.ts` | per-workspace sessions CRUD |
| `useMemory` | `hooks/useMemory.ts` | frames, stats, search, add/update/delete, `incrementFrameAccess` |
| `useKnowledgeGraph` | `hooks/useKnowledgeGraph.ts` | `{ nodes, edges }` via `getKnowledgeGraph` |
| `useEvents` | `hooks/useEvents.ts` | steps + live `subscribeEvents` SSE |
| `useNotifications` | `hooks/useNotifications.ts` | notifications, `unreadCount`, `markRead`, `markAllRead`, live SSE |
| `useRoomState` | `hooks/useRoomState.ts` | live sub-agent map via `subscribeSubagentStatus` |
| `useWaggleDance` | `hooks/useWaggleDance.ts` | signals, filter, `acknowledge`, `publish`, live SSE |
| `useAgentStatus` | `hooks/useAgentStatus.ts` | `getAgentStatus` polling |
| `useBilling` | `hooks/useBilling.ts` | tier, `startCheckout`, `openPortal`, `syncAfterCheckout` (auto-detects `?session_id=`) |
| `useFeatureGate` | `hooks/useFeatureGate.ts` | `{ planTier, isEnabled, gate }` from `lib/feature-gates.ts` |
| `useOnboarding` | `hooks/useOnboarding.ts` | persisted `OnboardingState` in `localStorage["waggle:onboarding"]` (`completed, step, tier, workspaceId, apiKeySet, templateId, personaId, tooltipsDismissed`); supports `?skipOnboarding=true`, `?forceWizard=true` |
| `useOfflineStatus` | `hooks/useOfflineStatus.ts` | polls `getSystemHealth` for the offline pill |
| `useProviders` | `hooks/useProviders.ts` | `getProviders` (`/api/providers`) |
| `useWindowManager` | `hooks/useWindowManager.ts` | window open/close/focus/minimize, per-window persona & autonomy |
| `useOverlayState` | `hooks/useOverlayState.ts` | boolean flags for every overlay |
| `useKeyboardShortcuts` | `hooks/useKeyboardShortcuts.ts` | global hotkeys → overlay/app open |
| `useDeveloperMode`, `useDockNudge`, `useDockLabels` | `hooks/use*.ts` | UI affordances |
### 5.3 Cross-component event bus (`window` CustomEvents)
The UI also coordinates through DOM events (no library). A rebuild must wire these:
| Event | Dispatched by | Consumed by |
|---|---|---|
| `waggle:tier-insufficient` (`{ required, actual, message }`) | `adapter.fetch` on 403 `TIER_INSUFFICIENT` | `UpgradeModal` |
| `waggle:open-app` (`{ appId, tab? }`) | `HarvestTab` and others | `Desktop.tsx` → `wm.openApp(appId)` |
---
## 6. Tier / Billing Gating (drives dock & feature visibility)
- **Dock visibility** is computed by `getDockForTier(tier, billingTier)` (`lib/dock-tiers.ts`). `UserTier` = `'simple' | 'professional' | 'power' | 'admin'` (UI complexity); `BillingTier` = `'TRIAL' | 'FREE' | 'PRO' | 'TEAMS' | 'ENTERPRISE'`. Entries with `minBillingTier` (e.g. `governance` and `approvals` = `TEAMS`) are filtered out below that tier; empty zone-parents are dropped.
- **Feature gating** uses `useFeatureGate()` → `lib/feature-gates.ts` (`isFeatureEnabled`, `getGate`, `dockTierToPlanTier`).
- **Trial / upgrade flow:** `Desktop.tsx` calls `getTier` on mount; `startTrial` → `POST /api/tier/start-trial`; checkout via `createCheckoutSession`; post-checkout sync via `useBilling` detecting `?session_id=` then `POST /api/stripe/sync`.
---
## 7. Rebuild Checklist (what Lovable must recreate, in order)
1. A single `adapter` module pointing at `http://127.0.0.1:3333`, with: server-url override in `localStorage["waggle:server-url"]`, session-token bootstrap (`GET /api/auth/session-token`), `Authorization: Bearer` injection, 403→`waggle:tier-insufficient` event, 10s/30s timeouts, and the read normalizers in §2.4.
2. `ServiceProvider` calling `adapter.connect()` once; expose `useService()`.
3. The window-manager desktop shell (open windows by `AppId`, dock per tier).
4. SSE plumbing for chat (`POST /api/chat`), events, notifications (+ named `subagent_status`), waggle-dance, harvest-progress; WS optional.
5. The 24 implemented apps + 13 overlays, each calling the endpoints in §4.
6. The cross-component `window` event bus (§5.3) and the tier-gating logic (§6).
---
## Counts
- **Endpoints documented:** ~135 distinct backend routes (across §3.1–§3.20).
- **Apps documented:** 24 implemented dock apps (+3 declared-but-unimplemented: `terminal`, `calculator`, `notes`).
- **Overlays documented:** 13.
- **Pages:** 2 (`Index`, `NotFound`).
- **Domain hooks documented:** 19.
- **Providers:** 1 (`ServiceProvider`).

404
docs/backend-map/sections/05a-subsystem-agent-runtime.md Normal file
View File

@@ -0,0 +1,404 @@
# 05a — Subsystem: Agent Runtime
**Purpose.** The agent runtime is the engine that turns one user message into one agent response. It assembles a layered system prompt (identity + memory + behavioral rules + persona + response scaffold), runs a tool-calling loop against an OpenAI-compatible LLM endpoint (LiteLLM), executes each tool through an explicit middleware chain (governance, hooks, injection-scan, loop-guard), and applies completion-time "gates" that force verification, real file writes, and skill distillation before accepting a final answer. This document is the contract a frontend needs to understand what `POST /api/chat` actually does between request and SSE response — every claim below is grounded in the code under `packages/agent/src/`.
> Scope note: this section documents the **agent runtime** (the loop, prompt, personas, tools, gates, cost). The HTTP surface (`/api/chat`, agent-run routes, SSE event names) is the API subsystem's job; this section only shows where the runtime plugs into `packages/server/src/local/routes/chat.ts` so the wiring is legible.
---
## 1. The mental model in one paragraph
A turn enters at `runAgentLoop(config)` in `agent-loop.ts`. It builds a `messages` array (`[{role:'system', content: systemPrompt}, ...history]`), converts `ToolDefinition[]` to OpenAI `tools` schema, then loops up to `maxTurns` times. Each iteration POSTs to `${litellmUrl}/chat/completions`. If the model returns **no tool calls**, the loop runs the **completion gates** (`maybeFireCompletionGate`) — and if none fire, returns the final `AgentResponse`. If the model returns **tool calls**, each is executed through `executeToolCall` (the middleware chain), results are pushed back as `role:'tool'` messages, and the loop continues. The system prompt itself is built by the `Orchestrator` (`buildSystemPrompt` + `recallMemory`, optionally the tier-adaptive `PromptAssembler`) and decorated by the route with persona, profile, runtime facts, behavioral spec, and workspace context.
---
## 2. Files in this subsystem
| File | Role |
|---|---|
| `packages/agent/src/agent-loop.ts` | `runAgentLoop()` — the core tool-calling loop. Re-exports retrieval-loop entry points. |
| `packages/agent/src/orchestrator.ts` | `Orchestrator` class — `buildSystemPrompt()`, `recallMemory()`, `buildAssembledPrompt()`, `autoSaveFromExchange()`, memory-layer accessors. |
| `packages/agent/src/tool-executor.ts` | `executeToolCall()` — 11-step per-tool middleware chain. |
| `packages/agent/src/loop-gates.ts` | `maybeFireCompletionGate()` + `GateState` — the D3/D4/D1 completion gates. |
| `packages/agent/src/loop-guard.ts` | `LoopGuard` — duplicate/oscillation tool-call detection. |
| `packages/agent/src/retry-policy.ts` | `handleNonOkResponse()` / `handleNetworkError()` — 429 / 5xx / network backoff. |
| `packages/agent/src/verification-gate.ts` | `assertsUnverifiedCompletion()` + `VERIFICATION_GATE_DIRECTIVE` (D3). |
| `packages/agent/src/skill-distillation.ts` | `planSkillDistillation()` / `shouldDistillSkill()` (D1). |
| `packages/agent/src/personas.ts` | `AgentPersona` interface + `composePersonaPrompt()`, `getPersona()`, `listPersonas()`. |
| `packages/agent/src/persona-data.ts` | `PERSONAS` array — 22 built-in personas (pure data). |
| `packages/agent/src/tool-filter.ts` | `filterToolsForContext()`, `filterAvailableTools()`, `filterOfflineTools()`. |
| `packages/agent/src/prompt-assembler.ts` | `PromptAssembler` — sixth-layer tier-adaptive prompt (feature-flagged). |
| `packages/agent/src/context-loader.ts` | `loadRecentContext()` / `loadRecentContextFrames()` + `ContextFrames` type. |
| `packages/agent/src/task-shape.ts` | `detectTaskShape()` — pure heuristic task classifier. |
| `packages/agent/src/model-tier.ts` | `tierForModel()` — maps a model id to `small` / `mid` / `frontier`. |
| `packages/agent/src/behavioral-spec.ts` | `BEHAVIORAL_SPEC` (v3.0) + `buildActiveBehavioralSpec()` + `COMPACTION_PROMPT`. |
| `packages/agent/src/cost-tracker.ts` | `CostTracker` + `DEFAULT_MODEL_PRICING` + `BudgetExceededError`. |
| `packages/agent/src/result-formatter.ts` | `formatCombinedResult()` — renders combined-retrieval results to markdown. |
| `packages/agent/src/turn-context.ts` | `generateTurnId()` / `logTurnEvent()` — per-turn trace correlation (H-AUDIT-1). |
---
## 3. The loop — `runAgentLoop(config)` (`agent-loop.ts`)
### 3.1 Input contract (`AgentLoopConfig`)
| Field | Type | Notes |
|---|---|---|
| `litellmUrl` | `string` | Base URL; loop POSTs to `${litellmUrl}/chat/completions`. |
| `litellmApiKey` | `string` | Sent as `Authorization: Bearer …`. |
| `model` | `string` | Model id (passed through to LiteLLM). |
| `systemPrompt` | `string` | Pre-built (by Orchestrator + route); becomes the first `system` message. |
| `tools` | `ToolDefinition[]` | Converted to OpenAI function-tool schema. |
| `messages` | `Array<{role,content}>` | Conversation history (no system message). |
| `onToken?` | `(token)=>void` | Streaming token callback (also receives retry-notice text). |
| `onToolUse?` | `(name,input)=>void` | Fires before each tool executes. |
| `onToolResult?` | `(name,input,result)=>void` | Fires after each tool, with **sanitized** result. |
| `maxTurns?` | `number` | Default **10**. The chat route sets **200** for persistent agents. |
| `stream?` | `boolean` | Default false. When true, sends `stream:true` + `stream_options.include_usage`. |
| `fetch?` | `typeof fetch` | Injectable for tests. |
| `hooks?` | `HookRegistry` | `pre:tool` / `post:tool` / `pre:memory-write` / `post:memory-write`. |
| `capabilityRouter?` | `CapabilityRouter` | Resolves unknown tool names to alternatives. |
| `pluginTools?` | `PluginToolProvider` | Merges active plugin tools into the toolset. |
| `maxTokenBudget?` | `number` | Loop terminates gracefully when `input+output` exceeds this. |
| `signal?` | `AbortSignal` | Client-disconnect; loop exits between turns and tears down in-flight fetch. |
| `governancePolicies?` | `{blockedTools?, allowedSources?}` | **`blockedTools` IS enforced; `allowedSources` is NOT yet enforced** (logs a warning only — tools lack source-provenance metadata). |
| `traceRecording?` | `{recorder, handle}` | Auto-wires trace callbacks (additive to user callbacks). |
| `turnId?` | `string` | UUID v4 correlation key (H-AUDIT-1). |
| `verificationGate?` | `boolean` | Default **true** (D3). |
| `skillDistillationGate?` | `boolean` | Default **true** (D1). |
| `onSkillDistillationFire?` | `(info)=>void` | AI-OS Phase 3 skill-diffusion observer; fires the moment D1 triggers. Errors swallowed. |
### 3.2 Output contract (`AgentResponse`)
```ts
interface AgentResponse {
content: string; // final answer
toolsUsed: string[]; // names of attempted tools
usage: { inputTokens: number; outputTokens: number };
}
```
`AgentMessage` is the internal message shape (note `tool_calls` and `tool_call_id` for OpenAI tool-call threading):
```ts
interface AgentMessage {
role: 'system' | 'user' | 'assistant' | 'tool';
content: string | null;
tool_calls?: Array<{ id: string; type: 'function'; function: { name: string; arguments: string } }>;
tool_call_id?: string;
}
```
### 3.3 Loop steps (per iteration, up to `maxTurns`)
1. **Abort check** — if `signal.aborted`, return `"Agent loop aborted (client disconnected)."`
2. **Build request body** — `{model, messages}`; add `tools` if any; add `stream`/`stream_options` if streaming. Request signal = `AbortSignal.any([clientSignal, AbortSignal.timeout(llmTimeoutMs)])`. Timeout default **300_000 ms** (`WAGGLE_LLM_TIMEOUT_MS`).
3. **POST** to `/chat/completions`.
- **Network rejection** → `handleNetworkError()`: retry with exponential backoff (cap 30 s, max 3) **without consuming a turn** (`turn--; continue`), unless a genuine client-abort (re-throws).
- **`!response.ok`** → `handleNonOkResponse()`: 429 honors `Retry-After` (cap 60 s); 502/503/504 exponential backoff (cap 30 s); anything else is fatal. Same `turn--` retry semantics.
4. **Parse response** — streaming via `parseChatCompletionStream` (accumulates tokens into `allStreamedContent`), or non-streaming via `response.json()` (`choices[0].message`). Track per-turn `prompt_tokens` / `completion_tokens`.
5. **Post-read abort check** — return early if aborted mid-read.
6. **Accumulate tokens**; reset retry counters on success.
7. **Token-budget check** — if `maxTokenBudget` exceeded, return (preferring `preservedAnswerForDistillation` if D1 already fired — issue #4).
8. **No tool calls** → run **`maybeFireCompletionGate`** (§5). If a gate fired, `continue`. Otherwise emit content and return `AgentResponse` (final content = `preservedAnswerForDistillation ?? content`).
9. **Has tool calls** → push the assistant message (content coerced to `''` when null, for LiteLLM→Anthropic compatibility), then run **`executeToolCall`** for each call (§4), pushing each result as a `role:'tool'` message. Continue loop.
10. **`maxTurns` reached** → return preserved answer, else accumulated content, else `"Max tool turns reached (...)"`.
> Re-exported from this file (implementation in `retrieval-agent-loop.ts`): `runSoloAgent`, `runRetrievalAgentLoop`, `runRetrievalAgentLoopWithRecovery`, and their config/result types. The structured-action **retrieval** loop is a sibling pattern; `agent-loop.ts` is the canonical entry point for both.
---
## 4. Tool execution — `executeToolCall()` (`tool-executor.ts`)
Each tool call passes through an **ordered 11-step middleware chain**. The order around sanitization (steps 8→9→10→11) is a load-bearing invariant — sanitized output is what flows into both model context AND every observer (audit/telemetry/team-sync/UI).
| Step | Action | Early-return? |
|---|---|---|
| 1 | `JSON.parse` arguments | Yes — invalid JSON → error result, `countedAsUsed:false` |
| 2 | `onToolUse(name, args)` callback | — |
| 3 | **Governance `blockedTools`** check | Yes — fires `onToolResult` with a policy message, `countedAsUsed:false` |
| 4 | `pre:tool` hook | Yes — cancel → `[BLOCKED] reason` |
| 5 | `pre:memory-write` hook (only `save_memory`) | Yes — cancel → `[BLOCKED] Memory write blocked` |
| 6 | **`LoopGuard.check`** | Produces an error result (not early-return) if a loop is detected |
| 7 | **Execute** the tool (or capability-router fallback, or unknown-tool error). Wrapped in try/catch. | — |
| 8 | **`scanForInjection(result, 'tool_output')`** — replaces output with `[SECURITY] … sanitized.` if unsafe | — |
| 9 | `onToolResult(name, args, sanitizedResult)` | — |
| 10 | `post:memory-write` hook (only `save_memory`) | — |
| 11 | `post:tool` hook | — |
`countedAsUsed` is **true** only when the tool was actually attempted (success OR thrown-inside error) — pre-execution rejections (parse error, governance block, hook cancel) do not count toward `toolsUsed`.
**Unknown tool with a `capabilityRouter`:** returns a list of alternative routes (`[source] name: description (available|not wired yet)`) and, if any are `missing` and `acquire_capability` is on offer, appends a tip to use it.
### LoopGuard (`loop-guard.ts`)
- Hashes `sha256(toolName + ':' + JSON.stringify(args))`.
- **Consecutive repeats:** flags after `maxRepeats` (default **3**) identical calls in a row.
- **Oscillation window:** rolling window of `windowSize` (default **10**); flags if the same hash appears `windowThreshold` (default **4**) times. Returns `false` → executor produces a "Loop detected" error instead of running the tool.
---
## 5. Completion gates — `maybeFireCompletionGate()` (`loop-gates.ts`)
Gates run only on the **no-tool-calls** (final-answer) branch. At most **one** fires per call; each is **one-shot** per `runAgentLoop` invocation via `GateState`. When a gate fires, it pushes the assistant content + a corrective `user` directive into `messages` and returns `fired:true`, so the loop continues for one more model turn.
`GateState` flags: `verificationCorrectionUsed`, `writeCorrectionUsed`, `skillDistillationUsed`, and `preservedAnswerForDistillation` (the real user answer captured at D1 fire time, surfaced in the final return instead of the skill-summary turn).
| Gate | Order | Fires when | Directive |
|---|---|---|---|
| **D3 — Verification** | 1st | `assertsUnverifiedCompletion(content, toolsUsed)`: content asserts "tests pass / build succeeds / it works / verified" but **no** verification-class tool (`test\|build\|run\|verif\|lint\|tsc\|pytest\|jest\|vitest\|exec\|bash\|compile\|spec`) was used | `VERIFICATION_GATE_DIRECTIVE` — "run the check now and quote real output, OR label UNVERIFIED" |
| **D4 — Phantom write** | 2nd | `writeToolAvailable` (a `write_file`/`edit_file` tool is on offer) AND `assertsPhantomWrite(content, toolsUsed)`: past-tense "created/wrote/saved the file" with no write-class tool (`write_file`/`edit_file`/`generate_docx`) used | `WRITE_GATE_DIRECTIVE` — "call write_file NOW with the full contents, or say you didn't intend to" |
| **D1 — Skill distillation** | 3rd | `planSkillDistillation(toolsUsed, content)` returns non-null: **≥5** tool calls AND not a self-incapacity/refusal turn (R2 sign-gate) | Directive instructing the model to call `create_skill` next turn (after `search_skills`). Also invokes `onSkillDistillationFire` (errors swallowed). |
`writeToolAvailable` is computed once in the loop: `toolMap.has('write_file') || toolMap.has('edit_file')`.
---
## 6. The Orchestrator — prompt + memory (`orchestrator.ts`)
`Orchestrator` owns the **memory layers** and produces the prompt pieces. Constructed with `{db, embedder, apiKey?, model?, mode?, version?, skills?}`. It instantiates `IdentityLayer`, `AwarenessLayer`, `FrameStore`, `SessionStore`, `HybridSearch`, `KnowledgeGraph`, `ImprovementSignalStore`, a `CognifyPipeline`, and the mind tools (`createMindTools`).
It can hold a **second mind**: `setWorkspaceMind(workspaceDb)` activates workspace-specific frames/search/knowledge/cognify alongside the personal mind. Identity always stays personal. `getMemoryStats()` returns `{frameCount, sessionCount, entityCount}` summed across both.
### 6.1 `buildSystemPrompt(): string`
Joins three sections (filtering empties):
1. **Identity** — cached (`cachedSection`), keyed on the full identity JSON. `# Identity\n` + `identity.toContext()`.
2. **Self-awareness** — uncached (runtime). `buildSelfAwareness(caps)` over `AgentCapabilities` (tools list, skills, model, memory stats, mode, version, actionable awareness). Defers signal marking to `commitSurfacedSignals()`.
3. **Preloaded context** — `loadRecentContext()` → `# Context From Your Memory\n…`.
### 6.2 `recallMemory(query, limit=10, opts?): Promise<{text, count, recalled?}>`
Automatic per-turn memory recall. Key behaviors:
- **Catch-up intent detection** — regexes like `catch me up`, `where are we`, `what's next`, `summariz`, `remind me`. On a hit with an active workspace, fetches **importance-ranked** frames (critical/important or `Decision%`/`%decided%`) by SQL instead of semantic search, deduped by frame id.
- **Authoritative-recall filter** — drops frames whose importance is `temporary` or `deprecated` (R2 sign-gate: self-incapacity frames are stored `temporary` so they don't re-enter as authoritative).
- **Optional `scoreFloor`** (PromptAssembler opt-in) — filters by `finalScore ?? score`.
- **Injection scan** — joins all recall lines and runs `scanForInjection(…, 'tool_output')`; on a hit, **blocks** the entire recall (returns empty) and logs a warning.
- **Provenance-honest framing** — the returned `text` includes strict instructions: attribute saved memory explicitly, never claim continuity ("welcome back") on a first message, never confabulate numbers/dates/names not present verbatim.
- **On error** — returns `"[Memory recall temporarily unavailable. Proceed without prior context.]"` (visible, not silent — silent empties train the model to confabulate).
`RecallOptions`: `{ profile?: ScoringProfile (default 'balanced'), scoreFloor?, tier?, turnId? }`.
### 6.3 `buildAssembledPrompt(query, persona, opts)` (feature-flagged)
Used only when `isEnabled('PROMPT_ASSEMBLER')`. Computes `tierForModel(model)`, calls `buildSystemPrompt()` for the core, loads typed `ContextFrames`, runs raw `HybridSearch` (limit 10, balanced) on personal + workspace, runs ONE injection scan (assembler trusts `scanSafe` and must not re-scan), and hands a `RecalledMemory` to `PromptAssembler.assemble()`.
### 6.4 Post-response write-back
`autoSaveFromExchange(userMsg, assistantMsg)` → `runPatternWriteBack(...)`: regex-driven heuristic save (preferences/corrections/style → personal mind; decisions/work-output → workspace or personal). `commitSurfacedSignals()` marks awareness signals as surfaced **after** a successful model call.
---
## 7. Personas (`personas.ts` + `persona-data.ts`)
### 7.1 `AgentPersona` interface (all fields)
| Field | Type | Meaning |
|---|---|---|
| `id` | `string` | Stable persona id. |
| `name` | `string` | Display name. |
| `description` | `string` | Short role description. |
| `icon` | `string` | Emoji icon. |
| `systemPrompt` | `string` | Role instructions appended **after** the core prompt. |
| `modelPreference` | `string` | Suggested model (user-overridable). All built-ins: `claude-sonnet-4-6`. |
| `tools` | `string[]` | Tool subset the persona uses (allowlist). |
| `workspaceAffinity` | `string[]` | Workspace types this persona suits. |
| `suggestedCommands` | `string[]` | Slash commands to suggest. |
| `defaultWorkflow` | `string \| null` | Auto-invoke workflow template. |
| `disallowedTools?` | `string[]` | Denylist — overrides `tools[]` on conflict. |
| `failurePatterns?` | `string[]` | Documented failure modes (tooltip). |
| `isReadOnly?` | `boolean` | `true` = no write tools ever. Drives whether `WRITE_DISCIPLINE` is appended. |
| `tagline?` | `string` | One-sentence picker-hover line. |
| `bestFor?` | `string[]` | 3 example tasks (user-facing). |
| `wontDo?` | `string` | Hard boundary statement. |
| `suggestedSkills?` | `string[]` | Marketplace skill names. |
| `suggestedConnectors?` | `string[]` | Connector ids. |
| `suggestedMcpServers?` | `string[]` | MCP server names. |
### 7.2 The 22 built-in personas (`PERSONAS` array)
> CLAUDE.md describes "13 + 4". The actual `persona-data.ts` array contains **22** personas: the original 8, the "Mega Test V2" 5, the universal/orchestration 4, and 5 more domain personas. Listed exactly as found:
| # | `id` | `name` | icon | readonly | `disallowedTools` | `defaultWorkflow` |
|---|---|---|---|---|---|---|
| 1 | `researcher` | Researcher | 🔬 | — | — | `research-team` |
| 2 | `writer` | Writer | ✍️ | — | bash, git_commit, git_push, spawn_agent | null |
| 3 | `analyst` | Analyst | 📊 | — | — | null |
| 4 | `coder` | Coder | 💻 | — | `[]` | null |
| 5 | `project-manager` | Project Manager | 📋 | — | git_commit, git_push, bash | `plan-execute` |
| 6 | `executive-assistant` | Executive Assistant | 📧 | — | bash, git_commit, git_push, spawn_agent | null |
| 7 | `sales-rep` | Sales Rep | 🎯 | — | git_commit, git_push, bash, spawn_agent | `research-team` |
| 8 | `marketer` | Marketer | 📢 | — | bash, git_commit, git_push, spawn_agent | null |
| 9 | `product-manager-senior` | Senior PM | 🗺️ | — | bash, git_commit, git_push | `plan-execute` |
| 10 | `hr-manager` | HR Manager | 👥 | — | bash, git_commit, git_push, spawn_agent | null |
| 11 | `legal-professional` | Legal Counsel | ⚖️ | — | bash, git_commit, git_push, spawn_agent | null |
| 12 | `finance-owner` | Business Finance | 💰 | — | bash, git_commit, git_push, spawn_agent | null |
| 13 | `consultant` | Strategy Consultant | 🎯 | — | `[]` | `research-team` |
| 14 | `general-purpose` | General Purpose | 🧠 | `false` | `[]` | null |
| 15 | `planner` | Planner | 🗂️ | **`true`** | write_file, edit_file, git_commit, git_push, git_merge, generate_docx, install_capability, spawn_agent, execute_step, save_memory | null |
| 16 | `verifier` | Verifier | 🔍 | **`true`** | write_file, edit_file, git_commit, git_push, git_merge, generate_docx, save_memory, install_capability, spawn_agent, execute_step | null |
| 17 | `coordinator` | Coordinator | 🎛️ | `false` | read_file, write_file, edit_file, bash, web_search, web_fetch, search_files, search_content, generate_docx, git_* | `coordinator` |
| 18 | `support-agent` | Customer Support | 🎧 | — | — | null |
| 19 | `ops-manager` | Operations Manager | ⚙️ | — | — | `plan-execute` |
| 20 | `data-engineer` | Data Engineer | 📈 | — | — | null |
| 21 | `recruiter` | Recruiter | 🤝 | — | — | null |
| 22 | `creative-director` | Creative Director | 🎨 | — | — | `review-pair` |
The 4 "universal + orchestration tier" personas (#14–17) match CLAUDE.md's "Target (17 personas — add 4)" goal: `general-purpose`, `planner`, `verifier`, `coordinator`. `coordinator` requires `FEATURE_FLAGS.COORDINATOR_MODE` (gates `spawn_agent` at runtime).
### 7.3 `composePersonaPrompt(corePrompt, persona, maxChars=32000, workspaceTone?)`
Builds the final system prompt around the persona:
1. Always appends `DOCX_HINT` to the core prompt.
2. If `workspaceTone` matches a `TONE_INSTRUCTIONS` key (`professional`/`casual`/`technical`/`legal`/`marketing`), appends a `## Communication Tone` section.
3. If `persona` is null → returns base prompt.
4. **Writable personas** (`isReadOnly` falsy) get the **`WRITE_DISCIPLINE`** block appended to their `systemPrompt` — an explicit "you MUST call write_file; text in your reply is NOT a file" directive that counters phantom-write narration. Read-only personas never get it.
5. Combined with `SEPARATOR` (`\n\n---\n\n`); persona prompt truncated to fit `maxChars` (~8000 tokens) if needed.
`getPersona(id)` → built-ins only. `listPersonas()` → built-ins + custom from disk (`loadCustomPersonas(dataDir)` after `setPersonaDataDir`).
---
## 8. Tool filtering (`tool-filter.ts`)
| Function | Behavior |
|---|---|
| `filterToolsForContext(tools, context, config?)` | `config.enabled_tools` (allowlist) wins; else `context` selects `CODE_TOOLS` or `RESEARCH_TOOLS` sets, or all for `'general'`; then `config.disabled_tools` is subtracted. |
| `filterAvailableTools(tools)` | Runs each tool's `checkAvailability()`; excludes those returning false or throwing. Tools without the check are always included. |
| `filterOfflineTools(tools)` / `getOfflineCapableToolNames(tools)` | Keep only `offlineCapable === true` (PM-6, offline mode). |
`ToolContext = 'general' | 'code' | 'research'`. `CODE_TOOLS` = bash, read/write/edit_file, search_files, search_content, git_status/diff/log/commit. `RESEARCH_TOOLS` = web_search, web_fetch, search_memory, get_identity, get_awareness, query_knowledge, read_file, search_files, search_content, find_connector, list_connector_categories.
> Note: persona-level allow/deny (`tools[]` / `disallowedTools[]`) is enforced separately (CLAUDE.md references a planned `assembleToolPool`). `filterToolsForContext` is the **context**-level filter; both can apply.
---
## 9. PromptAssembler — the sixth layer (`prompt-assembler.ts`)
Feature-flagged (`WAGGLE_PROMPT_ASSEMBLER=1`, default **off**). Produces a tier-adaptive, typed, scaffolded prompt. `PromptAssembler.assemble(input, opts): AssembledPrompt`.
```ts
interface AssembledPrompt {
system: string;
userPrefix: string; // currently ''
responseScaffold: string | null;
debug: { tier, taskShape, taskShapeConfidence, scaffoldApplied,
scaffoldStyle, sectionsIncluded, framesUsed, totalChars };
}
```
**Tier drives frame count** (`FRAME_LIMITS`): `small:3`, `mid:6`, `frontier:10`. Frames are deduped by id and sorted by `IMPORTANCE_WEIGHT` (critical:4 → deprecated:0) then recency.
**Sections, in order** (each tagged trimmable or never-trimmed):
Identity (never) → Persona (never) → State / I-frames (trim) → Recent changes / P+B-frames (**trim first**) → Active work (trim) → Personal preferences (never) → Recalled memory (only if `recalled.scanSafe`) → Response format (scaffold, gated). **Truncation order** when over `maxSystemChars` (default 32_000): `Recent changes` → `Active work` → `State`.
**Scaffold gating** (`selectScaffold`): only when `taskShape.confidence ≥ confidenceThreshold` (default 0.3) AND `tier !== 'frontier'`. Two matrices: **`COMPRESSION_SCAFFOLDS`** (v4 default — "say less") and **`EXPANSION_SCAFFOLDS`** (v5 — "say more in named sections", for dense instruction-tuned families like Gemma). `draft` and `mixed` shapes never get a scaffold at any tier; frontier always yields null.
`tierForModel(model)` (`model-tier.ts`): `frontier` = `claude-opus*`; `small` = `gemma-4-`, `qwen3*`, `llama-3`, `llama-4`; `mid` = `claude-sonnet*`, `claude-haiku*`, and **unknown models default to `mid`**.
---
## 10. Behavioral spec (`behavioral-spec.ts`)
`BEHAVIORAL_SPEC` (version **'3.0'**) is the core agent rulebook, split into 5 named sections, with a backward-compatible `.rules` getter that concatenates them:
| Section | Content |
|---|---|
| `coreLoop` | The 5-step internal process: **RECALL → ASSESS → ACT → LEARN → RESPOND**, plus two `=== CRITICAL ===` blocks: **Memory Conflict Protocol** (never blindly accept a contradiction; surface both; update only after confirmation) and **Verification Before Completion** (state a checkable "done" condition, run the check this turn, quote real output). |
| `qualityRules` | Anti-hallucination discipline, structured-output guidance, context grounding, contextual professional disclaimers. |
| `behavioralRules` | Memory-first, tool intelligence, narration heuristics, error recovery, planning for complex tasks. |
| `workPatterns` | Drafting-from-context, decision compression, research-in-context recipes. |
| `intelligenceDefaults` | The `# TOOLS` reference — web, memory (two minds: workspace + personal), system, git, documents, connectors (148+ catalog), skills (incl. capability-acquisition + skill-distillation), sub-agents, planning, workflow composition. |
`buildActiveBehavioralSpec(overrides)` returns the same shape with per-section overrides applied (self-evolution deploy path); the chat route uses `server.activeBehavioralSpec ?? BEHAVIORAL_SPEC`. `COMPACTION_PROMPT` is exported for context-window summarization (8 required sections; text-only, no tool calls).
---
## 11. Cost tracking (`cost-tracker.ts`)
`CostTracker` accumulates `UsageEntry[]` and computes `UsageStats`:
```ts
interface UsageStats {
totalInputTokens; totalOutputTokens; estimatedCost; turns;
byModel: Record<string, { input; output; cost }>;
}
```
- `DEFAULT_MODEL_PRICING` (per 1K tokens): `claude-sonnet-4-6` 0.003/0.015, `claude-haiku-3-5` 0.00025/0.00125, `claude-opus-4-6` 0.015/0.075 (+ dated aliases). `calculateCost` falls back to **Sonnet pricing** for unknown models.
- `setBudget(dailyUsd, mode)` with `mode: 'soft' | 'hard'`. `checkBudget()` returns `false` in soft mode when over budget, or **throws `BudgetExceededError`** in hard mode.
- `getDailyTotal()` is the session estimated cost (proxy for daily); `getWorkspaceCost(id)` filters by `workspaceId`.
---
## 12. Turn tracing (`turn-context.ts`)
H-AUDIT-1 contract: a `turnId` (UUID v4) is generated **once** at chat-route turn entry (`generateTurnId()`) and threaded **explicitly** as `turnId?: string` through every stage (agent-loop → orchestrator recall → search → prompt-assembler → cognify → tool calls). No globals/AsyncLocalStorage — propagation is tsc-verifiable. `logTurnEvent(turnId, payload)` is silent when `turnId` is undefined; tests can `startTurnCapture()` / `stopTurnCapture()`.
---
## 13. Production wiring (`packages/server/src/local/routes/chat.ts`)
`POST /api/chat` (SSE) assembles the full prompt stack in this order, then calls `runAgentLoop` (via `server.agentRunner ?? runAgentLoop`):
1. `userSystemPrompt` (highest priority, if set)
2. `assembled?.system ?? orch.buildSystemPrompt()` (identity + self-awareness + preloaded context, or the PromptAssembler output)
3. `# About the User` (profile: name/role/company/industry/writing-style/brand)
4. `# Who You Are` + `## Your Runtime` (date/time/platform/shell/workspace/session facts)
5. `activeSpec.rules` (behavioral spec, with self-evolution overrides)
6. Loaded skills section
7. `Workspace Now` structured state block
8. `# User Corrections` (actionable improvement signals)
9. `composePersonaPrompt(prompt, persona, undefined, workspaceTone)` — wraps everything with the active persona (override > workspace default) + tone
10. `## Response shape` (the assembler's `responseScaffold`, if any)
Per-turn memory recall (`sessionOrch.recallMemory(query)`) is concatenated as `recalledContext`. Persistent agents set `maxTurns: 200`. After the loop, `commitSurfacedSignals()` and `autoSaveFromExchange(message, result.content)` run.
---
## 14. One agent turn, end to end
```mermaid
flowchart TD
A[User message arrives at POST /api/chat] --> B[generateTurnId UUID v4]
B --> C[Orchestrator.recallMemory query<br/>catch-up vs semantic + injection scan + temporary/deprecated filter]
C --> D[buildSystemPrompt / buildAssembledPrompt<br/>identity + self-awareness + preloaded context]
D --> E[Route layers: profile + runtime facts + behavioral-spec.rules<br/>+ skills + Workspace Now + corrections]
E --> F[composePersonaPrompt<br/>persona systemPrompt + WRITE_DISCIPLINE if writable + tone]
F --> G[runAgentLoop config<br/>system + history messages, tools to OpenAI schema]
G --> H{turn < maxTurns?}
H -- no --> Z[Return: preservedAnswer ?? accumulated ?? Max-turns message]
H -- yes --> I[POST litellmUrl/chat/completions<br/>signal = client-abort + 300s timeout]
I -- network reject --> R1[handleNetworkError: backoff, turn--, retry]
I -- !ok 429/5xx --> R2[handleNonOkResponse: Retry-After / backoff, turn--, retry]
R1 --> H
R2 --> H
I -- ok --> J[Parse stream / json: content + tool_calls + usage]
J --> K{tool_calls present?}
K -- no --> L[maybeFireCompletionGate<br/>D3 verification -> D4 phantom-write -> D1 skill-distillation]
L -- a gate fired --> M[push assistant + corrective user directive] --> H
L -- none fired --> Y[Return AgentResponse content, toolsUsed, usage]
K -- yes --> N[push assistant message with tool_calls]
N --> O[for each tool_call: executeToolCall]
O --> P[11-step chain:<br/>parse args - onToolUse - governance block - pre:tool - pre:memory-write<br/>- LoopGuard.check - execute - scanForInjection - onToolResult - post hooks]
P --> Q[push role:tool result message] --> H
Y --> S[commitSurfacedSignals + autoSaveFromExchange]
Z --> S
```
---
## 15. Frontend rebuild — must-know facts
- The frontend never talks to the agent loop directly — it hits the **HTTP/SSE** surface (`POST /api/chat`, agent-run routes). The runtime returns `{content, toolsUsed, usage:{inputTokens, outputTokens}}`; the route streams tokens via `onToken` and tool events via `onToolUse`/`onToolResult`.
- **Personas are 22, not 13/17.** Build any picker from `listPersonas()` output (id/name/icon/tagline/bestFor/wontDo/failurePatterns). `isReadOnly` personas (`planner`, `verifier`) must surface "read-only" affordances; `coordinator` is feature-gated.
- **Gates change UX:** after a "final" answer the loop may inject ONE more turn (verification re-run, forced file write, or skill authoring). The UI should expect a brief continuation rather than treating the first no-tool answer as terminal.
- **Tool results are sanitized** before the UI ever sees them (`scanForInjection` step 8) — a `[SECURITY] … sanitized.` string means injection was caught, not a backend bug.
- **Cost/usage** comes from `usage` on the response plus `CostTracker` (`DEFAULT_MODEL_PRICING`, soft/hard budget). Surface token/$ estimates from there; unknown models bill at Sonnet rates.

337
docs/backend-map/sections/05b-subsystem-memory.md Normal file
View File

@@ -0,0 +1,337 @@
# 05b · Subsystem: Persistent Memory Engine (the MOAT)
**Purpose.** This is Waggle's durable, per-workspace memory substrate — the thing that makes agents remember across sessions. It is a single SQLite database (one file per "mind") layered into Identity → Awareness → Frames → Knowledge Graph, with **hybrid search** (vector + keyword fused), relevance scoring, index reconciliation, and a tiered embedding provider chain. Memory is **WRITTEN** by the `CognifyPipeline` (one frame per turn + entity/relation extraction) and **RECALLED** by `CombinedRetrieval` (workspace + personal + optional KVARK merge). For a frontend rebuild, treat this as a **contract**: you never touch SQLite directly — you call the sidecar HTTP routes that wrap these classes, and you render the typed shapes documented below.
> Grounding: every type, field, table, and constant below is quoted from
> `packages/hive-mind-core/src/mind/*` and `packages/agent/src/{cognify,combined-retrieval,memory-linker}.ts`.
> Where a fact is NOT in those files (e.g. exact HTTP route paths), it is flagged explicitly.
---
## 1. Mental model — five layers in one SQLite file
A "mind" is one `better-sqlite3` database (with the `sqlite-vec` extension loaded for the vector table). The schema (`schema.ts`, `SCHEMA_VERSION = '1'`) defines these layers. Each layer is a TypeScript class wrapping prepared SQL statements — there is **no ORM**.
| Layer | Class (file) | Table(s) | Cardinality | Role |
|---|---|---|---|---|
| 0 — Identity | `IdentityLayer` (`identity.ts`) | `identity` | exactly 1 row (`CHECK (id = 1)`) | Who the agent is: name/role/department/personality/capabilities/system_prompt. `<500 tokens` budget. |
| 1 — Awareness | `AwarenessLayer` (`awareness.ts`) | `awareness` | `MAX_ITEMS = 10` active | Short-term working state: active tasks, recent actions, pending items, context flags. Items can expire. |
| 2 — Frames | `FrameStore` (`frames.ts`) | `memory_frames` (+ `memory_frames_fts`, `memory_frames_vec`) | unbounded | The long-term memory store. I/P/B frame types grouped by session (`gop_id`). |
| 3 — Knowledge Graph | `KnowledgeGraph` (`knowledge.ts`) | `knowledge_entities`, `knowledge_relations` | unbounded | Entities + typed relations with temporal validity (bitemporal). |
| — Sessions | `SessionStore` (`sessions.ts`) | `sessions` | unbounded | Maps `gop_id` (Group-Of-Pictures id) → project. Parent of frames. |
The "GOP" naming (I-frame / P-frame / B-frame, `gop_id`, `t`) is borrowed from video compression: an **I-frame** is a self-contained keyframe (full state), a **P-frame** is a delta/update against its base I-frame, a **B-frame** is a bidirectional cross-reference frame. `t` is a per-session monotonic sequence number (`nextT` = `MAX(t)+1` for that `gop_id`).
```mermaid
graph TD
subgraph MindDB["One SQLite file per workspace (a 'mind')"]
ID[identity · 1 row]
AW["awareness · ≤10 items"]
SESS[sessions]
MF[memory_frames]
FTS["memory_frames_fts (FTS5)"]
VEC["memory_frames_vec (vec0 · float[1024])"]
KE[knowledge_entities]
KR[knowledge_relations]
end
SESS -->|gop_id FK| MF
MF -->|rowid = id| FTS
MF -->|rowid = id| VEC
KE -->|source_id / target_id| KR
```
---
## 2. Layer 2 — Frames (`FrameStore`)
The heart of the store. A `MemoryFrame` is:
| Field | Type | Notes |
|---|---|---|
| `id` | `number` | PK / rowid; same id used in FTS + vec tables |
| `frame_type` | `'I' \| 'P' \| 'B'` | `FrameType` |
| `gop_id` | `string` | session id this frame belongs to |
| `t` | `number` | per-`gop_id` monotonic sequence (`nextT`) |
| `base_frame_id` | `number \| null` | P/B frames point at their base I-frame |
| `content` | `string` | the actual text |
| `importance` | `'critical' \| 'important' \| 'normal' \| 'temporary' \| 'deprecated'` | `Importance` |
| `source` | `'user_stated' \| 'tool_verified' \| 'agent_inferred' \| 'import' \| 'system' \| 'personal' \| 'workspace' \| 'team_sync'` | `FrameSource` (TS union is wider than the DB `CHECK`, which only allows the first five) |
| `access_count` | `number` | incremented by `touch()` on every recall/dup-hit |
| `created_at` | `string` | ISO; harvest path can override to preserve source timestamp |
| `last_accessed` | `string` | ISO; drives temporal scoring |
### Write methods
- `createIFrame(gopId, content, importance='normal', source='user_stated', createdAt?)` — **dedup-guarded**: calls `findDuplicate(content)` first; if an identical frame exists it `touch()`es it and returns it instead of inserting. `createdAt` is honored only if it passes `isValidIsoTimestamp` (strict ISO-8601 with `T` + timezone) — used by the harvest path so imported frames keep their original timestamp.
- `createPFrame(gopId, content, baseFrameId, …)` — a delta against an I-frame.
- `createBFrame(gopId, content, baseFrameId, referencedFrameIds[])` — stores `{description, references}` as JSON in `content`.
Every create also runs `indexFts(frame)` to mirror content into the FTS5 table. **Vector indexing is NOT done here** — it happens in `HybridSearch.indexFrame()`, called by `CognifyPipeline` (see §6).
### Dedup (`findDuplicate`) — important quirks for the frontend
- Hash = `SHA-256( stripHmPrefix(content).trim() )`.
- `stripHmPrefix` removes a leading `[hm session:… src:… event:…] ` provenance prefix so two captures of the same turn from different sources collapse into one frame.
- **Only the last 500 frames** are scanned (cost bound). Dedup is best-effort beyond the recency window.
### Read / list methods
| Method | Returns |
|---|---|
| `getById(id)` | one frame |
| `getLatestIFrame(gopId)` | newest I-frame in a session |
| `getPFramesSinceLastI(gopId)` | P-frames after the latest I-frame |
| `getGopFrames(gopId)` | all frames in a session, `t ASC` |
| `reconstructState(gopId)` | `{ iframe, pframes }` — current state = latest I + its P-deltas |
| `getRecent(limit=50)` / `list({limit})` | newest frames, `id DESC` |
| `getRecentFiltered(limit, since?, until?)` | F20: date-bounded recent frames |
| `getStats()` | `{ total, byType, byImportance }` |
| `update(id, content, importance?)` | updates main + FTS + clears vec entry |
| `delete(id)` | removes from main + FTS + vec + `kg_entity_frames` and nulls referring `base_frame_id` |
### Compaction (`compact(maxTempAgeDays=30, maxDeprecatedAgeDays=90)`)
Maintenance op: deletes old `temporary` and `deprecated` frames, and for any `gop_id` with >10 P-frames merges all-but-the-5-most-recent P-frames into their I-frame (joined with `\n---\n`). Returns `{ temporaryPruned, deprecatedPruned, pframesMerged }`.
---
## 3. Hybrid Search (`HybridSearch`) — vector + keyword fusion
`search(query, options)` runs **keyword and vector searches in parallel**, fuses with **Reciprocal Rank Fusion (RRF)**, then multiplies by a relevance score.
`SearchOptions`:
| Field | Type | Default |
|---|---|---|
| `limit` | `number` | `20` |
| `gopId` | `string?` | (scope to one session) |
| `profile` | `'balanced' \| 'recent' \| 'important' \| 'connected'` | `'balanced'` |
| `context` | `ScoringContext` | `{}` |
| `since` / `until` | `string?` (ISO) | temporal filter on `created_at` |
`SearchResult`:
| Field | Type | Meaning |
|---|---|---|
| `frame` | `MemoryFrame` | the hit |
| `rrfScore` | `number` | fused rank score |
| `relevanceScore` | `number` | from `computeRelevance` |
| `finalScore` | `number` | `rrfScore * relevanceScore` — the sort key |
### Fusion algorithm (the actual constants)
1. Run `keywordSearch(query, limit*2, gopId)` and `vectorSearch(query, limit*2, gopId)` in parallel; each returns an ordered `number[]` of frame ids.
2. RRF with `RRF_K = 60`: each id accrues `1 / (RRF_K + rank)` from each list.
3. Fetch the union of frame ids (applying `since`/`until` filters here), compute `relevanceScore`, set `finalScore = rrfScore * relevanceScore`, sort desc, slice to `limit`.
### Keyword path (FTS5) — recall tuning
- Query is tokenized, punctuation stripped, a built-in **stop-word list** removed, tokens shorter than 3 chars dropped, then OR-joined (`"foo" OR "bar"`) for recall. FTS5 `ORDER BY rank`.
- On FTS5 parse error it falls back to `likeFallbackSearch` — OR-ed `LIKE … ESCAPE '\'` over `content` (parameterized, metachars escaped). This guarantees a query never returns a false "no memory found" because the user typed an FTS5 operator.
### Vector path (`sqlite-vec` vec0)
- `vectorSearch` embeds the query (`embedder.embed`), converts the `Float32Array` to a blob, and does a `MATCH ? AND k = ?` KNN query on `memory_frames_vec` (`ORDER BY distance`).
- When `gopId` is set it over-fetches (`k = limit*3`) then filters by session.
- The vec table is `float[1024]` — embeddings MUST be 1024-dim (see §7).
- All vec operations are wrapped in `try/catch` returning `[]` — if the `sqlite-vec` extension or table is absent, search silently degrades to keyword-only.
### Indexing (called by cognify, not by FrameStore)
- `indexFrame(frameId, content)` — embed + `INSERT INTO memory_frames_vec`. (rowid is inlined as a SQL literal because vec0 doesn't accept a parameterized rowid.)
- `indexFramesBatch(frames[])` — batch embed + transactional insert.
---
## 4. Relevance scoring (`scoring.ts`)
`computeRelevance(frame, weights, context)` = weighted sum of four sub-scores. Profiles pick the weights:
| Profile | temporal | popularity | contextual | importance |
|---|---|---|---|---|
| `balanced` | 0.4 | 0.2 | 0.2 | 0.2 |
| `recent` | 0.6 | 0.1 | 0.2 | 0.1 |
| `important` | 0.1 | 0.1 | 0.2 | 0.6 |
| `connected` | 0.1 | 0.1 | 0.6 | 0.2 |
Sub-scores:
- **temporal** — `1.0` if `last_accessed` within `RECENCY_BOOST_DAYS = 7`, else exponential decay with `HALF_LIFE_DAYS = 30`.
- **popularity** — `1 + log10(1 + access_count) * 0.1`.
- **contextual** — graph proximity: distance 0→1.0, 1→0.7, 2→0.4, 3→0.2, else 0 (needs `context.graphDistances`, a `Map<frameId, BFS-distance>`).
- **importance** — `critical 2.0 / important 1.5 / normal 1.0 / temporary 0.7 / deprecated 0.3`.
`ScoringContext = { recentEntityIds?: number[]; graphDistances?: Map<number, number> }`.
---
## 5. Knowledge Graph (`KnowledgeGraph`)
Bitemporal entity-relation store. `Entity` and `Relation` both carry `valid_from` / `valid_to` (null = currently valid) plus `recorded_at`.
`Entity`: `{ id, entity_type, name, properties(JSON string), valid_from, valid_to, recorded_at }`
`Relation`: `{ id, source_id, target_id, relation_type, confidence(REAL), properties(JSON), valid_from, valid_to, recorded_at }`
Key methods:
| Method | Purpose |
|---|---|
| `createEntity(type, name, props, temporal?)` | validates against optional schema, inserts |
| `getEntitiesByType(type, limit=500)` / `getEntities(limit, offset)` | active entities (`valid_to IS NULL`) |
| `searchEntities(query, limit=100)` | `name LIKE` (escaped) |
| `getEntityTypeCounts()` / `getEntityCount()` | dashboard counts without full fetch |
| `getEntitiesValidAt(isoTime)` | time-travel: entities valid at a past instant |
| `createRelation(src, tgt, type, confidence=1.0, props)` | validated insert |
| `getRelationsFrom(id, type?)` / `getRelationsTo(id, type?)` | adjacency |
| `retireEntity(id)` / `retireRelation(id)` | sets `valid_to = now` (soft-delete, never hard delete) |
| `traverse(startId, relationType, maxDepth)` | BFS returning reached entities |
| `bfsDistances(startId, maxDepth)` | `Map<entityId, distance>` — feeds the `contextual` score |
Optional `setValidationSchema(ValidationSchema)` enforces required properties per entity-type and an allowed-relations list (throws on violation). Without a schema, all writes pass.
---
## 6. WRITE path — `CognifyPipeline` (`cognify.ts`)
`cognify(content, importance='normal', gopId?, turnId?)` is the canonical "remember this turn" call. Steps:
1. **Ensure session** — `sessions.ensureActive()` (transaction-wrapped to avoid the twin-session race).
2. **Save frame** — if a latest I-frame exists for the session → `createPFrame` (delta), else `createIFrame` (keyframe). Dedup applies inside the FrameStore.
3. **Extract entities** — `extractEntities(content.slice(0, 10_000))` (from `entity-extractor.ts`).
4. **Upsert entities** into the KG (skip if same type+name exists; cached per-type to avoid N queries).
5. **Co-occurrence relations** — `co_occurs_with` (confidence 0.8) between every entity pair found in the same text.
6. **Semantic relations** — `extractRelations` → typed relations (`led_by`, `reports_to`, `depends_on`, …) matched back to KG entities.
7. **Vector index** — `search.indexFrame(frame.id, content)`.
8. **Optional linking** — if `enableLinking`, `MemoryLinker.findRelated(content)` returns related frames (self excluded).
`CognifyResult = { frameId, entitiesExtracted, relationsCreated, relatedFrames? }`.
Other entry points: `cognifyFrame(frameId)` (re-process one imported frame — used post-harvest) and `cognifyBatch(frameIds[])` (sequential, so each frame's new entities can link to the next).
### `MemoryLinker` (`memory-linker.ts`)
Thin wrapper over `HybridSearch.search`. `findRelated(content, limit=5)` returns `MemoryLink[] = { frameId, content, score }`, filtered by a `threshold` (default `0.1` on `finalScore`).
```mermaid
flowchart TD
A["agent turn / harvest / MCP save_memory"] --> B["CognifyPipeline.cognify(content)"]
B --> C["SessionStore.ensureActive() → gop_id"]
B --> D{latest I-frame exists?}
D -- no --> E["FrameStore.createIFrame (dedup-guarded)"]
D -- yes --> F["FrameStore.createPFrame"]
E --> G["indexFts (FTS5)"]
F --> G
B --> H["extractEntities → KnowledgeGraph.upsert"]
H --> I["co_occurs_with + semantic relations"]
B --> J["HybridSearch.indexFrame → memory_frames_vec (1024-dim)"]
B --> K["MemoryLinker.findRelated (optional)"]
```
---
## 7. Embeddings — provider chain (`embedding-provider.ts`)
`createEmbeddingProvider(config?)` returns an `EmbeddingProviderInstance` (implements the `Embedder` interface: `embed`, `embedBatch`, `dimensions`). Default `targetDimensions = 1024` — matches the `vec0 float[1024]` table.
**Auto fallback chain** (`provider: 'auto'`): `inprocess → ollama → voyage → openai → mock`. Each is probed with a 1024-dim test embedding; the first that succeeds becomes active.
| Provider (`EmbeddingProviderType`) | Default model | Needs |
|---|---|---|
| `inprocess` | `Xenova/all-MiniLM-L6-v2` (Transformers.js) | nothing — fully local |
| `ollama` | `nomic-embed-text` | local Ollama server |
| `voyage` | `voyage-3-lite` | `voyage.apiKey` (from Vault) |
| `openai` | `text-embedding-3-small` | `openai.apiKey` |
| `litellm` | `text-embedding` | `litellm.url` |
| `mock` | `deterministic-mock` | always available; **semantically meaningless** (last resort) |
**Tier gating**: provider availability is gated by `TIER_CAPABILITIES[tier].embeddingProviders`; `embeddingQuotaPerMonth` is enforced per `user_id` per month in the `embedding_usage` table (`-1` = unlimited). Quota throws `EmbeddingQuotaExceededError` (carries `tier/quota/current/upgradeUrl`); over-tier provider request throws `TierError`. `WAGGLE_EVAL_MODE=1` disables all gating (eval harness only). `getStatus()` / `getQuotaStatus()` / `reprobe()` expose state for a settings UI.
> Frontend note: if the active provider is `mock`, surface a "semantic search degraded" warning — `getStatus().activeProvider === 'mock'` and `lastError` tell you. Mismatched embedder dimensions would break the vec table, so the provider hard-asserts 1024 on probe.
---
## 8. RECALL path — `CombinedRetrieval` (`combined-retrieval.ts`)
The merge engine the agent calls to answer "what do I know about X". Merges **workspace** + **personal** memory and optionally **KVARK** enterprise search. Pure data in / out (no formatting).
`search(query, opts)` → `CombinedRetrievalResult`:
| Field | Type |
|---|---|
| `query` | `string` |
| `workspaceResults` / `personalResults` / `kvarkResults` | `CombinedResult[]` |
| `kvarkAvailable` | `boolean` |
| `kvarkSkipped` | `boolean` (available but coverage was sufficient) |
| `kvarkError?` | `string` |
| `hasConflict` | `boolean` |
| `conflictNote?` | `string` |
`CombinedResult = { content, source: 'workspace'|'personal'|'kvark', attribution, score, metadata }` where `attribution` is a human tag like `[workspace memory]` / `[personal memory]`, and `metadata` carries `frameId/frameType/importance` (memory) or `documentId/documentType` (KVARK).
`CombinedSearchOptions = { limit=10, profile='balanced', scope: 'all'|'personal'|'workspace', turnId? }`.
**KVARK gating logic** (`shouldQueryKvark`): KVARK is queried only when a client exists, `scope==='all'`, AND local coverage is insufficient — `hasSufficientLocalCoverage` = fewer than `LOCAL_COVERAGE_MIN_COUNT = 3` results with `score ≥ LOCAL_COVERAGE_SCORE_THRESHOLD = 0.7`. KVARK failures degrade gracefully (local results preserved, `kvarkError` set).
**Conflict detection** (`detectConflict`): if both workspace and KVARK have strong results (`score ≥ CONFLICT_SCORE_THRESHOLD = 0.6`) and their top-3 texts disagree on status polarity (`POSITIVE_STATUS` words like *approved/selected* vs `NEGATIVE_STATUS` like *rejected/cancelled*), it returns a human-readable `conflictNote` for the UI to surface ("these sources may be out of sync").
```mermaid
flowchart TD
Q["CombinedRetrieval.search(query, scope)"] --> WS["searchWorkspace → HybridSearch"]
Q --> PS["searchPersonal → HybridSearch"]
WS --> LC{"sufficient local coverage?\n≥3 results @ score ≥0.7"}
PS --> LC
LC -- yes / scope≠all / no client --> OUT["return local results, kvarkSkipped"]
LC -- no --> KV["searchKvark (tier-gated, graceful fail)"]
KV --> CF["detectConflict(workspace, kvark)"]
CF --> OUT2["return merged + hasConflict/conflictNote"]
```
---
## 9. Reconciliation & integrity (`reconcile.ts`)
A crash between frame insert and FTS/vec indexing leaves frames that exist but aren't searchable. The reconcile functions repair this (idempotent, cron-friendly):
- `reconcileFtsIndex(db)` — re-index frames missing from FTS5 (no embedder needed).
- `reconcileVecIndex(db, embedder)` — embed + re-index frames missing from the vec table (batches of 50).
- `cleanOrphanFts(db)` / `cleanOrphanVectors(db)` — drop FTS/vec rows whose frame was deleted.
- `reconcileIndexes(db, embedder?)` — runs all of the above; FTS-only if no embedder. Returns `{ ftsFixed, vecFixed }`.
---
## 10. Sessions (`SessionStore`)
`Session = { id, gop_id, project_id, status: 'active'|'closed'|'archived', started_at, ended_at, summary }`. `gop_id` format: `session:<ISO>:<rand6>`.
| Method | Purpose |
|---|---|
| `create(projectId?)` | new timestamped session |
| `ensureActive(projectId?)` | **transaction-wrapped** — returns existing active session or creates one (prevents twin-session race; used by cognify) |
| `ensure(gopId, …)` | idempotent named session (e.g. a stable `harvest` parent) |
| `close(gopId, summary?)` / `archive(gopId)` | lifecycle |
| `getByProject` / `getActive` / `getByGopId` | queries |
---
## 11. Schema reference (DDL, verbatim from `schema.ts`)
`SCHEMA_VERSION = '1'`. Tables relevant to this subsystem:
| Table | Key columns / constraints |
|---|---|
| `identity` | `id CHECK (id = 1)` (single row), name/role/department/personality/capabilities/system_prompt, created_at, updated_at |
| `awareness` | category `CHECK IN ('task','action','pending','flag')`, content, priority, `metadata` (JSON), created_at, expires_at |
| `sessions` | `gop_id UNIQUE`, project_id, status `CHECK IN ('active','closed','archived')`, started_at, ended_at, summary; index `(project_id, started_at)` |
| `memory_frames` | frame_type `CHECK IN ('I','P','B')`, gop_id (FK→sessions), t, base_frame_id (self-FK), content, importance `CHECK IN (critical/important/normal/temporary/deprecated)`, source `CHECK IN (user_stated/tool_verified/agent_inferred/import/system)`, access_count, created_at, last_accessed; indexes on (gop_id,t),(frame_type,gop_id),(base_frame_id) |
| `memory_frames_fts` | `CREATE VIRTUAL TABLE … USING fts5(content, content_rowid='id', tokenize='porter unicode61')` |
| `memory_frames_vec` | `CREATE VIRTUAL TABLE … USING vec0(embedding float[1024])` (separate `VEC_TABLE_SQL`, requires `sqlite-vec`) |
| `knowledge_entities` | entity_type, name, properties(JSON), valid_from, valid_to, recorded_at; indexes on type and name |
| `knowledge_relations` | source_id/target_id (FK→entities), relation_type, confidence(REAL), properties(JSON), valid_from, valid_to, recorded_at; indexes on (source_id,relation_type),(target_id,relation_type) |
| `embedding_usage` | (in `embedding-provider.ts`) user_id, year_month, count, updated_at; `UNIQUE(user_id, year_month)` — monthly quota counter |
| `harvest_sources` | source UNIQUE, display_name, source_path, last_synced_at, items_imported, frames_created, auto_sync, sync_interval_hours, last_content_hash, created_at |
> Note: the TS `FrameSource` union (`frames.ts`) includes `'personal' | 'workspace' | 'team_sync'` which the DB `CHECK` does **not** list — those extra sources are application-level and not written through the constrained column path.
---
## 12. HTTP surface (how the frontend reaches this)
These engine classes are **server-side only**; the frontend talks to the Fastify sidecar and the `hive-mind` MCP server, not to SQLite. The exact route paths are defined in `packages/server/src` (outside this section's read scope) — **do not invent them**. What this subsystem guarantees, and the MCP tool names that wrap it (from the `hive-mind` MCP server, visible in this environment), are:
| MCP tool | Wraps |
|---|---|
| `recall_memory` | `CombinedRetrieval.search` / `HybridSearch.search` |
| `save_memory` | `CognifyPipeline.cognify` |
| `save_entity` / `create_relation` / `search_entities` | `KnowledgeGraph` |
| `get_identity` / `set_identity` | `IdentityLayer` |
| `get_awareness` / `set_awareness` / `clear_awareness` | `AwarenessLayer` |
| `cleanup_frames` / `cleanup_entities` | `FrameStore.compact` / KG retire |
| `create_workspace` / `list_workspaces` | per-mind DB lifecycle |
| `harvest_import` / `harvest_sources` / `ingest_source` | harvest → `cognifyFrame`/`cognifyBatch` |
For the precise sidecar REST routes (method + path), consult the server-routes section of this backend map — they are the authoritative contract the Lovable frontend will call.

261
docs/backend-map/sections/05c-subsystem-harvest.md Normal file
View File

@@ -0,0 +1,261 @@
# Subsystem 05c — Harvest (Conversation & File Ingestion Pipeline)
## Purpose
Harvest is the ingestion subsystem that turns external AI-chat exports (ChatGPT, Claude, Gemini, Perplexity) and local files/URLs (Markdown, plain text, PDF, web pages, the Claude Code `~/.claude` directory) into normalized **`UniversalImportItem`** objects, then persists them as memory **frames** in the per-workspace "mind" (SQLite). Source code lives at `packages/hive-mind-core/src/harvest/**`; the HTTP surface that the frontend calls lives at `packages/server/src/local/routes/harvest.ts`. This is the "Memory Harvest" feature surfaced in the Memory app's **HarvestTab** (`apps/web/src/components/os/apps/memory/HarvestTab.tsx`).
> Important architectural note for the rebuild: the *committed* harvest route (`POST /api/harvest/commit`) does **NOT** run the 4-pass LLM distillation pipeline (`HarvestPipeline`). It parses with an adapter and writes the raw normalized items directly to frames, then runs a separate **cognify** + **wiki-compile** post-step. The `HarvestPipeline` class (classify → extract → synthesize → dedup) exists and is exported, but the production commit route bypasses it. Both flows are documented below — build your UI against the route contract, not the pipeline class.
---
## 1. Core Data Shapes
These are the exact TypeScript types the frontend will see across the wire (from `packages/hive-mind-core/src/harvest/types.ts`).
### `UniversalImportItem` — the normalized unit every adapter produces
| Field | Type | Notes |
|---|---|---|
| `id` | `string` | `randomUUID()` generated per item by the adapter |
| `source` | `ImportSourceType` | which source it came from (see union below) |
| `type` | `ImportItemType` | `conversation` \| `memory` \| `instruction` \| `preference` \| `artifact` \| `rule` \| `decision` \| `document` |
| `title` | `string` | human-readable label; `'Untitled'` fallback |
| `content` | `string` | flattened text. For conversations: `messages.map(m => \`${m.role}: ${m.text}\`).join('\n\n')` |
| `messages?` | `ConversationMessage[]` | present only for conversation-type items |
| `timestamp` | `string` | ISO-8601; source-original where available, else `new Date().toISOString()` |
| `metadata` | `Record<string, unknown>` | adapter-specific (conversationId, messageCount, filePath, etc.) |
### `ConversationMessage`
| Field | Type |
|---|---|
| `role` | `'user' \| 'assistant' \| 'system'` |
| `text` | `string` |
| `timestamp?` | `string` |
### `ImportSourceType` (full union)
`chatgpt` · `claude` · `claude-code` · `claude-desktop` · `gemini` · `google-ai-studio` · `perplexity` · `grok` · `cursor` · `copilot` · `manus` · `genspark` · `qwen` · `minimax` · `z-ai` · `openclaw` · `cowork` · `elevenlabs` · `google-flow` · `markdown` · `plaintext` · `pdf` · `url` · `unknown`
> Only `chatgpt`, `claude`/`claude-desktop`, `claude-code`, `gemini`/`google-ai-studio` have dedicated adapters wired in the route's `getAdapter()`; everything else falls through to `UniversalAdapter`.
### `HarvestSource` — the source-tracking row (GET /api/harvest/sources returns these)
| Field | Type | Notes |
|---|---|---|
| `id` | `number` | autoincrement PK |
| `source` | `ImportSourceType` | UNIQUE |
| `displayName` | `string` | e.g. `"ChatGPT"`, `"Claude"` |
| `sourcePath` | `string \| null` | local dir, if filesystem source |
| `lastSyncedAt` | `string \| null` | ISO datetime |
| `itemsImported` | `number` | cumulative |
| `framesCreated` | `number` | cumulative |
| `autoSync` | `boolean` | stored as `0/1` integer in SQLite |
| `syncIntervalHours` | `number` | default `24` |
| `lastContentHash` | `string \| null` | dedup-skip digest (see §5) |
| `createdAt` | `string` | ISO datetime |
### `HarvestRun` — one commit invocation's lifecycle (GET /api/harvest/runs returns these)
| Field | Type | Notes |
|---|---|---|
| `id` | `number` | autoincrement PK |
| `source` | `ImportSourceType` | |
| `status` | `'running' \| 'completed' \| 'failed' \| 'abandoned'` | |
| `totalItems` | `number` | |
| `itemsSaved` | `number` | updated via heartbeat every 10 frames |
| `startedAt` / `updatedAt` / `finishedAt` | `string` / `string` / `string\|null` | ISO datetimes |
| `errorMessage` | `string \| null` | truncated to 2000 chars on failure |
| `inputCachePath` | `string \| null` | path to cached input JSON for resume |
### `DistilledKnowledge` — output of the `HarvestPipeline` (NOT used by the commit route)
| Field | Type | Notes |
|---|---|---|
| `targetLayer` | `'identity' \| 'frame' \| 'kg_entity' \| 'kg_relation' \| 'awareness'` | where the knowledge should land |
| `frameType?` | `'I' \| 'P'` | I = identity-ish, P = procedural |
| `importance` | `'critical' \| 'important' \| 'normal' \| 'temporary'` | |
| `content` | `string` | self-contained memory statement |
| `entities?` / `relations?` | `{name,type}[]` / `{source,target,relation}[]` | KG payload |
| `provenance` | `KnowledgeProvenance` | `{originalSource, importedAt, distillationModel, confidence, pass}` |
---
## 2. Source Adapters
Every adapter implements `SourceAdapter` (`{ sourceType, displayName, parse(input): UniversalImportItem[] }`). Filesystem-scanning adapters additionally implement `FilesystemAdapter` (adds `scan(dirPath)`). All adapters are exported from `packages/hive-mind-core/src/harvest/index.ts` and re-exported via `@waggle/core`.
| Adapter | `sourceType` | `displayName` | Input it expects | What it parses out |
|---|---|---|---|---|
| `ChatGPTAdapter` | `chatgpt` | `ChatGPT` | ChatGPT JSON export — top-level array OR `{conversations:[...]}`. Each conv has a `mapping` object (node-id tree). | Walks `mapping` nodes, filters nodes with `message.content.parts`, sorts by `create_time`, builds messages (skips `system`). Also extracts top-level `user_custom_instructions` (→ `instruction` item) and `memories[]` (→ `memory` items). Per-conv `custom_instructions` folded into metadata. |
| `ClaudeAdapter` | `claude` (also handles `claude-desktop`) | `Claude` | Claude web/desktop JSON export — array OR `{conversations:[...]}`. Messages in `chat_messages` (or `messages`). | Role from `sender==='human'`/`role==='user'`. Text from content blocks (`type:'text'`) or `text`/`content`. Also extracts `projects[].docs[]` → `artifact` items tagged `type:'project_knowledge'`. |
| `ClaudeCodeAdapter` | `claude-code` | `Claude Code` | A **directory path** string (default `~/.claude`). Implements `FilesystemAdapter.scan()`. | Reads disk: `projects/*/memory/*.md` (frontmatter `type`→ImportItemType via MEMORY_TYPE_MAP), `rules/**/*.md` (→`rule`), `plans/*.md` (→`artifact`), `settings.json` (→`preference`), `projects/*/CLAUDE.md` (→`artifact`), `projects/*/.mind/*.md` (→`decision`/`artifact`). Then runs **decision extraction** (regex `DECISION_PATTERNS`) over all items to mint extra `decision` items. |
| `GeminiAdapter` | `gemini` (also handles `google-ai-studio`) | `Gemini` | Google Takeout `{conversations:[...]}`, Gemini API `{history:[...]}`, or bare array. Messages in `messages`/`turns`/`history`. | Role resolved from `role`/`author`/`sender` (`model`/`gemini`→assistant). Text from Gemini `parts:[{text}]` or `text`/`content`. |
| `PerplexityAdapter` | `perplexity` | `Perplexity` | `{threads:[...]}`, `{conversations:[...]}`, `{items:[...]}`, single-thread `{messages:[...]}`, or bare array. | Role from `role`/`author`/`sender`/`type`. **Flattens citations**: per-message `sources`/`citations`/`web_results` appended to text as `"Sources: <url1>, <url2>"`. Sets `metadata.hasCitations`. |
| `MarkdownAdapter` | `markdown` | `Markdown` | A `.md` file path (auto-detected: short, no newline → tries `fs.readFileSync`) OR raw markdown string. | Splits by `#`/`##`/`###` headings (`splitByHeadings`). Each section → one `document` item. Extracts `**bold**` terms as `concept` entities (max 10) into metadata. Caps content at 4000 chars. |
| `PlaintextAdapter` | `plaintext` | `Plain Text` | A `.txt` file path or raw text. | Chunks by paragraphs (`chunkByParagraphs`, ~2000 char chunks) → `document` items titled `"<file> (part N)"`. |
| `PdfAdapter` | `pdf` | `PDF Document` | A PDF **file path** via async `parseFile()` (`parse()` returns `[]`). | Dynamic-imports optional `pdf-parse`; throws a clear "not installed" error if absent. Extracts full text + `getInfo()` (Title/Author/numPages), chunks at ~3000 chars → `document` items with `contentType:'paper'`. |
| `UrlAdapter` | `url` | `Web URL` | Pre-fetched HTML via sync `parse()`, OR a URL via async `fetchAndParse(url)`. | `fetchAndParse` does `fetch()` with 15s timeout + custom User-Agent. `stripHtml` removes script/style/nav/footer/header, converts headings to markdown, strips tags, decodes entities. Short pages → 1 item; long pages split by `#` sections. `contentType:'article'`. |
| `UniversalAdapter` | `unknown` | `Universal (Auto-detect)` | Any string or JSON. **Fallback for all Tier-2 sources** (grok, manus, genspark, qwen, minimax, z-ai, openclaw, cowork, elevenlabs, google-flow). | Heuristic `detectSource()` from content cues. `findConversations()` probes common keys (`conversations`/`chats`/`threads`/`sessions`/`history`/`data`). Text mode splits on headings/`===`/`---`/`Conversation N` and regex-extracts `Speaker: text` turns. Raw JSON with no conversation structure → single `memory` item (capped 50000 chars). |
### Adapter helper notes
- **`raw-types.ts`** provides safe narrowing helpers (`asRecord`, `getString`, `getNumber`, `getArray`, `firstString`) so malformed external JSON degrades to "skip" instead of throwing. All JSON adapters use these — they never cast to `any`.
- **`chunk-utils.ts`** — `chunkByParagraphs(text, maxLen=2000)` splits on blank lines and packs paragraphs up to `maxLen`. Used by plaintext + pdf.
- Adapters always cap `content` at 4000 chars at the item level (route caps again at 10000 — see §4).
---
## 3. The 4-Pass Distillation Pipeline (`HarvestPipeline`)
`packages/hive-mind-core/src/harvest/pipeline.ts`. Accepts `UniversalImportItem[]`, returns `HarvestPipelineResult`. **Not invoked by the production commit route** — provided for callers (MCP tools, CLI, future flows) that want LLM-distilled knowledge instead of raw frames.
| Pass | Name | Model tier | What it does | Prompt |
|---|---|---|---|---|
| 0 | Injection scan | none (local) | Drops any item whose title + first 4KB of content trips `scanForInjection(probe, 'tool_output')`. Logs blocked items into `errors[]`. | — |
| 1 | Classify | `'fast'` (Haiku) | Tags each item `{domain, value, categories}`. `value:'skip'` items are dropped. | `CLASSIFY_PROMPT` |
| 2 | Extract | `'accurate'` (Sonnet) | Pulls `{decisions, preferences, facts, knowledge, entities, relations}` per item. "Only what the USER stated." | `EXTRACT_PROMPT` |
| 3 | Synthesize | `'accurate'` (Sonnet) | Converts extractions into `DistilledKnowledge` frames (`targetLayer`, `frameType`, `importance`, `content`, `confidence`). | `SYNTHESIZE_PROMPT` |
| 4 | Dedup | none (local) | `dedup()` removes duplicates/near-dups vs existing frame contents (see §5). | — |
**Mechanics:** items are batched (`BATCH_SIZE=20`, configurable), run with a tumbling-window concurrency cap (`CONCURRENCY_CAP=3`). `llmCall(prompt, 'fast'|'accurate')` is injected by the caller. `classifyFailureFallback` defaults to `'skip'` (drop the batch) vs legacy `'pass-through-medium'`. Per-item JSON budget in synthesize is `PER_ITEM_BUDGET=1200`. The pipeline does **not** persist anything itself — `framesSaved/entitiesCreated/relationsCreated/costUsd` are returned as `0` for the caller to fill in.
`HarvestPipelineResult` reports: `itemsReceived`, `itemsClassified`, `itemsSkipped`, `itemsExtracted`, `knowledgeDistilled[]`, `duplicatesSkipped`, `identityUpdates`, `errors[]`, `durationMs`.
---
## 4. The Production Commit Flow (what the frontend actually triggers)
`POST /api/harvest/commit` is the real ingest path. It does NOT call `HarvestPipeline`. Sequence:
1. Resolve adapter via `getAdapter(source)`.
2. Parse: `adapter.parse(data)` — OR for `{scanLocal:true}` requests, `adapter.scan(defaultDir)` (only `claude-code` supports this; default dir `~/.claude`).
3. **Content-hash skip**: `harvestSetHash(items)` vs `HarvestSource.lastContentHash`. If unchanged → returns `{saved:0, skipped:true}` immediately.
4. **Resumability**: write input JSON to `dataDir/harvest-cache/<uuid>.json` (atomic tmp+rename) and open a `HarvestRunStore` row (`status:'running'`).
5. Ensure a stable `'harvest'` session row exists (`SessionStore.ensure('harvest', ...)`) — frames FK to sessions.
6. For each item: write a frame via `FrameStore.createIFrame('harvest', \`${label}\n\n${content}\`, 'normal', 'import', <ts>)`.
- `label` = `[Harvest:<source>] <title>`; content capped at `HARVEST_PREVIEW_CAP_CHARS = 10_000`.
- **Timestamp preservation**: `item.timestamp` is validated by strict `isIsoTimestamp()` (requires `T` separator + timezone). Valid → passed as the frame's `created_at` override. Invalid/missing → falls back to `datetime('now')` and logs a warning (counts `timestampFallbacks`). This keeps date-scoped retrieval working on real exports.
- Heartbeat every 10 frames (`runStore.heartbeat`) + emits `harvest-progress` SSE.
7. Update source tracking: `HarvestSourceStore.upsert()` + `recordSync(source, items.length, saved, incomingHash)`.
8. **Post-harvest cognify** (non-fatal, best-effort): if a *real* embedder is active (`fastify.embeddingProvider.getActiveProvider() !== 'mock'`), run `CognifyPipeline.cognifyBatch()` over the just-saved frames to extract entities + relations into the KnowledgeGraph. If embedder is mock/absent → skipped with `cognifySkippedReason:'no_real_embedder'`.
9. **Post-harvest wiki recompile** (non-fatal): same embedder gate; runs `WikiCompiler.compile({incremental:true})`. Skips with `wikiSkippedReason:'no_real_embedder'` if no real embedder.
10. `runStore.complete()` + delete cache. On error: `runStore.fail()`, cache preserved for resume.
`createIFrame` signature (from `mind/frames.ts`): `createIFrame(gopId, content, importance='normal', source='user_stated', createdAt?)`. The harvest route always uses `gopId='harvest'`, `importance='normal'`, `source='import'`.
---
## 5. Dedup Logic (`dedup.ts`)
Two distinct dedup mechanisms exist:
1. **Set-level skip (route)** — `harvestSetHash(items)`: order-independent SHA-256 digest of all items (each item hashed by `id+title+content`, hashes sorted then re-hashed). Stored as `HarvestSource.lastContentHash`. If a re-sync produces the same digest, the entire commit is skipped (cheap "no changes since last sync"). A same-id content edit changes the digest.
2. **Content-level dedup (pipeline Pass 4)** — `dedup(incoming, existingContents, similarityThreshold=0.75)`:
- `contentHash` = first 16 hex of SHA-256 over normalized (lowercased, whitespace-collapsed) text. Exact hash match → skip.
- Otherwise trigram cosine similarity (`trigramSimilarity`) vs each existing content. `>= 0.75` → duplicate, skip.
- `0.4 <= sim < 0.75` AND `importance==='important'` → flagged as a **contradiction** (returned in `contradictions[]`, not auto-resolved).
- Returns `{ unique, duplicatesSkipped, contradictions }`.
> Note: the commit route relies on (1) the set-hash skip plus `FrameStore.createIFrame`'s own content-dedup (frames dedup on content per the run-store comment), NOT on `dedup()` from Pass 4. `dedup()` only runs inside `HarvestPipeline`.
---
## 6. API Endpoints (`packages/server/src/local/routes/harvest.ts`)
All routes require `fastify.multiMind.personal` (the personal-workspace SQLite handle); they return `503 {error:'Personal mind not available'}` if absent. All operate on the **personal** mind only.
| Method | Path | Purpose |
|---|---|---|
| POST | `/api/harvest/preview` | Parse `{data, source}` with the adapter, return `{itemCount, types, preview[]}` (first 10 items) — no persistence. |
| POST | `/api/harvest/commit` | Run the full ingest flow (parse → frames → cognify → wiki). Body `{data, source}` or `{resumeFromRun}` or `{scanLocal:true}+source`. Returns `{saved, cognified, entitiesExtracted, relationsCreated, wikiCompiled, runId, ...}`. |
| GET | `/api/harvest/sources` | List all registered `HarvestSource` rows. |
| POST | `/api/harvest/sources` | Register/update a source `{source, displayName, sourcePath?, autoSync?, syncIntervalHours?}`. |
| DELETE | `/api/harvest/sources/:source` | Remove a registered source. Returns `{ok:true}`. |
| PATCH | `/api/harvest/sources/:source` | Toggle auto-sync `{autoSync?, syncIntervalHours?}`. |
| GET | `/api/harvest/progress` | **SSE** stream of `{phase, current, total, source}` events during commit (phases: `saving`, `cognifying`, `wiki-compile`). |
| GET | `/api/harvest/runs/latest-interrupted` | Latest `running`/`failed` run with a surviving cache file → `{run}` (or `{run:null}`). UI renders the "Resume?" banner from this. |
| GET | `/api/harvest/runs` | List recent runs (`?limit=`, default 50, max 500). |
| POST | `/api/harvest/runs/:id/abandon` | Discard an interrupted run, delete its cache. |
| POST | `/api/harvest/extract-identity` | LLM-scan last 50 harvest frames (Haiku via internal proxy `/v1/chat/completions`), extract `{name,role,company,industry,bio}` identity suggestions, persist to `profile.identitySuggestions` for user review. Returns `{suggestions}`. |
| POST | `/api/harvest/scan-claude-code` | Scan `~/.claude` via `ClaudeCodeAdapter`, return `{found, path, itemCount, types, preview[]}` (first 20) — no persistence. |
### Frontend wiring notes (from `HarvestTab.tsx`)
- The UI opens an SSE connection to `/api/harvest/progress` and renders live `{phase, current, total}` during a commit.
- Claude Code import is sent as `harvestCommit({ scanLocal: true }, 'claude-code')`.
- On mount the UI polls `/api/harvest/runs/latest-interrupted` to decide whether to show a Resume banner; resume calls commit with `{resumeFromRun: <id>}`.
---
## 7. End-to-End Flow Diagram
```mermaid
flowchart TD
subgraph Frontend["Frontend (HarvestTab.tsx)"]
UI[User picks source + uploads export / file / URL]
SSE[SSE: GET /api/harvest/progress]
RESUME[Poll: GET /api/harvest/runs/latest-interrupted]
end
UI -->|POST /api/harvest/preview| PREVIEW[Adapter.parse -> itemCount + first 10]
PREVIEW -.preview shown.-> UI
UI -->|POST /api/harvest/commit| COMMIT[harvest route]
subgraph Adapters["Source Adapters (parse / scan)"]
A1[ChatGPTAdapter]
A2[ClaudeAdapter]
A3[ClaudeCodeAdapter scan ~/.claude]
A4[GeminiAdapter]
A5[PerplexityAdapter]
A6[Markdown/Plaintext/Pdf/Url]
A7[UniversalAdapter fallback]
end
COMMIT -->|getAdapter source| Adapters
Adapters -->|UniversalImportItem array| NORM[Normalized items]
NORM --> HASH{harvestSetHash == lastContentHash?}
HASH -->|yes| SKIP[Return saved:0 skipped:true]
HASH -->|no| CACHE[Write input to harvest-cache + open HarvestRun running]
CACHE --> SESS[Ensure 'harvest' session row]
SESS --> LOOP[For each item: FrameStore.createIFrame gop=harvest src=import]
LOOP -->|validate item.timestamp ISO| FRAMES[(memory_frames in personal.mind)]
LOOP -->|every 10| SSE
LOOP -->|heartbeat| RUNDB[(harvest_runs)]
FRAMES --> TRACK[HarvestSourceStore.recordSync + lastContentHash]
FRAMES --> COG{real embedder?}
COG -->|yes| COGNIFY[CognifyPipeline -> KnowledgeGraph entities + relations]
COG -->|mock/absent| COGSKIP[skip: no_real_embedder]
COGNIFY --> WIKI{real embedder?}
WIKI -->|yes| COMPILE[WikiCompiler.compile incremental]
WIKI -->|mock/absent| WIKISKIP[skip: no_real_embedder]
COMPILE --> DONE[runStore.complete + delete cache]
COGSKIP --> DONE
WIKISKIP --> DONE
DONE -->|JSON result| UI
RESUME -.->|resumeFromRun id| COMMIT
subgraph Optional["HarvestPipeline (exported, NOT called by commit route)"]
P0[Pass0 injection scan] --> P1[Pass1 Classify Haiku]
P1 --> P2[Pass2 Extract Sonnet]
P2 --> P3[Pass3 Synthesize Sonnet]
P3 --> P4[Pass4 dedup local]
P4 --> DK[DistilledKnowledge array]
end
```
---
## 8. Must-Know Facts for the Frontend Rebuild
- **Two-step UX**: call `POST /api/harvest/preview` first to show item counts + a 10-item preview, then `POST /api/harvest/commit` to actually persist. Both take `{data, source}`.
- **Progress is SSE**, not polling: subscribe to `GET /api/harvest/progress` and render `{phase, current, total, source}` where `phase ∈ {saving, cognifying, wiki-compile}`.
- **Resume banner**: on mount, GET `/api/harvest/runs/latest-interrupted`; if it returns a run, offer Resume (`commit {resumeFromRun:id}`) or Abandon (`POST /api/harvest/runs/:id/abandon`).
- **Claude Code is a local filesystem scan**, not a file upload — send `commit {scanLocal:true, source:'claude-code'}`; server reads `~/.claude` itself. Use `/api/harvest/scan-claude-code` for a no-persist dry run.
- **Source registry** (`/api/harvest/sources`) drives a "connected sources" list with `lastSyncedAt`, `itemsImported`, `framesCreated`, `autoSync` — render these as status chips.
- **Cognify/wiki may be skipped** with reason `no_real_embedder` when no embedding API key is configured — the commit response includes `cognifySkippedReason`/`wikiSkippedReason`; surface this so users understand semantic search/wiki won't update without a real embedder.
- **The commit route writes raw frames directly** (no LLM distillation). The 4-pass `HarvestPipeline` is a separate, unused-by-this-route capability — do not assume committed imports are LLM-summarized.

352
docs/backend-map/sections/05d-subsystem-evolution.md Normal file
View File

@@ -0,0 +1,352 @@
# Subsystem: Self-Evolution Loop
## Purpose
The Evolution subsystem is Waggle's **self-improvement engine**: it mines the agent's own execution history, evolves better persona prompts and behavioral-spec text using a GEPA population-search + EvolveSchema mutation pipeline scored by an LLM-as-judge, passes survivors through safety gates, persists every attempt as an auditable `EvolutionRun`, and lets the user **accept** (deploy as an override file) or **reject** each proposal. It never auto-deploys — every accepted change writes a versioned, rollback-able override and hot-reloads the live spec. The frontend surface is the **Memory app → Evolution tab**.
This section is the contract for the frontend rebuild. The endpoints, payload shapes, status enums, and SSE event names below are quoted directly from the backend code (`packages/server/src/local/routes/evolution.ts`, `packages/agent/src/*`, `packages/hive-mind-core/src/mind/{execution-traces,evolution-runs}.ts`).
---
## 1. The Closed Loop (mental model)
```
every chat turn ──► TraceRecorder ──► execution_traces table
│
(manual "New Run" OR background EvolutionService tick)
│
▼
EvolutionOrchestrator.runOnce()
1. trigger-check (enough eligible traces?)
2. dataset (EvalDatasetBuilder mines traces → EvalExample[])
3. compose (ComposeEvolution = EvolveSchema then IterativeGEPA)
4. gates (runGates: size / growth / structural / regression)
5. persist (EvolutionRunStore.create → status 'proposed')
│
▼
evolution_runs table (status: proposed)
│
user reviews in Evolution tab │
┌───────────────────────────┴───────────────────────────┐
▼ ▼
POST .../accept POST .../reject
│ │
deploy callback writes status → 'rejected'
override JSON file + (terminal)
emits cache-invalidation event
│
┌─────────┴─────────┐
▼ ▼
status 'deployed' status 'failed'
(deploy threw)
```
Key design fact (from `evolution-orchestrator.ts`): **the orchestrator is pure and does NOT touch the filesystem.** Deploy is a pluggable callback supplied by the server route. This means the frontend talks only to HTTP; all file writes happen server-side behind the `accept` endpoint.
---
## 2. HTTP API — every endpoint
Base path is `/api/evolution`. All routes registered in `packages/server/src/local/routes/evolution.ts` via `evolutionRoutes`. The web app calls them through `adapter.fetch(...)` (see `EvolutionTab.tsx`).
| Method | Path | Purpose |
|---|---|---|
| GET | `/api/evolution/runs` | List runs (newest first). Filterable. |
| GET | `/api/evolution/runs/:uuid` | Single run detail, with `winner_schema_json` / `artifacts_json` / `gate_reasons_json` pre-parsed into `winnerSchema` / `artifacts` / `gateReasons`. |
| POST | `/api/evolution/runs/:uuid/accept` | Accept a `proposed` run → deploy override file → mark `deployed` (or `failed`). |
| POST | `/api/evolution/runs/:uuid/reject` | Reject a `proposed` run → mark `rejected`. |
| GET | `/api/evolution/targets` | Enumerate evolvable targets: persona list + behavioral-spec section names + a default schema. Populates the New-Run dropdowns. |
| GET | `/api/evolution/baseline` | Fetch the live baseline text (`?kind=&name=`) for a target plus a default `schemaBaseline`. |
| POST | `/api/evolution/run` | Trigger a real evolution run synchronously (Haiku-backed). Returns JSON, **or streams SSE** when `Accept: text/event-stream`. |
| GET | `/api/evolution/status` | Aggregate status counts for the dashboard + `pendingCount`. |
### 2.1 GET `/api/evolution/runs`
Query params (all optional):
| Param | Type | Notes |
|---|---|---|
| `status` | string \| string[] | One of `proposed\|accepted\|rejected\|deployed\|failed`. Repeat the param for multiple. |
| `targetKind` | string | e.g. `persona-system-prompt`. |
| `targetName` | string | e.g. `coder`. |
| `since` | ISO string | Lower bound on `created_at`. |
| `limit` | string | Parsed to int, clamped `1..500`, default `50`. |
Response: `{ runs: EvolutionRun[], count: number }`.
### 2.2 GET `/api/evolution/runs/:uuid`
404 `{ error: 'Run not found' }` when missing. On success returns the full `EvolutionRun` row **plus** three decoded fields:
```jsonc
{
... all EvolutionRun columns ...,
"winnerSchema": <parsed winner_schema_json | null>,
"artifacts": <parsed artifacts_json | null>,
"gateReasons": <parsed gate_reasons_json | []> // [{ gate, verdict, reason }]
}
```
### 2.3 POST `/api/evolution/runs/:uuid/accept`
Body: `{ note?: string }`.
Guards: 404 if not found; **409** `{ error: 'Run is in status "X" — only proposed runs can be accepted' }` when status ≠ `proposed`.
On success returns the updated `EvolutionRun` (status will be `deployed` on a successful deploy, `failed` if the deploy callback threw). Server-side side effects: writes an override file (see §6) and emits `persona:reloaded` or `behavioral-spec:reloaded` on the server event bus to invalidate the chat route's system-prompt cache.
### 2.4 POST `/api/evolution/runs/:uuid/reject`
Body: `{ reason?: string }`. Same 404 / 409 guards. Returns the updated run (status `rejected`).
### 2.5 GET `/api/evolution/targets`
```jsonc
{
"personas": [{ "id": "coder", "name": "Coder", "description": "...", "icon": "..." }],
"sections": ["coreLoop","qualityRules","behavioralRules","workPatterns","intelligenceDefaults"],
"defaultSchema": { "name": "generic_baseline", "version": 1, "fields": [ ... ] }
}
```
### 2.6 GET `/api/evolution/baseline?kind=&name=`
Both query params **required** (400 if missing). Valid `kind`: `persona-system-prompt` or `behavioral-spec-section`.
- `persona-system-prompt`: returns the persona's live `systemPrompt`. 404 if unknown persona.
- `behavioral-spec-section`: returns the **active** section text (deployed overrides already applied via `server.activeBehavioralSpec`, falling back to compile-time `BEHAVIORAL_SPEC`). 404 if unknown section.
Response: `{ baseline: string, schemaBaseline: Schema }`.
### 2.7 POST `/api/evolution/run` — trigger a real run
Request body:
| Field | Type | Required | Notes |
|---|---|---|---|
| `targetKind` | `EvolutionTarget` | yes | Must be one of `persona-system-prompt`, `behavioral-spec-section`, `tool-description`, `skill-body`, `generic`. |
| `targetName` | string | yes | Non-empty. |
| `baseline` | string | yes | Non-empty current instruction text. |
| `schemaBaseline` | `Schema` | yes | Object with string `name` + array `fields` (+ `version`). |
| `minDelta` | number | no | Improvement threshold to create a proposal. Default `0.02` (2pp). |
| `gepa` | object | no | `{ populationSize?, generations?, miniEvalSize?, anchorEvalSize?, seed?, concurrency? }`. Route defaults `concurrency` to `4`. |
| `schema` | object | no | `{ populationSize?, generations?, evalSize?, anchorEvalSize?, seed? }`. |
| `gateOptions` | `GateOptions` | no | Override gate caps/tolerances. |
Status codes:
| Code | Meaning |
|---|---|
| 200 | Orchestrator ran — inspect `body.outcome` (see §3.4). |
| 400 | Validation error (bad/missing body field). |
| 422 | No Anthropic API key in the vault (`Add one in Settings → Vault`). |
| 503 | `@ax-llm/ax` unavailable / LLM init failed. |
| 500 | Run threw mid-flight (JSON path only). |
JSON response payload (`buildResultPayload`):
```jsonc
{
"outcome": "proposed | skipped-trigger | skipped-gates | skipped-delta | aborted",
"reason": "<string | undefined>",
"run": <EvolutionRun | undefined>, // present for proposed + skipped-gates
"gateResults": <GateResult[] | undefined>,
"composeSummary": { // null when compose didn't run
"combinedDelta": 0.07,
"fullyImproved": true,
"schemaImproved": true,
"schemaDelta": 0.03,
"instructionImproved": true,
"instructionDelta": 0.05,
"winnerId": "g3-m1"
}
}
```
**SSE mode** — when the client sends `Accept: text/event-stream`, the route streams `text/event-stream` (the New-Run modal does this). Events:
| `event:` | `data:` payload |
|---|---|
| `open` | `{ targetKind, targetName }` — stream-is-live signal. |
| `progress` | `EvolutionProgress` `{ phase, message?, detail? }` (see §3.1). |
| `done` | the same `buildResultPayload` object as the JSON path. |
| `error` | `{ error: string }`. |
The orchestrator run **continues to completion even if the client disconnects** (cancelling mid-flight would waste LLM spend already incurred).
### 2.8 GET `/api/evolution/status`
Query: `targetKind?`, `targetName?`, `since?`.
Response: `{ counts: Record<EvolutionRunStatus, number>, pendingCount: number }` where `pendingCount === counts.proposed`.
---
## 3. Core data shapes
### 3.1 `EvolutionProgress` (orchestrator phases, also the SSE `progress` payload)
```ts
phase: 'trigger-check' | 'dataset' | 'compose' | 'gates' | 'persist' | 'skipped' | 'done'
message?: string
detail?: unknown // nested ComposeProgress / GEPAProgress / EvolveSchemaProgress event
```
### 3.2 `EvolutionRun` (the `evolution_runs` SQLite row — source of truth for the UI list/detail)
| Column | Type | Notes |
|---|---|---|
| `id` | number | Autoincrement PK. |
| `run_uuid` | string | UNIQUE; the public identifier used in all route paths. |
| `target_kind` | `EvolutionRunTarget` | `persona-system-prompt` \| `behavioral-spec-section` \| `tool-description` \| `skill-body` \| `generic`. |
| `target_name` | string \| null | Persona id or spec-section name. |
| `baseline_text` | string | Pre-evolution text. |
| `winner_text` | string | Evolved winner text. |
| `winner_schema_json` | string \| null | JSON-encoded `Schema` when a schema stage ran. |
| `delta_accuracy` | number | Winner − baseline judge score (0..1 fraction; UI renders as `pp`). |
| `gate_verdict` | `'pass' \| 'fail'` | Overall gate verdict. |
| `gate_reasons_json` | string | JSON array of `{ gate, verdict, reason }`. |
| `status` | `EvolutionRunStatus` | `proposed` \| `accepted` \| `rejected` \| `deployed` \| `failed`. |
| `artifacts_json` | string \| null | Per-gen history / Pareto size / example count blob. |
| `user_note` | string \| null | Accept note or reject reason. |
| `failure_reason` | string \| null | Set when status is `failed`. |
| `created_at` | string | ISO/SQLite datetime. |
| `decided_at` | string \| null | When accepted/rejected. |
| `deployed_at` | string \| null | When deployed/failed. |
> **Frontend note:** the current `EvolutionTab.tsx` type uses `gate_verdict: 'pass' | 'fail' | 'warn'` and a `resolved_at` field, but the backend only ever emits `'pass' | 'fail'` and uses `decided_at` / `deployed_at` (there is no `resolved_at` column or `warn` verdict). Build against the backend columns above; treat `warn` as cosmetic.
### 3.3 `ExecutionTrace` / `TracePayload` (the raw fuel — `execution_traces` table)
The loop's input. Written by `TraceRecorder` (agent-side facade) on every chat turn / workflow phase. Frontend does not write these, but they explain where eval data comes from.
| Column | Type | Notes |
|---|---|---|
| `id` | number | PK. |
| `session_id` / `persona_id` / `workspace_id` | string \| null | Filtering keys. |
| `model` | string \| null | Model used. |
| `task_shape` | string \| null | Task classification. |
| `outcome` | `TraceOutcome` | `success` \| `corrected` \| `abandoned` \| `verified` \| `pending`. |
| `trace_json` | string | Serialized `TracePayload`. |
| `cost_usd` / `duration_ms` | number | Metrics. |
| `created_at` / `finalized_at` | string \| null | Timestamps. |
`TracePayload` = `{ input, output, reasoning[], toolCalls[], artifacts[], tokens, harness?, correctionFeedback?, tags? }`. Tool-call args are secret-scrubbed before persistence (`SECRET_ARG_KEYS` in `trace-recorder.ts`).
### 3.4 `OrchestratorOutcome` (drives `outcome` in run-trigger responses)
| Value | Meaning |
|---|---|
| `proposed` | Run created, gates passed, awaiting accept/reject. `run` populated. |
| `skipped-trigger` | Auto-trigger threshold not met / no eligible traces to form a dataset. |
| `skipped-gates` | Compose ran but gates failed; run is created then **immediately auto-rejected** (kept for audit). `run` populated. |
| `skipped-delta` | Winner improved less than `minDelta`; no proposal created. |
| `aborted` | Abort signal fired. |
### 3.5 `Schema` (DSPy-style typed signature evolved by EvolveSchema)
```ts
Schema = { name: string; version: number; fields: SchemaField[] }
SchemaField = { name; type: FieldType; description; required: boolean; constraints: FieldConstraint[] }
FieldType = 'string'|'number'|'boolean'|'array'|'object'|'enum'
FieldConstraint = { kind: 'minLength'|'maxLength'|'pattern'|'enum'|'range'|'custom'; value: string|number|string[] }
```
Default schema (from both `evolution.ts` route and `evolution-service.ts`): two fields `reasoning` (string, optional) then `answer` (string, required).
---
## 4. The evolution pipeline internals (for accurate UI labels/tooltips)
### 4.1 EvalDatasetBuilder (`eval-dataset.ts`)
Mines `execution_traces` into `EvalExample { input, expected_output, metadata }`. Positive examples come from `success`/`verified` outcomes; `corrected` traces become negatives whose `correctionFeedback` is the ground truth. Filter pipeline order: **secret scan → length/low-signal heuristic → optional judge → input-hash dedup → deterministic 60/20/20 train/val/holdout split.** Deterministic given a seed.
### 4.2 EvolveSchema — Stage 1 (`evolve-schema.ts`)
Evolves output **structure**. Three per-generation phases: **A Structure Discovery** (add/remove/replace fields — biggest-impact mutation class), **B Field-Order Probes** (e.g. move `reasoning` before `answer`), **C Failure-Driven Refinement** (edit descriptions / tighten constraints from worst-example feedback). 8 typed mutations: `add_output_field`, `remove_field`, `edit_field_desc`, `change_field_type`, `add_constraint`, `remove_constraint`, `reorder_fields`, `replace_output_fields`. 2-D Pareto selection on **(accuracy ↑, complexity ↓)**. Defaults: population 5, generations 3, evalSize 32, anchorEvalSize 100, seed 1.
### 4.3 IterativeGEPA — Stage 2 (`iterative-optimizer.ts`)
Freezes the winning schema; evolves the **instruction prompt** via multi-generation population search with reflective mutations driven by judge feedback ("ASI"). Per generation: micro-screen (default 50) → mini-eval (default 64) → mutate top Pareto candidate. Anchor eval default 400. 3-D Pareto on **(correctness, procedureFollowing, conciseness)**. 7 strategies cycle: `expand-edge-cases`, `tighten-format`, `add-examples`, `clarify-constraints`, `reduce-length`, `restructure-steps`, `targeted-feedback`. **Safety guard:** GEPA throws unless passed a `makeRunningJudge`-wrapped judge (or `allowBareJudge: true` for tests) — a bare judge would optimize prompt-text-vs-expected similarity (a meaningless gradient).
### 4.4 ComposeEvolution (`compose-evolution.ts`)
Runs Stage 1 then Stage 2. **Feedback separation** is the critical design detail: Stage 2's judge is wrapped by `filterJudgeFeedback` so **structural** complaints (e.g. "missing reasoning field") are stripped, leaving only value-level signals — otherwise GEPA would mutate the instruction to undo the schema Stage 1 just evolved. Numeric scores are preserved; only the textual `feedback` is filtered (`defaultFeedbackFilter`).
### 4.5 LLMJudge (`judge.ts`)
Rubric scorer. Weights: **correctness 0.5, procedureFollowing 0.3, conciseness 0.2** (must sum to 1.0), each 0–10 then normalized, multiplied by a length penalty (default target 2000 chars, tolerance 0.5, floor 0.5). Produces `JudgeScore { overall, weighted, correctness, procedureFollowing, conciseness, lengthPenalty, feedback, parsed }`. The `feedback` string is the input that drives GEPA's reflective mutations.
### 4.6 Evolution Gates (`evolution-gates.ts`)
A candidate must pass **all** gates; first failure short-circuits to `fail`. Gate categories and the structured `GateResult { gate, verdict: 'pass'|'fail', reason, detail? }` the UI renders:
| Gate `gate` | Checks |
|---|---|
| `non-empty` | Candidate not blank. |
| `size` | Hard char cap by target: persona ≤ 3000, tool-description ≤ 500, skill-body ≤ 15000, behavioral-spec-section ≤ 4000, generic ≤ 8000. |
| `growth` | ≤ +20% over baseline length (default `maxGrowthRatio` 0.2). |
| `balanced-fences` | Even number of ``` ``` ``` markdown fences. |
| `no-placeholders` | Rejects `[PLACEHOLDER]`, `<placeholder>`, `{{var}}`. |
| `no-todos` | Rejects leftover `TODO:` / `FIXME:` / `XXX:` line labels. |
| `regression` | Candidate score must not drop below `maxRegression` (default −0.02). Only runs when scores provided. |
When the overall verdict is `fail`, the orchestrator **creates the run then immediately rejects it** (outcome `skipped-gates`) so dangerous candidates never appear in the review queue but stay in history.
### 4.7 Evolution LLM Wiring (`evolution-llm-wiring.ts`)
Binds real LLMs (default **Claude 4.5 Haiku** via `@ax-llm/ax`, dynamic-imported) into the model-agnostic primitives: `buildJudgeLLMCall`, `buildGEPAMutateFn`, `buildSchemaExecuteFn`, `makeRunningJudge`. Includes exponential-backoff retry (5s → 15s → 45s → 135s → 150s, 6 attempts) over retryable HTTP statuses/codes. The `RUNNING_JUDGE_BRAND` symbol marks judges that execute the candidate prompt against a real LLM before scoring.
---
## 5. Triggering: manual vs autonomous
| Mode | Trigger | Where |
|---|---|---|
| **Manual** | User clicks "New Run" → `POST /api/evolution/run` (SSE). | `EvolutionTab.tsx` New-Run modal. |
| **Autonomous** | Background daemon `EvolutionService` ticks (default every 6h), picks one eligible target whose new-trace count ≥ `minTracesPerTarget` (default 20), runs `runOnce`, produces a `proposed` run. | `evolution-service.ts`, wired in `index.ts`. |
The autonomous daemon is **disabled by default** — opt-in via env `WAGGLE_EVOLUTION_AUTO_ENABLED=1` (also `WAGGLE_EVOLUTION_TICK_INTERVAL_MS`, `WAGGLE_EVOLUTION_MIN_TRACES`). It **never auto-deploys**; accept/reject is always manual. Default targets = every registered persona + every behavioral-spec section.
---
## 6. Deploy mechanics (what "Accept & Deploy" does server-side)
From `evolution-deploy.ts` + the `deployFromRun` dispatcher in the route:
- **`persona-system-prompt`** → `deployPersonaOverride(dataDir, …)` writes `{dataDir}/personas/{id}.json` (shadows the built-in persona via `loadCustomPersonas`). Emits `persona:reloaded`.
- **`behavioral-spec-section`** → `deployBehavioralSpecOverride(dataDir, …)` writes `{dataDir}/behavioral-overrides/{section}.json` (merged into `BEHAVIORAL_SPEC` at load). Emits `behavioral-spec:reloaded`.
- **`tool-description` / `skill-body` / `generic`** → **not yet implemented**; the deploy throws, so accepting such a run lands it in status `failed`.
Every writer is **atomic** (write `.tmp` then rename) and **backs up** the previous version to `{file}.bak`, enabling `rollbackPersonaOverride` / `rollbackBehavioralSpecOverride`. The five behavioral-spec sections are exactly: `coreLoop`, `qualityRules`, `behavioralRules`, `workPatterns`, `intelligenceDefaults`.
---
## 7. Loop diagram (full)
```mermaid
flowchart TD
subgraph capture["Trace capture (every turn)"]
AL[Agent loop / chat route] -->|onToolUse / onToolResult| TR[TraceRecorder]
TR --> ETS[(execution_traces)]
end
subgraph trigger["Trigger"]
UI[Evolution tab: New Run] -->|POST /api/evolution/run| ORC
SVC[EvolutionService daemon\nWAGGLE_EVOLUTION_AUTO_ENABLED] -->|tick → runOnce| ORC
end
ORC[EvolutionOrchestrator.runOnce]
ETS --> EDB[EvalDatasetBuilder\nmine traces → EvalExample]
EDB --> CMP
subgraph compose["ComposeEvolution"]
CMP[ComposeEvolution] --> ES[Stage 1: EvolveSchema\n2D Pareto accuracy/complexity]
ES -->|frozen winner schema| GEPA[Stage 2: IterativeGEPA\n3D Pareto + reflective mutate]
JUDGE[LLMJudge\ncorrectness/procedure/conciseness] -.scores.-> ES
JUDGE -.filtered feedback.-> GEPA
LLM[Haiku via @ax-llm/ax\nmakeRunningJudge / mutate / execute] -.-> JUDGE
end
ORC --> CMP
GEPA -->|winner_text + delta| GATES[runGates\nsize/growth/structural/regression]
GATES -->|pass| STORE[(evolution_runs)\nstatus = proposed]
GATES -->|fail| STOREF[(evolution_runs)\ncreate then auto-reject]
STORE -->|GET /runs, /status| REVIEW[Evolution tab review]
REVIEW -->|POST /accept| DEPLOY[deployFromRun\npersona/spec override file]
REVIEW -->|POST /reject| REJECTED[status = rejected]
DEPLOY -->|success| DEPLOYED[status = deployed\n+ emit *:reloaded event]
DEPLOY -->|throws| FAILED[status = failed]
DEPLOYED -.hot-reload.-> AL
```

413
docs/backend-map/sections/05e-subsystem-waggledance-aios.md Normal file
View File

@@ -0,0 +1,413 @@
# 05e — Subsystem: WaggleDance + the AI-OS Arc
**Purpose.** This subsystem is Waggle's multi-agent coordination layer. It has two halves that meet at one shared protocol (`WaggleMessage`): (1) **WaggleDance** — a message protocol + dispatcher + in-memory ring buffer (`SignalBus`) that carries "what's happening across agents/tools right now"; and (2) the **AI-OS arc** — the path that detects 7 external AI tools on the user's machine, installs reversible hooks into them, launches them with a workspace-context env var, then lets those tools' hooks emit signals back into Waggle so the UI shows cross-tool activity in real time. There is also a local sub-agent layer (spawn/list/result, orchestrated workflows, cross-workspace messaging) that shares the same vocabulary.
This file is the contract for rebuilding the frontend surfaces that consume these endpoints (the **WaggleDanceApp** activity feed, the **LauncherApp** dock, the **Mission Control** inventory tile, and the **Room** sub-agent tiles). Every identifier and path below is quoted from the source under `packages/waggle-dance/`, `packages/agent/src/`, `packages/hive-mind-shim-core/src/`, `packages/shared/src/`, and `packages/server/src/local/`.
---
## 1. Mental model — two signal planes
There are **two distinct signal shapes** in this subsystem, and a **bridge** that converts one into the other. Do not confuse them.
| Plane | Shape | Subtypes | Where it lives | Who consumes it |
|---|---|---|---|---|
| **v2 protocol bus** (WaggleDance) | `WaggleMessage` | 10 protocol subtypes (`discovery`, `task_delegation`, …) | `SignalBus` ring buffer (`packages/server/src/local/signal-bus.ts`) | `GET /api/waggle-dance/signals`; the bridge |
| **Legacy UX stream** | `WaggleSignal` | 5 UX categories (`discovery`/`handoff`/`insight`/`alert`/`coordination`) | in-memory array in `waggle-signals.ts` | `GET /api/waggle/signals` + `GET /api/waggle/stream` (SSE); the **WaggleDanceApp UI** |
The **bridge** (`installWaggleDanceBridge`, `waggle-dance-bridge.ts`) subscribes to the v2 bus and re-emits every v2 message as a legacy UX signal, so the existing UI surfaces cross-tool activity with **zero frontend changes**. A frontend rebuild can target *either* plane; the legacy stream is the one the current UI already speaks and is SSE-streamed.
```mermaid
flowchart LR
subgraph v2["v2 protocol plane (WaggleMessage)"]
POST["POST /api/waggle-dance/signal"]
DISP["WaggleDanceDispatcher.dispatch()"]
BUS["SignalBus ring buffer (cap 500)"]
GET2["GET /api/waggle-dance/signals"]
end
subgraph legacy["legacy UX plane (WaggleSignal)"]
EMIT["emitWaggleSignal()"]
ARR["in-memory array (cap 500)"]
GET1["GET /api/waggle/signals"]
SSE["GET /api/waggle/stream (SSE)"]
end
POST --> DISP --> BUS --> GET2
BUS -->|installWaggleDanceBridge subscribes| EMIT
EMIT --> ARR --> GET1
ARR --> SSE
SSE -->|consumed by| UI["WaggleDanceApp UI (useWaggleDance)"]
```
---
## 2. The WaggleDance protocol (`packages/waggle-dance` + `packages/shared/src/types.ts`)
### 2.1 `WaggleMessage` — canonical protocol envelope
Defined in `packages/shared/src/types.ts`. This is the v2 signal shape.
| Field | Type | Nullable | Meaning |
|---|---|---|---|
| `id` | `string` | no | UUID; server-assigned (`randomUUID()`) on POST |
| `teamId` | `string` | no | Team id; defaults to `personal::<senderId>` when omitted (the personal-tier moat) |
| `senderId` | `string` | no | Logical sender (`'claude-code-hook'`, `'cursor-hook'`, `'local'`, `agent-loop:<persona>`) |
| `type` | `MessageType` | no | One of `'broadcast' \| 'request' \| 'response'` |
| `subtype` | `MessageSubtype` | no | One of 10 subtypes (see 2.2) |
| `content` | `Record<string, unknown>` | no | Free-form structured payload; preserved verbatim |
| `referenceId` | `string \| null` | yes | For `response` types: correlates back to a prior request |
| `routing` | `Array<{ userId: string; reason: string }> \| null` | yes | For `routed_share`: directed recipients + rationale |
| `createdAt` | `Date` | no | Server-assigned timestamp |
### 2.2 `MessageType` × `MessageSubtype` and valid combinations
`MessageType = 'broadcast' | 'request' | 'response'`
`MessageSubtype` (10 values):
`knowledge_check`, `task_delegation`, `skill_request`, `model_recommendation`, `knowledge_match`, `task_claim`, `discovery`, `routed_share`, `skill_share`, `model_recipe`.
Valid combos are enforced by `validateMessageTypeCombo(type, subtype)` in `packages/waggle-dance/src/protocol.ts` against `VALID_COMBINATIONS`:
| `type` | Allowed `subtype`s |
|---|---|
| `request` | `knowledge_check`, `task_delegation`, `skill_request`, `model_recommendation` |
| `response` | `knowledge_match`, `task_claim` |
| `broadcast` | `discovery`, `routed_share`, `skill_share`, `model_recipe` |
`isRoutedMessage(subtype)` returns true only for `routed_share`. An invalid combo at the route returns HTTP 400.
### 2.3 The dispatcher — `WaggleDanceDispatcher` (`dispatcher.ts`)
`dispatch(message: WaggleMessage): Promise<DispatchResult>` validates the combo, then branches on `subtype`. `DispatchResult = { handled: boolean; response?: string; error?: string }`.
Dependencies injected via `DispatchDeps` — **v1** (team-internal) deps are required; **v2** (cross-tool bus) deps are all optional, so the route layer can wire them incrementally and unwired branches still return `handled: true`:
| Dep | Signature | Used by subtypes |
|---|---|---|
| `searchMemory` | `(query) => Promise<string>` | `knowledge_check` |
| `resolveCapability` | `(query) => Array<{source,name,description,available}>` | `skill_request` |
| `spawnWorker` | `(task, role, context?) => Promise<string>` | `task_delegation` |
| `emitSignal?` | `(msg) => Promise<void>` | `discovery`, `routed_share`, `model_recipe` (broadcasts) |
| `recordResponse?` | `(msg) => Promise<void>` | `knowledge_match`, `task_claim` (responses) |
| `recommendModel?` | `(query, context) => Promise<string \| null>` | `model_recommendation` |
**Per-subtype dispatch behavior (the 6 v2 branches the prompt asks about plus the 4 v1):**
| Subtype | Handler | Behavior | Required `content` fields |
|---|---|---|---|
| `task_delegation` (v1) | `handleTaskDelegation` | Calls `spawnWorker(task, role, context)` | `content.task` (required); `content.role` (default `'analyst'`), `content.context` |
| `knowledge_check` (v1) | `handleKnowledgeCheck` | Calls `searchMemory(query)` | `content.query` or `content.topic` |
| `skill_request` (v1) | `handleSkillRequest` | Calls `resolveCapability(query)`, returns a formatted route list | `content.skill` or `content.query` |
| `skill_share` (v1) | `handleSkillShare` | Returns JSON `{action:'install_shared_skill', skillName, skillContent, sharedBy}` | `content.name`/`content.skill` + `content.content` |
| `discovery` (v2) | `handleBroadcastSignal` | If `emitSignal` set → emit to bus; else accept | none enforced |
| `routed_share` (v2) | `handleBroadcastSignal` | Same; UI shows as a directed handoff | `routing` carries recipients |
| `model_recipe` (v2) | `handleBroadcastSignal` | Same; shares a model+prompt recipe | none enforced |
| `knowledge_match` (v2) | `handleResponseRelay` | If `recordResponse` set → record; correlated via `referenceId` | none enforced |
| `task_claim` (v2) | `handleResponseRelay` | Same; a worker claiming a delegated task | none enforced |
| `model_recommendation` (v2) | `handleModelRecommendation` | Calls `recommendModel(query, restOfContent)`; friendly fallback if unwired | `content.query` (required) |
`hive-query.ts` defines `HiveQuery { topic; scope? }` and `HiveQueryResult { entities[]; relatedTasks[]; relatedMessages[] }` — DB-backed hive queries are executed by the server's `MessageService`, not in this package (the package is types + dispatcher only). `packages/waggle-dance/src/index.ts` barrel-exports `protocol.js`, `hive-query.js`, `dispatcher.js`.
---
## 3. `SignalBus` — the v2 ring buffer (`packages/server/src/local/signal-bus.ts`)
In-memory ring buffer of `WaggleMessage`. Created once per server, decorated as `server.signalBus`. **Not durable** — survives until sidecar restart or rollover.
- `DEFAULT_BUFFER_SIZE = 500`. On overflow the **oldest** is dropped (`shift()`).
- `record(signal)` → appends, drops oldest if over capacity, notifies all subscribers synchronously (subscriber errors are swallowed so one bad subscriber can't poison the bus), returns the signal.
- `query(filter)` → snapshot, **newest first**. `SignalFilter`:
| Filter field | Type | Effect |
|---|---|---|
| `subtype` | `MessageSubtype` | exact match |
| `tool` | `string` | matches `content.tool` |
| `teamId` | `string` | exact match |
| `since` | ISO string | `createdAt > since` |
| `limit` | number | cap result count |
- `subscribe(sub)` → returns an unsubscribe fn (this is how the bridge attaches).
- `size` getter; `clear()` (tests).
---
## 4. The two streaming UX endpoints (legacy plane) — `waggle-signals.ts`
`WaggleSignal` (the legacy/UX shape the current UI renders):
| Field | Type | Nullable | Meaning |
|---|---|---|---|
| `id` | `string` | no | `sig-<ts>-<rand>` |
| `type` | `string` | no | e.g. `agent:started`, `tool:called`, `memory:saved`, `agent:completed`, or bridged `waggle-dance:<category>` |
| `workspaceId` | `string` | no | Defaults to `'global'` |
| `content` | `string` | no | Human-readable primary text |
| `metadata` | `Record<string,unknown>` | yes | Provenance (bridge fills `subtype`/`senderId`/`tool`/`teamId`/`referenceId`/`routing`/`priority`/`protocolMessage`) |
| `timestamp` | `string` (ISO) | no | Emit time |
| `acknowledged` | `boolean` | no | Ack state |
Store is a capped in-memory array (`MAX_SIGNALS = 500`, newest-first via `unshift`). `emitWaggleSignal(...)` is the exported publish fn used by the chat loop **and** the bridge.
---
## 5. The bridge — v2 → legacy (`waggle-dance-bridge.ts`)
`installWaggleDanceBridge(bus: SignalBus): () => void` subscribes to the v2 bus and re-emits each `WaggleMessage` via `emitWaggleSignal`. It is auto-installed the first time `server.signalBus` is created (inside `waggle-dance.ts`).
**Subtype → UX category mapping** (`categorizeSubtype`, exported for tests):
| v2 `subtype` | UX category | Legacy `type` emitted |
|---|---|---|
| `discovery`, `knowledge_check`, `skill_request` | `discovery` | `waggle-dance:discovery` |
| `task_delegation`, `skill_share`, `routed_share` | `handoff` | `waggle-dance:handoff` |
| `knowledge_match` | `insight` | `waggle-dance:insight` |
| `task_claim`, `model_recipe`, `model_recommendation` | `coordination` | `waggle-dance:coordination` |
| *(any subtype)* with `priority === 'critical'` | `alert` (override) | `waggle-dance:alert` |
Priority is resolved by `inferPriority`: `content.priority` (`low/normal/high/critical`) wins; else `content.importance` (`high`→high, `critical`→critical); else `normal`. `buildLegacyContent` builds the display string as `"<subtype>: <topic>"` where topic falls back through `content.topic → title → query → task → skill`, then to a key summary. Full provenance (including the verbatim `protocolMessage`) is carried in `metadata`.
---
## 6. AI-OS Phase 0 — tool detection (`packages/shared/src/tool-detection.ts` + `packages/agent/src/tool-detection.ts`)
### 6.1 The 7 supported tools
`SUPPORTED_TOOLS` (`@waggle/shared`), with display names from `TOOL_DISPLAY_NAMES`:
| `ToolId` | Display name | Detection strategy | Linux/macOS/Win paths |
|---|---|---|---|
| `claude-code` | Claude Code | PATH lookup (`claude`) | `which`/`where.exe` |
| `claude-desktop` | Claude Desktop | candidate paths | `/Applications/Claude.app/...`, `%LOCALAPPDATA%\AnthropicClaude\Claude.exe` |
| `cursor` | Cursor | candidate paths | `/Applications/Cursor.app/...`, `…\Programs\cursor\Cursor.exe` |
| `codex` | Codex CLI | PATH lookup (`codex`) | `which`/`where.exe` |
| `codex-desktop` | Codex Desktop | candidate paths (unreleased; speculative vendor paths) | `/Applications/Codex.app/...` |
| `hermes` | Hermes Agent | PATH lookup (`hermes`) | `which`/`where.exe` |
| `openclaw` | OpenClaw | PATH lookup (`openclaw`) | `which`/`where.exe` |
`LAUNCH_COHORT` = all 7. (The launcher's runtime guard message still says "claude-code, cursor, claude-desktop" — a stale Phase-2 string — but the actual array includes all 7.)
### 6.2 `DetectedTool` (per-tool result)
| Field | Type | Meaning |
|---|---|---|
| `id` | `ToolId` | tool id |
| `displayName` | `string` | from `TOOL_DISPLAY_NAMES` |
| `installed` | `boolean` | true iff binary found at a known path |
| `installedPath` | `string \| null` | absolute path to binary |
| `version` | `string \| null` | best-effort (`--version`); null if exec failed |
| `hooksInstalled` | `boolean` | true iff hook pointer file exists **and** its referenced `backup` file still exists (partial rollback → false) |
| `hookPointerPath` | `string \| null` | the pointer file probed (`~/<config-dir>/hive-mind-install.json`) |
| `diagnostic?` | `string` | optional human reason for a partial failure (e.g. `'--version exec failed'`) |
### 6.3 `ToolDetectionResult` (envelope) — returned by `GET /api/tools/detect`
| Field | Type | Meaning |
|---|---|---|
| `platform` | `NodeJS.Platform \| 'other'` | platform detection ran on |
| `detectedAt` | `string` (ISO) | completion timestamp |
| `tools` | `DetectedTool[]` | per-tool results, **in `SUPPORTED_TOOLS` order** (stable for the UI) |
Implementation notes: `detectInstalledTools(opts?)` runs every detector in parallel (`Promise.all`) but preserves `SUPPORTED_TOOLS` order in the output. All fs/exec/PATH calls are injectable (`ToolDetectionDeps`) for hermetic tests. Hook-pointer config dirs are in `HOOK_POINTER_BY_TOOL` (e.g. `claude-code` → `.claude/hive-mind-install.json`, `cursor` → `.cursor/...`, `codex`/`codex-desktop` → `.codex/...`).
---
## 7. AI-OS Phase 2 — launcher + hook installer (`packages/agent/src/tool-launcher.ts` + `tool-process-tracker.ts`)
### 7.1 `launchTool(opts) → LaunchResult`
Spawns the tool **detached** (`detached: true`, `stdio: 'ignore'`, `unref()` so it outlives the sidecar). Injects `WAGGLE_WORKSPACE_ID` into the child env when `workspaceId` is given — this is the single env var the tool's hooks pick up (via shim-core `workspace-resolver`) to tag captured memory.
`LaunchOptions`: `{ id: ToolId; installedPath: string (required); workspaceId?; cwd? (default = dirname of binary); args?; deps? }`.
`LaunchResult`: `{ ok: boolean; pid: number | null; executed: { binary; args; cwd? }; error? }`.
Guards: `id` must be in `LAUNCH_COHORT` (all 7); `installedPath` required (caller passes it from a fresh detect).
### 7.2 `runHookCommand(opts) → HookCommandResult` (reversible hooks)
Runs `npx --yes @waggle/hive-mind-hooks-<id> <install|verify|uninstall>` (adds `--cli-path <path>` only on `install`). Captures stdout/stderr/exit-code.
- `HookAction = 'install' | 'verify' | 'uninstall'`.
- **`HOOKS_COHORT`** = `['claude-code', 'codex', 'codex-desktop', 'cursor', 'hermes', 'openclaw']` (6 tools — every tool with a real `bin`). `claude-desktop` is excluded (binless stub). Hook actions gate on `HOOKS_COHORT`, **not** `LAUNCH_COHORT`, so the UI never offers a hook action npx can't fulfil.
- `hookPackageFor(id)` → `@waggle/hive-mind-hooks-${id}`.
- `HookCommandResult`: `{ ok; action; packageName; stdout; stderr; code; error? }`.
Reversibility: install writes a pointer file `~/<config>/hive-mind-install.json` whose `backup` field points at the original config; uninstall restores it; detection's `hooksInstalled` flips false if the backup is gone (partial rollback is honestly reported).
### 7.3 `ToolProcessTracker` — the 'Running' badge backend
In-memory map of `TrackedProcess { pid; toolId; startedAt; workspaceId? }`. **Not durable** (dropped on restart — losing attribution is preferred over reporting stale liveness).
| Method | Behavior |
|---|---|
| `register(pid, toolId, workspaceId?)` | record a freshly spawned pid (idempotent on pid) |
| `list()` | GC dead pids (`process.kill(pid,0)` liveness), return alive ones |
| `forget(pid)` | drop a pid |
| `kill(pid, gracefulTimeoutMs=3000)` | refuses untracked pids; SIGTERM → wait → SIGKILL escalation; returns `{ ok; pid; reason }` where reason ∈ `not-tracked` / `already-dead` / `sigterm-ok` / `sigkill-ok` / `sigterm-failed-sigkill-failed` |
---
## 8. AI-OS Phase 1D — the shim-core emitter (`packages/hive-mind-shim-core/src/signal-emitter.ts`)
This is the library a tool's hook calls to push a signal back into Waggle. **Fail-open by design**: any error (ENOTFOUND, ECONNREFUSED, non-2xx, parse) returns `null` and logs one stderr warning — the host AI tool's hook chain never sees a failure. Uses only Node 20 built-in `fetch`, 2-second default timeout.
URL resolution: `options.url > env.WAGGLE_SIDECAR_URL > http://127.0.0.1:3333`.
- `emitSignalToWaggleDance(opts: EmitSignalOptions): Promise<EmittedSignal | null>` — POSTs to `<url>/api/waggle-dance/signal`.
- `EmitSignalOptions`: `{ type: SignalType; subtype: SignalSubtype; content: Record<string,unknown>; senderId? (default 'hook'); teamId?; referenceId?; routing?; url?; timeoutMs? (2000); fetchImpl?; onWarn? }`.
- `SignalType` / `SignalSubtype` mirror the protocol enums exactly.
- `EmittedSignal` is the server-persisted message echoed back: `{ id; teamId; senderId; type; subtype; content; referenceId; routing; createdAt }`.
- `maybeEmitDiscovery(eventType, importance, payload, opts)` — convenience policy: emit a `broadcast`/`discovery` only when `importance` is `'high'|'critical'` **and** `eventType` is `'stop'|'pre-compact'`. Phase 1E wires the claude-code Stop hook to this (opt-in via `WAGGLE_SIGNAL_EMIT`).
Hook event vocabulary (`hook-event-types.ts`): `EventType` = `session-start | session-end | user-prompt-submit | pre-compact | stop | pre-tool-use | post-tool-use`; `ShimSource` = `claude-code | cursor | hermes | codex | opencode | openclaw`.
---
## 9. The end-to-end AI-OS flow: detect → launch → signal → UI
```mermaid
sequenceDiagram
participant UI as Web UI (LauncherApp / Mission Control)
participant SC as Sidecar (Fastify @ :3333)
participant DET as detectInstalledTools()
participant LT as launchTool()
participant EXT as External AI tool (Claude Code, …)
participant HOOK as Tool hook (+ shim-core)
participant BUS as SignalBus (v2)
participant BR as installWaggleDanceBridge
participant STREAM as /api/waggle/stream (SSE)
UI->>SC: GET /api/tools/detect
SC->>DET: run 7 detectors in parallel
DET-->>SC: ToolDetectionResult { tools[] }
SC-->>UI: 200 ToolDetectionResult
UI->>SC: POST /api/tools/hooks { id, action:'install' }
SC->>EXT: npx @waggle/hive-mind-hooks-<id> install (reversible)
SC-->>UI: 200 HookCommandResult
UI->>SC: POST /api/tools/launch { id, installedPath, workspaceId }
SC->>LT: spawn detached, env WAGGLE_WORKSPACE_ID
LT-->>SC: { ok, pid }
SC->>SC: toolProcessTracker.register(pid,...)
SC-->>UI: 202 LaunchResult { pid }
EXT->>HOOK: lifecycle event (stop / pre-compact)
HOOK->>SC: POST /api/waggle-dance/signal (emitSignalToWaggleDance, fail-open)
SC->>SC: WaggleDanceDispatcher.dispatch()
SC->>BUS: emitSignal → bus.record(WaggleMessage)
BUS->>BR: subscriber fires
BR->>SC: emitWaggleSignal(mapped WaggleSignal)
SC-->>STREAM: event: signal (SSE)
STREAM-->>UI: live cross-tool activity
```
Rollback anchor for the whole arc: git tag `checkpoint/pre-ai-os-2026-05-20`.
---
## 10. Local sub-agent + cross-workspace layer (same vocabulary, different transport)
These are **agent tools** (callable by the LLM), not HTTP routes, but the frontend sees their effects through the notifications/Room stream and they share the coordination theme.
### 10.1 Sub-agent spawning — `createSubAgentTools(deps)` (`subagent-tools.ts`)
Exposes **3 tools** to the main agent. Sub-agents run in-process via the shared `runLoop`. In-memory registries: `activeAgents` + `agentResults` (capped at `MAX_AGENT_RESULTS = 100`, stale eviction at `STALE_THRESHOLD_MS = 30 min`).
| Tool | Inputs | Output |
|---|---|---|
| `spawn_agent` | `name`, `role`, `task` (required); `context?`, `tools?` (role=`custom` only), `model?`, `max_turns?` (50) | Markdown sub-agent result (role, duration, tools used, tokens, content) |
| `list_agents` | none | Active + completed agents with previews |
| `get_agent_result` | `agent_id` (or name) | Full stored result |
`ROLE_TOOL_PRESETS` maps role → tool allowlist: `researcher`, `writer`, `coder`, `analyst`, `reviewer`, `planner` (default fallback = `analyst`). Lifecycle is surfaced to the UI via `onSubAgentStatus` (`running`/`done`/`error`) which the server wires to `emitSubagentStatus` → the Room canvas. `onSubAgentComplete` persists results to the active mind (best-effort, never throws).
`SubAgentResult`: `{ agentId; agentName; role; response; usage{inputTokens,outputTokens}; toolsUsed[]; duration; completedAt }`. `SubAgentStatusEvent`: `{ agentId; name; role; status; task; toolsUsed[]; startedAt; completedAt? }`.
### 10.2 Workflow orchestration — `SubagentOrchestrator` (`subagent-orchestrator.ts`)
Supervisor/worker pattern over `spawn_agent`. `runWorkflow(template)` executes steps in **dependency order** (topological, sequential), injects upstream results as context, then aggregates. Emits `worker:status` events (an `EventEmitter`).
- `WorkflowStep`: `{ name; role; task; tools?; dependsOn?; contextFrom?; maxTurns? }`.
- `WorkflowTemplate`: `{ name; description; steps[]; aggregation: 'concatenate' | 'last' | 'synthesize' }`.
- `WorkerState`: `{ id; name; role; status: 'pending'|'running'|'done'|'failed'; task; startedAt?; completedAt?; result?; error?; toolsUsed[]; usage }`.
- Adds `synthesizer` + `summarizer` to the role presets. Circular dependency → remaining steps marked `failed` with a clear error.
### 10.3 Cross-workspace messaging — `AgentMessageBus` + `createAgentCommsTools`
`AgentMessageBus` (`agent-message-bus.ts`) is an **in-memory, per-machine** bus for concurrent agent sessions in different workspaces (distinct from team/PostgreSQL messaging). `AgentMessage`: `{ id; from (workspaceId); to (workspaceId); content; correlationId?; timestamp; ttlMs }`. Default TTL = 5 min; `receive()` is one-shot (drains + filters expired); `peek()` is non-destructive; `cleanup()` GCs expired.
`createAgentCommsTools(bus, currentWorkspaceId, isSessionActive?)` exposes **2 tools**:
| Tool | Inputs | Notes |
|---|---|---|
| `send_agent_message` | `workspace`, `message`, `correlationId?` | Rejects self-send; rejects target whose session isn't active |
| `check_agent_messages` | none | Consumes pending messages for the current workspace |
---
## 11. API reference — every route
All routes are served by the **local sidecar** (Fastify, loopback `:3333`). Registration in `packages/server/src/local/index.ts`: `toolsRoutes` (1990), `waggleDanceRoutes` (1991), `waggleSignalRoutes` (2016). The `signalBus` and `toolProcessTracker` decorations propagate to the parent instance via `fastify-plugin`.
| Method | Full path | Request shape | Response shape | Streaming? |
|---|---|---|---|---|
| POST | `/api/waggle-dance/signal` | `{ type, subtype, content, senderId?, teamId?, referenceId?, routing? }` (Zod `signalRequestSchema`) | `201 { dispatched:true, response, message: WaggleMessage }` · `400 { error, details? }` | No |
| GET | `/api/waggle-dance/signals` | query `{ subtype?, tool?, teamId?, limit?(≤1000), since? }` | `200 { signals: WaggleMessage[], total }` (newest-first) · `400` | No |
| GET | `/api/waggle/signals` | query `{ limit?(≤200, default 50), unacked?('1') }` | `{ signals: WaggleSignal[], total }` | No |
| POST | `/api/waggle/signals` | `{ type, content, workspaceId?, metadata? }` | `201 WaggleSignal` · `400 { error }` | No |
| PATCH | `/api/waggle/signals/:id/ack` | path `:id` | `{ acknowledged:true, id }` | No |
| GET | `/api/waggle/stream` | — (SSE; Origin echoed only if allow-listed) | `event: signal` frames of `WaggleSignal`; `:heartbeat` every 30s | **Yes (SSE)** |
| GET | `/api/tools/detect` | — | `200 ToolDetectionResult` · `500 { error, message }` | No |
| POST | `/api/tools/launch` | `{ id, installedPath, workspaceId?, cwd?, args? }` (Zod) | `202 LaunchResult` · `400 LaunchResult`/validation | No |
| GET | `/api/tools/processes` | — | `200 { processes: TrackedProcess[], total }` | No |
| POST | `/api/tools/kill` | `{ pid }` | `200 { ok, pid, reason }` · `404 not-tracked` · `500` | No |
| POST | `/api/tools/hooks` | `{ id, action:'install'\|'verify'\|'uninstall', cliPath? }` | `200 HookCommandResult` · `400` · `500 { error, message }` | No |
**Server-side defaulting on POST `/api/waggle-dance/signal`:** `id`=`randomUUID()`, `createdAt`=now, `senderId`=`'local'` if omitted, `teamId`=`personal::<senderId>` if omitted, `referenceId`/`routing`=`null`. The route wires real v2 deps (`emitSignal`→`bus.record`, `recordResponse`→`bus.record`, `recommendModel`→`null` stub). v1 deps are local stubs (the team-internal paths run through the cloud team workspace, not this loopback route).
---
## 12. Phase 3 — skill diffusion (closed-loop → bus)
When the agent's D1 closed learning loop fires (`onSkillDistillationFire`, wired in `chat.ts`), the server records a `skill_share` broadcast directly onto `server.signalBus`:
```
{ type:'broadcast', subtype:'skill_share',
teamId:'personal::<workspace>', senderId:'agent-loop:<persona>',
content:{ tool:'waggle-agent', patternKey, toolsUsed[], directive, sessionId, workspaceId } }
```
Via the bridge this surfaces in the UI as a `waggle-dance:handoff` signal, so MCP-consuming external tools can adopt the soon-to-be-authored skill. The whole emission is best-effort and gated on `server.signalBus` existing.
---
## 13. Frontend rebuild checklist
- **Activity feed (WaggleDanceApp):** consume `GET /api/waggle/stream` (SSE) for live signals and `GET /api/waggle/signals` for backfill; render by the 5 UX categories; ack via `PATCH /api/waggle/signals/:id/ack`. Drill-down detail is in `metadata.protocolMessage` (the raw `WaggleMessage`). If you want the raw protocol plane instead, poll `GET /api/waggle-dance/signals`.
- **Launcher dock (LauncherApp):** `GET /api/tools/detect` to list the 7 tools (installed/version/hooksInstalled); `POST /api/tools/hooks` for install/verify/uninstall (only enable for the 6 `HOOKS_COHORT` tools; `claude-desktop` has no hook action); `POST /api/tools/launch` (pass `installedPath` from the detect result + the current `workspaceId`); poll `GET /api/tools/processes` for the 'Running' badge; `POST /api/tools/kill` for stop.
- **Mission Control inventory tile:** the same `ToolDetectionResult` (count installed, count hooked).
- **Room sub-agent tiles:** driven by the notifications stream (`subagent_status`), fed by `onSubAgentStatus`/`emitSubagentStatus`; the underlying tools are `spawn_agent`/`list_agents`/`get_agent_result`.
---
## 14. Source map
| Concern | File |
|---|---|
| Protocol types | `packages/shared/src/types.ts` (`WaggleMessage`, `MessageType`, `MessageSubtype`) |
| Combo validation | `packages/waggle-dance/src/protocol.ts` |
| Dispatcher | `packages/waggle-dance/src/dispatcher.ts` |
| Hive-query types | `packages/waggle-dance/src/hive-query.ts` |
| v2 ring buffer | `packages/server/src/local/signal-bus.ts` |
| v2 routes | `packages/server/src/local/routes/waggle-dance.ts` |
| Bridge v2→legacy | `packages/server/src/local/waggle-dance-bridge.ts` |
| Legacy stream + SSE | `packages/server/src/local/routes/waggle-signals.ts` |
| Tool-detection types | `packages/shared/src/tool-detection.ts` |
| Tool-detection impl | `packages/agent/src/tool-detection.ts` |
| Launcher + hooks | `packages/agent/src/tool-launcher.ts` |
| Process tracker | `packages/agent/src/tool-process-tracker.ts` |
| Tools routes | `packages/server/src/local/routes/tools.ts` |
| Shim-core emitter | `packages/hive-mind-shim-core/src/signal-emitter.ts` |
| Hook event vocab | `packages/hive-mind-shim-core/src/hook-event-types.ts` |
| Sub-agent tools | `packages/agent/src/subagent-tools.ts` |
| Orchestrator | `packages/agent/src/subagent-orchestrator.ts` |
| Cross-workspace bus | `packages/agent/src/agent-message-bus.ts` + `agent-comms-tools.ts` |
| Skill-diffusion wiring | `packages/server/src/local/routes/chat.ts` (`onSkillDistillationFire`) |

439
docs/backend-map/sections/05f-subsystem-capabilities-tiers.md Normal file
View File

@@ -0,0 +1,439 @@
# 05f — Subsystem: Capabilities, Connectors, Trust & Tier Gating
**Purpose.** This section is the contract + mental model for everything an agent *can do* and *is allowed to do*: how it discovers a missing capability and proposes installing one, how integrations (connectors) become agent tools, the curated MCP server catalog the UI browses, and the two orthogonal gates that decide whether an action runs — the **trust/autonomy model** (Normal / Trusted / YOLO) and the **subscription tier** (FREE → PRO → TEAMS → ENTERPRISE, which alone unlocks the KVARK enterprise tools). If you are rebuilding the frontend, this tells you which lists to render, which POST/GET endpoints to call, what JSON each returns, and which buttons must show approval/upgrade gates.
All identifiers below are quoted verbatim from source. Files: `packages/agent/src/{capability-router.ts, capability-acquisition.ts, trust-model.ts, permissions.ts, confirmation.ts, credential-pool.ts, connector-registry.ts, connector-sdk.ts, kvark-tools.ts, connector-search.ts}`, `packages/agent/src/connectors/*`, `packages/shared/src/{mcp-catalog.ts, types.ts}`, `packages/server/src/local/routes/{connectors.ts, capabilities.ts}`, `packages/server/src/local/setup-connectors.ts`, `packages/server/src/middleware/assert-tier.ts`.
---
## 1. The Big Picture (mental model)
```mermaid
flowchart TD
subgraph Discovery["Capability Discovery (agent-side)"]
CR["CapabilityRouter.resolve(query)<br/>capability-router.ts"]
CA["searchCapabilities(need)<br/>capability-acquisition.ts"]
FC["find_connector tool<br/>connector-search.ts → MCP_CATALOG"]
end
subgraph Trust["Trust + Permission scoring"]
TM["assessTrust()<br/>trust-model.ts"]
PM["PermissionManager<br/>permissions.ts (read-only sandbox)"]
end
subgraph Gates["Run-time gates (per tool call)"]
CONF["needsConfirmationWithAutonomy()<br/>confirmation.ts (Normal/Trusted/YOLO)"]
TIER["requireTier() preHandler<br/>assert-tier.ts (FREE/PRO/TEAMS/ENTERPRISE)"]
end
subgraph Exec["Capability execution surfaces"]
CONN["ConnectorRegistry → connector_<id>_<action> tools<br/>connector-registry.ts + connectors/*"]
KVARK["createKvarkTools() — kvark_search etc.<br/>kvark-tools.ts (ENTERPRISE / KVARK-configured only)"]
CRED["CredentialPool<br/>credential-pool.ts (key rotation)"]
end
CR --> CA --> TM
FC --> CONN
TM --> CONF
CONN --> CONF
KVARK --> TIER
PM --> CONF
CONN --> CRED
```
**Two independent axes gate every action:**
1. **Trust / autonomy** — *"is this action destructive enough to need a click?"* Computed per tool call from the tool name/args and the session's `AutonomyLevel`. Pure runtime, no subscription involved. (`confirmation.ts`)
2. **Tier** — *"does this user's plan include this feature at all?"* Enforced on HTTP routes (`requireTier()`) and on tool *registration* (KVARK tools only exist when KVARK is configured, which is an ENTERPRISE concern). (`assert-tier.ts`, `kvark-tools.ts`)
A capability can be **discovered** without being **available** (e.g. a connector that exists in the registry but has no credentials), and **available** without being **runnable** (e.g. a write action that still needs confirmation at Normal autonomy).
---
## 2. Capability Routing — `CapabilityRouter` (`capability-router.ts`)
A pure, in-memory resolver that answers *"where could capability X come from?"* It ranks candidate sources by confidence. It does **not** install anything — it's the read-side of discovery.
### 2.1 Types
`CapabilitySource` (union): `'native' | 'skill' | 'plugin' | 'mcp' | 'subagent' | 'connector' | 'missing'`
**`CapabilityRoute`** (one ranked match):
| Field | Type | Nullable | Meaning |
|---|---|---|---|
| `source` | `CapabilitySource` | no | Which kind of provider this route points to |
| `name` | `string` | no | Provider identifier (tool/skill/plugin/server/role name, or the original query if `missing`) |
| `confidence` | `number` | no | 0–1 ranking score (see table 2.3) |
| `description` | `string` | no | Human-readable explanation of the match |
| `available` | `boolean` | no | Whether it can be used right now |
| `suggestion` | `string` | yes | Next-step hint (e.g. connect credentials, search marketplace) |
**`ConnectorInfo`** (input describing a registered connector for routing): `id: string`, `name: string`, `service: string`, `connected: boolean`, `actions: string[]`.
**`CapabilityRouterDeps`** (constructor input): `toolNames: string[]`, `skills: {name, content}[]`, `plugins: {name, description, skills?, mcpServers?}[]`, `mcpServers: string[]`, `subAgentRoles: string[]`, `mcpRuntime?: { isServerHealthy(name): boolean }`, `connectors?: ConnectorInfo[]`.
### 2.2 Resolution order (`resolve(query)`)
Sources are scanned in this fixed order, all matches collected, then sorted by `confidence` descending. If nothing matches, a single `source: 'missing'` route is returned with a `suggestion`.
| Order | Source | Match rule | Confidence |
|---|---|---|---|
| 1 | `native` | tool name exact-equals query | `1.0` |
| 1 | `native` | tool name partial-includes query | `0.8` |
| 1.5 | `connector` | query mentions connector id/service/name, or an action (underscores→spaces) | `0.75` (`available = connector.connected`) |
| 2 | `skill` | skill name matches query | `0.7` |
| 2 | `skill` | skill content matches query | `0.5` |
| 3 | `plugin` | plugin description or listed skill matches | `0.6` |
| 4 | `mcp` | server name matches; `available` from `mcpRuntime.isServerHealthy()` (else assumed healthy) | `0.45` |
| 5 | `subagent` | query hits `ROLE_KEYWORDS` for a role | `0.4` |
`ROLE_KEYWORDS` (hardcoded): `researcher`, `writer`, `coder`, `analyst`, `reviewer`, `planner` — each maps to ~5 trigger words (e.g. `coder: ['code','implement','program','develop','build']`).
### 2.3 Frontend note
There is no dedicated HTTP route that exposes `CapabilityRouter.resolve()` directly; it is invoked inside the agent loop. The frontend observes its results indirectly through chat output and the **capability-request marker** (§4.3).
---
## 3. Capability Acquisition — `capability-acquisition.ts`
The write-side proposal engine behind the agent tool **`acquire_capability`**. It searches active skills, on-disk starter skills, native tools, and pre-fetched marketplace candidates, scores them by keyword overlap, attaches a **trust assessment** to each, and returns a single human-grade proposal with one recommendation.
### 3.1 Types
`CapabilitySourceType`: `'native' | 'skill' | 'plugin' | 'mcp' | 'connector' | 'marketplace'`
`CapabilityAvailability`: `'active' | 'installed_inactive' | 'installable' | 'unavailable'`
**`CapabilityCandidate`:**
| Field | Type | Meaning |
|---|---|---|
| `name` | `string` | Candidate identifier |
| `type` | `CapabilitySourceType` | Kind of capability |
| `availability` | `CapabilityAvailability` | Lifecycle state |
| `description` | `string` | Human description (first line of skill or hint) |
| `source` | `string` | Origin label: `"native-tools"`, `"installed"`, `"starter-pack"`, `"marketplace"` |
| `matchScore` | `number` | 0–1 internal ranking |
| `matchReason` | `string` | e.g. `"name matches: risk; content mentions: project"` |
| `installAction` | `string \| null` | `"install_capability"` for installable; `null` if native/active |
| `trust` | `TrustAssessment?` | Attached during search (see §5) |
**`AcquisitionProposal`** (return of `searchCapabilities`): `need`, `gapDetected: boolean`, `summary: string` (Markdown), `candidates: CapabilityCandidate[]` (capped at **8**), `recommendation: CapabilityCandidate | null`, `alreadyHandled: boolean`.
### 3.2 Scoring & thresholds (load-bearing constants)
- Keyword extraction drops a built-in `STOP_WORDS` set and tokens `< 3` chars.
- `scoreMatch`: **name hits = 2 points, content hits = 1 point**, normalized by keyword count, clamped to 1.0.
- Inclusion thresholds: native `>= 0.15`, installed skills `>= 0.1`, starter skills `>= 0.1`, marketplace `>= 0.1`.
- `alreadyHandled` requires a best active candidate with `matchScore >= 0.4`.
- `gapDetected = !alreadyHandled && some candidate is installable`.
`NATIVE_TOOL_HINTS` maps ~18 native tool names to hint keyword strings used for scoring (e.g. `web_search`, `read_file`, `spawn_agent`, `generate_docx`, `query_knowledge`).
### 3.3 Install validation — `validateInstallCandidate(name, source, starterSkillsDir, installedSkillNames)`
Returns `InstallValidation { valid, error?, candidateName, candidateType, source, starterPath? }`. **Only `source === 'starter-pack'` is installable via the tool path.** Rejects unknown sources, missing files, and already-installed skills. (Marketplace/MCP/connector installs go through the UI marker path, not this tool — see §4.3.)
---
## 4. Agent-facing capability tools (request shapes)
These are **LLM tools**, not HTTP routes. The frontend never calls them directly, but it must render their *effects* (proposals, install cards, approval prompts). Defined in `skill-tools.ts` and `connector-search.ts`.
| Tool | Params (required) | Effect / output |
|---|---|---|
| `acquire_capability` | `need: string` | Runs `searchCapabilities`; returns the proposal `summary` (Markdown). Records an audit event when a gap with a recommendation is found. |
| `install_capability` | `name: string`, `source: string` | Installs a **starter-pack** skill into the active set, hot-loads it, returns its content. **Always confirmation-gated** (in `ALWAYS_CONFIRM`). Rejects names containing `..`, `/`, `\`. |
| `find_connector` | `query: string` (opt: `limit` default 10/cap 30, `category`) | Searches `MCP_CATALOG`; returns JSON `{query, catalogSize, matchCount, matches:[{id,name,category,description,capabilities,installCmd,url,official,matchScore}]}`. `offlineCapable: true`. |
| `list_connector_categories` | — | Lists every `MCP_CATEGORIES` entry with server counts. |
### 4.1 KVARK tools (tier-gated) — see §9.
### 4.2 Persona allowlists
Some personas list `'acquire_capability'`, `'install_capability'`, `'find_connector'` in their tool allowlist (`persona-data.ts`); read-only personas (e.g. `planner`) explicitly *cannot* install.
### 4.3 The capability-request marker (UI contract)
For **non–starter-pack** sources (marketplace / MCP / connector), the agent does **not** call `install_capability`. Instead it emits, on its own line, verbatim:
```
<!--waggle:capability_request {"name":"<name>","source":"<source>","reason":"<one-line why>"}-->
```
The frontend parser (`capability-request-parser.ts` → `CapabilityRequestCard`) turns this into a one-click **Install** card. `reason` is sanitized (strips `{}<>`, collapses whitespace, ≤140 chars). **The new frontend must detect and render this marker.**
---
## 5. Trust Model (`trust-model.ts`)
Source trust (provenance) and execution risk (what the content does) are scored **independently**, then summed. This is what powers the risk badge on install cards.
### 5.1 Core types
| Type | Values |
|---|---|
| `RiskLevel` | `'low' \| 'medium' \| 'high'` |
| `TrustSource` | `'builtin' \| 'starter_pack' \| 'local_user' \| 'third_party_verified' \| 'third_party_unverified' \| 'unknown'` |
| `ApprovalClass` | `'standard' \| 'elevated' \| 'critical'` |
| `AssessmentMode` | `'declared' \| 'heuristic' \| 'mixed'` |
| `RiskFactor` | `local_code_execution, filesystem_access, network_access, external_service_access, secret_access, browser_automation, unknown_source, missing_metadata, unverified_publisher` |
**`PermissionSummary`** (all `boolean`): `fileSystem, network, codeExecution, externalServices, secrets, browserAutomation`.
**`TrustAssessment`** (return of `assessTrust`): `riskLevel`, `trustSource`, `permissions: PermissionSummary`, `approvalClass`, `assessmentMode`, `explanation: string`, `factors: RiskFactor[]`.
### 5.2 Scoring
- **Source risk points** (`SOURCE_RISK_POINTS`): builtin `0`, starter_pack `0`, local_user `1`, third_party_verified `2`, third_party_unverified `4`, unknown `5`.
- **Permission risk points** (`PERMISSION_RISK_POINTS`): fileSystem `1`, network `1`, codeExecution `2`, externalServices `1`, secrets `2`, browserAutomation `1`.
- Empty content (non-builtin) adds `+1` and a `missing_metadata` factor.
- **`classifyRisk(points)`**: `<=2 → low`, `<=4 → medium`, `5+ → high`.
- **`deriveApprovalClass`**: low→`standard`, medium→`elevated`, high→`critical`.
Permissions are detected **heuristically** by regex pattern sets (`FS_PATTERNS`, `NET_PATTERNS`, `EXEC_PATTERNS`, `EXT_SERVICE_PATTERNS`, `SECRET_PATTERNS`, `BROWSER_PATTERNS`) and merged (OR) with any `declaredPermissions`. `resolveTrustSource(type, source)` maps `'starter-pack'→starter_pack`, `'installed'/'user-created'→local_user`, native→`builtin`, etc.
`formatTrustSummary(assessment)` renders the compact block shown on install cards: `Risk: **High** (heuristic) | Source: ... | Approval: Critical` + active permissions list.
---
## 6. Permissions sandbox (`permissions.ts`)
A simple allow/deny tool filter used to put an agent into read-only mode.
- **`READONLY_TOOLS`** (frozen): `read_file, search_files, search_content, git_status, git_diff, git_log, web_search, web_fetch, show_plan, search_memory, get_identity, get_awareness, query_knowledge, query_audit, list_skills, search_skills, list_connectors, list_harnesses`.
- **`PermissionManager`**: `{ blacklist?, whitelist? }`. `isAllowed(name)` = not blacklisted **and** (no whitelist or whitelisted). `filterTools(tools)` returns the allowed subset. `PermissionManager.sandbox()` = whitelist of `READONLY_TOOLS`.
This is the static/persona-level allowlist; it is distinct from the per-call confirmation gate (§7).
---
## 7. Autonomy Tiers — confirmation gating (`confirmation.ts`)
This is the **Normal / Trusted / YOLO** model the task asks about. It decides, per tool call, whether the UI must show an approval prompt. It is a **UX lever, not a permission system** — a hardcoded critical blacklist always wins.
`AutonomyLevel = 'normal' | 'trusted' | 'yolo'`.
### 7.1 What always needs confirmation at Normal — `needsConfirmation(toolName, args)`
- **`ALWAYS_CONFIRM`** set: `write_file, edit_file, generate_docx, git_commit, git_push, git_pr, git_merge, install_capability, read_other_workspace, list_workspace_files, read_other_workspace_file`.
- **Connector tools** (`connector_*`): risk derived from the **tool name only** (never trust args — anti-injection). `send_email`/`send_template` are `CONNECTOR_HIGH_RISK_ACTIONS` (always confirm); otherwise confirm if name matches `_(create|update|delete|send|post|transition|remove|add|set|put)_`.
- **`bash`**: auto-approved only if it matches a `SAFE_BASH_PATTERNS` entry **and** contains no chain operator (`&& || ; |`); confirmed if it matches `DESTRUCTIVE_BASH_PATTERNS` (`rm -rf`, `del`, `format`, `dd if=`, `git push/reset/rebase`, `sudo`, `powershell`, exfil `curl -d`, etc.); unknown bash → confirm by default.
### 7.2 Approval class — `getApprovalClass(toolName, args)`
Connector high-risk → `critical`; connector write → `elevated`; `install_capability` reads `args._riskLevel` (`high`→critical, `medium`→elevated); everything else → `standard`.
### 7.3 The three levels — `needsConfirmationWithAutonomy(toolName, args, level)`
| Level | Behavior |
|---|---|
| `normal` | Exactly reproduces `needsConfirmation` — gate everything it flags. |
| `trusted` | Auto-pass `TRUSTED_AUTOPASS` = `{write_file, edit_file, generate_docx, read_other_workspace, read_other_workspace_file}` and non-critical `bash`. **Still gate** git push/commit/pr/merge, `install_capability`, connector writes, cross-workspace writes. |
| `yolo` | Auto-pass everything **except** `isCriticalNeverAutopass`. |
### 7.4 Critical blacklist — `isCriticalNeverAutopass()` (never auto-passes, even at YOLO)
- bash matching `CRITICAL_NEVER_AUTOPASS`: `rm -rf /` or `~`, `rm -rf $HOME`, `rm -rf /*`, any `sudo`, `format C:`, `mkfs`, `reg delete`, `dd if=… of=/dev`, forced `git push --force` to main/master/production, fork bomb.
- `install_capability` with `_riskLevel === 'high'`.
- `git_push` with `force` to `main`/`master`/`production`.
### 7.5 Frontend contract for autonomy
The chat request body carries an autonomy override (read in `chat.ts`):
```jsonc
{ "autonomy": { "level": "trusted" | "yolo", "expiresAt": <ms epoch> } }
```
If `expiresAt` is absent or in the past, the **server falls back to `normal`** (it owns the final say). When elevated autonomy pre-approves a tool that would normally gate, the stream emits a `step` event `⚡ <tool> auto-approved (<level>)` and an `approval_auto` audit row. The frontend should let users pick a level + a temporary expiry, and surface the auto-approved steps.
```mermaid
flowchart TD
T["Tool call (name, args)"] --> B{needsConfirmation?}
B -- no --> RUN["Run silently"]
B -- yes --> L{AutonomyLevel}
L -- normal --> PROMPT["Show approval prompt"]
L -- trusted/yolo --> C{isCriticalNeverAutopass?}
C -- yes --> PROMPT
C -- no --> D{trusted?}
D -- "yolo" --> AUTO["Auto-approve + audit ⚡"]
D -- "trusted" --> E{in TRUSTED_AUTOPASS or safe bash?}
E -- yes --> AUTO
E -- no --> PROMPT
```
---
## 8. Connectors
### 8.1 Runtime SDK (`connector-sdk.ts`)
**`WaggleConnector`** interface (server-side executable). Key readonly fields: `id`, `name`, `description`, `service`, `authType: 'bearer'|'oauth2'|'api_key'|'basic'`, `actions: ConnectorAction[]`, `substrate: 'waggle'|'kvark'`, `logoUrl?`, `category?`, `setupGuide?`. Methods: `connect(vault)`, `healthCheck(): Promise<ConnectorHealth>`, `execute(action, params): Promise<ConnectorResult>`, `toDefinition(status): ConnectorDefinition`.
**`ConnectorAction`** (runtime): `name`, `description`, `inputSchema` (JSON Schema), `outputSchema?`, `riskLevel: 'low'|'medium'|'high'`.
**`ConnectorResult`**: `{ success: boolean, data?: unknown, error?: string }`.
`BaseConnector` (abstract) supplies `toDefinition()` and `deriveCapabilities()` — maps action risk to `read`/`write`/`search` capability tags (low→read, medium/high→write, name contains search/find/list→search) — plus `safeErrorText()` truncation defense.
### 8.2 Registry (`connector-registry.ts`)
`ConnectorRegistry(vault, auditLogger?)`:
| Method | Returns | Notes |
|---|---|---|
| `register(c)` / `unregister(id)` | — / `boolean` | Map keyed by `connector.id` |
| `getAll()` / `get(id)` | `WaggleConnector[]` / `WaggleConnector?` | |
| `getConnected()` | `WaggleConnector[]` | A connector is "connected" if vault holds a non-expired credential, **or** its id is in `ALWAYS_CONNECTED = {slack-mock, teams-mock, discord-mock}` |
| `getDefinitions()` | `ConnectorDefinition[]` | Serializable; status `connected`/`expired`/`disconnected` from vault |
| `healthCheck(id)` | `ConnectorHealth \| null` | |
| `generateTools()` | `ToolDefinition[]` | **One tool per action**, named `connector_<id>_<action>`; description `[<Name>] <action desc>`; every execution is audit-logged with `requiresApproval = action.riskLevel !== 'low'`; errors returned as `{success:false,error}` JSON |
### 8.3 Built-in connectors registered at startup (`setup-connectors.ts`)
`registerConnectors(vault)` registers **30** connectors:
GitHub, Slack, Jira, Email, Google Calendar, Discord, Linear, Asana, Trello, Monday, Notion, Confluence, Obsidian, HubSpot, Salesforce, Pipedrive, Airtable, GitLab, Bitbucket, Dropbox, Postgres, Gmail, Google Docs, Google Drive, Google Sheets, MS Teams, Outlook, OneDrive, OneNote, Composio.
(Connector implementation files live in `packages/agent/src/connectors/*-connector.ts`.)
### 8.4 Shared serializable types (`packages/shared/src/types.ts`)
`ConnectorStatus = 'connected' | 'disconnected' | 'expired' | 'error'`.
**`ConnectorActionMeta`**: `name`, `description`, `riskLevel: 'low'|'medium'|'high'`.
**`ConnectorDefinition`** (what the UI renders):
| Field | Type | Nullable | Meaning |
|---|---|---|---|
| `id` | `string` | no | Connector id |
| `name` | `string` | no | Display name |
| `description` | `string` | no | What it does |
| `service` | `string` | no | Underlying service |
| `authType` | `'api_key'\|'oauth2'\|'bearer'\|'basic'` | no | Credential type the connect form needs |
| `status` | `ConnectorStatus` | no | Live from vault |
| `capabilities` | `('read'\|'write'\|'search')[]` | no | Derived from action risk |
| `substrate` | `'waggle'\|'kvark'` | no | Which substrate manages it |
| `tools` | `string[]` | no | Tool names it provides when connected (`connector_<id>_<action>`) |
| `config` | `Record<string,unknown>` | yes | Connector-specific config |
| `actions` | `ConnectorActionMeta[]` | yes | Present when SDK connector loaded |
| `logoUrl` | `string` | yes | SVG logo CDN url |
| `category` | `'productivity'\|'development'\|'crm'\|'data'\|'communication'\|'storage'\|'integration'` | yes | |
| `setupGuide` | `string` | yes | 1–2 sentences: which credential and where to get it |
**`ConnectorHealth`**: `id`, `name`, `status: ConnectorStatus`, `lastChecked: string` (ISO), `error?`, `tokenExpiresAt?`.
### 8.5 Connector HTTP routes (`connectors.ts`)
| Method | Path | Request body | Response | Streaming |
|---|---|---|---|---|
| GET | `/api/connectors` | — | `{ connectors: ConnectorDefinition[] }` (`registry.getDefinitions()`; `{connectors:[]}` if no registry) | no |
| GET | `/api/connectors/:id/health` | — | `ConnectorHealth`; `404` if not found; `502` sanitized degraded `ConnectorHealth` on probe throw | no |
| POST | `/api/connectors/:id/connect` | `{ token?, apiKey?, refreshToken?, expiresAt?, scopes?, email? }` (one of `token`/`apiKey` required) | `{ connected: true, connectorId }`; `400` if no credential; `404` unknown id; `503` if vault unavailable. Stores via `vault.setConnectorCredential`; re-runs `connector.connect`; extra `email` → `connector:<id>:email` | no |
| POST | `/api/connectors/:id/disconnect` | — | `{ disconnected, connectorId, cleanedKeys }`. Deletes `connector:<id>` + all `connector:<id>:*` sub-keys | no |
---
## 9. KVARK enterprise tools (tier-gated) — `kvark-tools.ts`
KVARK is the sovereign enterprise substrate at the top of the funnel. These tools are the agent's interface to KVARK retrieval. **They are only registered when KVARK is configured** — `getKvarkConfig(vault)` reads `kvark:connection` from the vault and returns `null` when absent. The wiring guard (verified by `packages/server/tests/kvark/kvark-wiring.test.ts`) is effectively `if (getKvarkConfig(vault)) tools.push(...createKvarkTools({client}))` → **exactly 4 tools** when configured, **0** otherwise. FREE/PRO/TEAMS users with no KVARK connection never see them; KVARK connection is an ENTERPRISE concern.
`createKvarkTools({ client: KvarkClientLike })` returns:
| Tool | Params (required) | Output (formatted text) |
|---|---|---|
| `kvark_search` | `query: string` (opt `limit`, default 10) | Ranked enterprise docs with score + `[KVARK: <type>: <title>]` attribution + doc id |
| `kvark_feedback` | `document_id: number`, `query: string`, `useful: boolean` (opt `reason`) | Confirms feedback; "not supported" if client lacks `feedback` |
| `kvark_action` | `action_type, entity_type, entity_id, payload, reason` (all required) | **Governed write** (e.g. Jira comment, Slack post). Returns executed/denied/queued with `auditRef`. **Requires user approval.** |
| `kvark_ask_document` | `document_id: string`, `question: string` | Answer + source references for one document |
All calls route through `KvarkClient` only (no direct fetch). Errors map by `err.name` to friendly degradations: `KvarkUnavailableError`, `KvarkAuthError`, `KvarkNotImplementedError`, `KvarkNotFoundError`, `KvarkServerError`. `parseSearchResults()` produces `KvarkStructuredResult { content, documentId, title, score, documentType, attribution }` for combined retrieval.
> Per `CLAUDE.md §9`, only TEAMS/ENTERPRISE expose KVARK; the live gate here is the **KVARK-configured** guard plus the enterprise-pack route below.
---
## 10. Tier gating (subscription axis) — `assert-tier.ts`
`requireTier(minimumTier: Tier)` is a Fastify `preHandler`. It reads the effective tier from `config.json` (`getEffectiveTier(tier, trialStartedAt)` — TRIAL decays to FREE), then `assertTierCapability`. On failure returns **`403`**:
```json
{ "error": "TIER_INSUFFICIENT", "message": "This feature requires the PRO tier. You are on FREE.",
"required": "PRO", "actual": "FREE", "upgradeUrl": "https://waggle-os.ai/upgrade" }
```
The frontend must catch `403 TIER_INSUFFICIENT` and show an upgrade prompt (use `required`/`upgradeUrl`). Default tier when unreadable is `FREE`.
**Tier-gated routes (verified `requireTier` usage):**
| Method | Path | Min tier |
|---|---|---|
| POST | `/api/marketplace/install` | PRO |
| POST | `/api/marketplace/publish` | PRO |
| GET | `/api/marketplace/enterprise-packs` | ENTERPRISE (also requires KVARK configured, else returns `{ kvarkRequired:true }`) |
| POST | `/api/personas` | PRO |
| POST | `/api/personas/generate` | PRO |
| GET | `/api/cost/by-workspace` | TEAMS |
| POST | `/api/cloud-sync/toggle` | TEAMS |
| GET | `/api/admin/overview` | TEAMS |
| POST | `/api/admin/audit-export` | TEAMS |
| POST | `/api/team/connect` | TEAMS |
| GET | `/api/team/governance/permissions` | ENTERPRISE |
---
## 11. Capability status dashboard — `capabilities.ts` routes
Read-only introspection over plugins, MCP servers, skills, tools, commands, hooks, and workflow templates. Powers a "what can this agent do" panel.
| Method | Path | Request | Response | Streaming |
|---|---|---|---|---|
| GET | `/api/capabilities/status` | — | `{ plugins:[{name,state,tools,skills}], mcpServers:[{name,state,healthy,tools}], skills:[{name,length}], tools:{count,native,plugin,mcp}, commands:[{name,description,usage}], hooks:{registered:10, recentActivity:[{event,timestamp,cancelled,reason}]}, workflows:[{name,description,steps}] }`; `500` on error | no |
| POST | `/api/capabilities/plugins/:name/enable` | — | `{ ok:true, name, state:'active' }`; `503` if no runtime; `400` on failure | no |
| POST | `/api/capabilities/plugins/:name/disable` | — | `{ ok:true, name, state:'disabled' }`; `503`/`400` | no |
---
## 12. MCP Server Catalog — `mcp-catalog.ts`
A curated, **static** catalog the ConnectorsApp MCP tab browses and `find_connector` searches. Not connected at runtime — it's a discovery/install-command directory.
**`McpServer`**: `id`, `name`, `description`, `author`, `category` (one of `MCP_CATEGORIES`), `url` (repo), `installCmd` (e.g. `npx @modelcontextprotocol/server-postgres`), `capabilities: string[]`, `official?: boolean`, `logo?: string`.
`MCP_CATEGORIES` (14): `Database, Files, Web, Code, Communication, Productivity, Analytics, Cloud, DevTools, Business, AI & ML, Security, Media, Utilities`. `CATEGORY_EMOJI` maps each to an emoji.
`MCP_CATALOG` holds the full server list (~140 entries spanning all categories; e.g. PostgreSQL, GitHub, Slack, Notion, Stripe, OpenAI, HashiCorp Vault, Figma, etc.). **Uniqueness is enforced at module load** by `assertCatalogUnique()` — it throws (failing the build) on a colliding normalized id or duplicate repo `url`. `normalizeMcpId(raw)` collapses npm scopes / `mcp-server-` / `server-mcp` affixes so `"GitHub"`, `"github-mcp"`, `"mcp-server-github"`, `"@modelcontextprotocol/server-github"` all normalize to `github`; reuse it when matching against external lists.
> There is no HTTP route in this section that serves the raw catalog; it is imported directly by the agent (`find_connector`) and bundled into the web app via `@waggle/shared`. The frontend can import `MCP_CATALOG`/`MCP_CATEGORIES` from shared.
---
## 13. Credential Pool — `credential-pool.ts`
Round-robin API-key rotation with automatic cooldown — the throughput/rate-limit layer behind provider keys (orthogonal to connector credentials, which live in vault).
**`CredentialEntry`**: `name` (vault key), `key`, `status: 'active'|'cooldown'|'disabled'`, `cooldownUntil: number|null`, `lastError: string|null`, `successCount`, `errorCount`.
**`PoolStatus`** (for monitoring UIs): `provider`, `totalKeys`, `activeKeys`, `cooldownKeys`, `disabledKeys`, `entries: [{name,status,cooldownUntil,lastError,successCount,errorCount}]`.
**`CredentialPool`** key methods: `addCredential(name,key)`, `getKey(): string|null` (round-robin, auto-recovers expired cooldowns), `getNameForKey(key)`, `reportSuccess(key)`, `reportError(key, statusCode, msg?): boolean` (returns whether other keys remain), `hasAvailableKeys()`, `getStatus()`, `size`.
**Cooldown policy (`reportError`):**
| HTTP code | Effect |
|---|---|
| `401` | Permanently `disabled` (invalid/revoked) |
| `402` | `cooldown` for `paymentCooldownMs` (default **24h**) |
| `429` | `cooldown` for `rateLimitCooldownMs` (default **1h**) |
| other (500/503/…) | `cooldown` for **5 min** |
`loadCredentialPool(vault, provider, maxKeys=10)` loads vault keys by convention `provider`, `provider-2`, `provider-3`, … (stops at first gap). `extractStatusCode(err)` pulls a status from `.status`/`.statusCode` or the error message.
---
## 14. How tier + trust gate together (worked examples)
| Scenario | Tier check | Trust/autonomy check | Net result |
|---|---|---|---|
| FREE user, agent calls `connector_github_create_issue` | none (connector run is not a `requireTier` route) | `needsConfirmation` → true (write) → Normal prompts; Trusted still gates; YOLO auto-passes | Runs after approval (or auto at YOLO) — **if** the connector is connected |
| FREE user clicks "Install marketplace pack" | `POST /api/marketplace/install` → `403 TIER_INSUFFICIENT` (needs PRO) | n/a (blocked before reaching the gate) | Blocked → upgrade prompt |
| ENTERPRISE user, KVARK configured, agent calls `kvark_action` | KVARK tools registered (config present) | `kvark_action` requires user approval | Governed action runs after approval, with audit ref |
| Any tier, agent runs `bash: ls` | none | matches `SAFE_BASH_PATTERNS`, no chain op → no confirmation | Runs silently |
| Any tier, YOLO, agent runs `bash: sudo rm -rf /` | none | `isCriticalNeverAutopass` → true | **Still prompts** even at YOLO |
| PRO user installs starter-pack skill via `install_capability` | none | in `ALWAYS_CONFIRM`; approval class from `_riskLevel` | Prompts (critical if high-risk) |
**Rule of thumb:** *Tier decides whether the door exists; trust/autonomy decides whether it needs a key turn each time you walk through.*

547
docs/backend-map/sections/05g-subsystem-skills-marketplace-wiki.md Normal file
View File

@@ -0,0 +1,547 @@
# Subsystem 05g — Skills Lifecycle · Marketplace · Wiki-Compiler
**Purpose.** This subsystem covers everything a user installs, authors, recommends, retires, browses-and-buys, and compiles-into-knowledge. It spans three packages — `@waggle/agent` (skill lifecycle helpers), `@waggle/marketplace` (the package catalog + installer + security gate), and `@waggle/wiki-compiler` (turns memory frames into an interlinked wiki) — all exposed to the frontend through Fastify routes under `/api/skills/*`, `/api/plugins/*`, `/api/marketplace/*`, and `/api/wiki/*`. **Strategically, skills + connectors are the paid-tier upgrade trigger:** custom skills and the marketplace are gated to PRO and above (`customSkills: false` on FREE, `requireTier('PRO')` on install/publish), while Memory + Harvest + the wiki stay free.
---
## 1. Mental Model
A **skill** is a Markdown file (`~/.waggle/skills/{name}.md`) with optional YAML frontmatter. Its content is injected into the agent's system prompt. Skills have a full lifecycle: authored/templated → recommended in context → usage-tracked → auto-retired when idle → hot-reloaded when the file changes on disk.
The **marketplace** is a local SQLite catalog (`~/.waggle/marketplace.db`) of three installable package kinds — `skill`, `plugin`, `mcp` — synced from ~30 external sources (GitHub, npm, ClawHub, SkillsMP, LobeHub, awesome-lists, web registries). Installing a package runs a multi-layer **SecurityGate** scan, then writes files to `~/.waggle/skills/`, `~/.waggle/plugins/`, or `.mcp.json`.
The **wiki-compiler** reads the personal memory substrate (FrameStore + KnowledgeGraph + HybridSearch) and produces five page types (entity / concept / synthesis / index / health) as Markdown, persisted to a `wiki_pages` SQLite table, with incremental compilation via a frame-ID watermark and exporters to Obsidian and Notion.
```mermaid
flowchart TD
subgraph Skills["Skill Lifecycle (@waggle/agent)"]
SC[skill-creator.ts<br/>generateSkillMarkdown / detectWorkflowPattern]
SF[skill-frontmatter.ts<br/>parseSkillFrontmatter / scopes]
SR[skill-recommender.ts<br/>SkillRecommender.recommend]
SU[skill-usage.ts<br/>recordSkillUsage JSON store]
SRet[skill-retirement.ts<br/>retireStaleSkills]
SW[skill-watcher.ts<br/>watchSkillDirectory hot-reload]
DISK[(~/.waggle/skills/*.md)]
SC --> DISK
DISK --> SF
DISK --> SR
DISK <--> SW
SU --> SRet
DISK --> SRet
end
subgraph MP["Marketplace (@waggle/marketplace)"]
MDB[(marketplace.db<br/>sources/packages/packs/installations/scan_history)]
SYNC[MarketplaceSync<br/>9 adapters]
INST[MarketplaceInstaller]
GATE[SecurityGate<br/>4 scan layers]
SYNC --> MDB
MDB --> INST
INST --> GATE
INST --> DISK
INST --> MCPJSON[(.mcp.json)]
INST --> PLUG[(~/.waggle/plugins/)]
end
subgraph Wiki["Wiki Compiler (@waggle/wiki-compiler)"]
WC[WikiCompiler]
WS[CompilationState<br/>wiki_pages + watermark]
SYN[resolveSynthesizer<br/>anthropic→ollama→echo]
MIND[(personal.mind:<br/>KnowledgeGraph/FrameStore/HybridSearch)]
MIND --> WC
WC --> SYN
WC --> WS
WS --> OBS[Obsidian export]
WS --> NOT[Notion export]
end
API[Fastify routes<br/>/api/skills /api/plugins<br/>/api/marketplace /api/wiki] --> Skills
API --> MP
API --> Wiki
```
---
## 2. Skill Frontmatter & Scopes (`packages/agent/src/skill-frontmatter.ts`)
Skills may begin with a `---`-delimited YAML-ish block. The parser is hand-rolled (no YAML dependency): it handles top-level `key: value` pairs plus a nested `permissions:` block.
### `SkillFrontmatter`
| Field | Type | Nullable | Meaning |
|---|---|---|---|
| `name` | `string` | yes | Display name |
| `description` | `string` | yes | One-line description |
| `scope` | `SkillScope` | yes | Where the skill is available. Defaults to `'personal'` when omitted |
| `promoted_from` | `SkillScope[]` | yes | Audit trail of prior scopes; appended on each promotion, never rewritten |
| `permissions` | `Partial<{fileSystem, network, codeExecution, externalServices, secrets, browserAutomation: boolean}>` | yes | Declared permission flags |
### `SkillScope` & promotion chain
`type SkillScope = 'personal' | 'workspace' | 'team' | 'enterprise'`. The ordered constant `SKILL_SCOPE_ORDER` drives one-step-at-a-time promotion (demotion is NOT supported).
| Export | Signature | Behavior |
|---|---|---|
| `parseSkillFrontmatter` | `(content: string) => ParsedSkill` | Returns `{ frontmatter, body }`. If no leading `---` or no closing `\n---`, returns empty frontmatter + full content as body |
| `nextScope` | `(current: SkillScope) => SkillScope \| null` | Next scope up, or `null` at `enterprise` |
| `serializeFrontmatter` | `(fm: SkillFrontmatter, body: string) => string` | Rebuilds a `SKILL.md` string preserving scope + `promoted_from`; used by `promote_skill` to write the change to disk |
`ParsedSkill` = `{ frontmatter: SkillFrontmatter; body: string }`.
> Note: the **server-side** skill loader (`loadSkills` in `prompt-loader.ts`) does NOT parse frontmatter — it loads each `.md` file as `{ name, content }` (trimmed). Frontmatter parsing is used by promotion/validation paths, not by the prompt-injection path.
---
## 3. Skill Creation (`packages/agent/src/skill-creator.ts`)
### `SkillTemplate`
| Field | Type | Meaning |
|---|---|---|
| `name` | `string` | Human name (kebab-cased on output) |
| `description` | `string` | Description line |
| `triggerPatterns` | `string[]` | Phrases that should activate the skill |
| `steps` | `string[]` | Ordered procedure |
| `tools` | `string[]` | Tools the skill uses |
| `category` | `string` | Free-form category |
| Export | Signature | Behavior |
|---|---|---|
| `generateSkillMarkdown` | `(template: SkillTemplate) => string` | Emits a valid `SKILL.md` with `---name/description---` frontmatter, then `# Title`, optional `## Trigger Patterns`, `## Steps` (numbered), `## Tools Used`, `## Category`. Name is kebab-cased. Output is compatible with `validateSkillMd` (`@waggle/sdk`) and `parseSkillFrontmatter` |
| `detectWorkflowPattern` | `(messages: Array<{role, content, toolsUsed?}>) => SkillTemplate \| null` | Mines session history for a repeatable tool sequence. Requires ≥6 messages and ≥3 tools; finds a 3-5-tool window repeated ≥2×; infers name (`tool-then-tool-then-tool`), description, trigger patterns (first 3 user msgs truncated to 80 chars), and a category via `inferCategory` (keyword map → research/coding/knowledge/writing/planning/general) |
---
## 4. Skill Recommendation (`packages/agent/src/skill-recommender.ts`)
A keyword/synonym/bigram + TF-IDF scorer that suggests installed skills relevant to a conversation context. No embeddings — pure lexical.
### Types
```
SkillRecommendation = { skillName: string; reason: string; relevanceScore: number /* 0-1 */ }
SkillRecommenderDeps = { getSkills: () => Array<{name, content}>; activeSkills?: string[] }
```
`class SkillRecommender(deps)` → `recommend(context: string, topN = 3): SkillRecommendation[]`.
**Scoring weights** (multiplied by an IDF weight `log(totalDocs/df)+1`, rarer terms score higher):
| Signal | Weight |
|---|---|
| Exact keyword in skill name | `3×` |
| Synonym match in name | `2×` |
| Exact keyword in content (× small TF boost, capped) | `1×` |
| Synonym match in content | `0.7×` |
| Bigram (phrase) overlap in combined text | `+0.5` flat per bigram |
Score normalized against `keywords.length * 3`, capped at 1.0, rounded to 3 decimals; results `< 0.05` dropped. Active skills (`deps.activeSkills`) are excluded. There are 13 hardcoded `SYNONYM_CLUSTERS` (review/code/write/research/plan/decide/risk/meeting/brainstorm/task/team/explain/retrospective) and a STOP_WORDS set. The `reason` string is human-readable, e.g. `Skill name matches your topic: "review"`.
---
## 5. Skill Usage Tracking (`packages/agent/src/skill-usage.ts`)
A standalone JSON sidecar store at `~/.waggle/skill-usage.json` (NOT SQLite). Atomic writes (`.{pid}.tmp` + rename). Missing/corrupt file ⇒ `{}`.
**Schema:** `Record<skillStem, { lastUsedAt: ISOstring; count: number }>` (`SkillUsageIndex` / `SkillUsageEntry`).
| Export | Signature | Behavior |
|---|---|---|
| `getSkillUsagePath` | `(waggleHome) => string` | `{waggleHome}/skill-usage.json` |
| `loadSkillUsage` | `(waggleHome) => SkillUsageIndex` | `{}` on any FS/parse failure |
| `saveSkillUsage` | `(waggleHome, index) => void` | Atomic write; mkdirs home if absent |
| `recordSkillUsage` | `(waggleHome, skillName, nowFn?) => SkillUsageEntry` | Sets `lastUsedAt=now`, increments `count` |
| `forgetSkillUsage` | `(waggleHome, skillName) => void` | Deletes the entry (called after retirement) |
---
## 6. Skill Retirement / Decay (`packages/agent/src/skill-retirement.ts`)
Walks `~/.waggle/skills/` and **moves** (not deletes) skills idle longer than `maxIdleDays` (default **90**) into `~/.waggle/skills-archive/` with a timestamp-prefixed filename. Recoverable; clears the usage entry on success.
**Retirement rules:** only personal-scope skills auto-retire (team/enterprise are co-owned → admin-only; workspace skills are left alone). "Last activity" = `skill-usage.json` timestamp, falling back to the file's mtime (so a never-used skill's decay window starts at install time). On stat error, assumed fresh (no over-retirement).
```
RetireOptions = { maxIdleDays?=90; dryRun?=false; now?: ()=>Date; improvementSignals?: ImprovementSignalStore }
RetireReport = { scanned: number; retired: string[]; archived: string[]/*abs paths*/;
skipped: Array<{name, reason}>; dryRun: boolean }
```
`retireStaleSkills(waggleHome, opts?) => RetireReport`. Best-effort per file (never throws on one bad file). When `improvementSignals` is provided, emits a `workflow_pattern` signal `retire:{skillName}` for observability.
---
## 7. Skill Hot-Reload Watcher (`packages/agent/src/skill-watcher.ts`)
Uses Node's built-in `fs.watch()` (not chokidar — keeps the Tauri binary small) on a single directory, filtered to `.md` direct children, with a **150 ms debounce** (editors emit 2-3 events per save). Creates the dir if missing; silently no-ops if `fs.watch` throws on the platform.
```
SkillWatcherOptions = { onChange: (changedFiles: string[]) => void; debounceMs?=150 }
SkillWatcherHandle = { close(): void /* idempotent */ }
watchSkillDirectory(dir, opts) => SkillWatcherHandle
```
Callback errors are swallowed so they never kill the watcher.
---
## 8. Skills/Plugins/Hooks HTTP API (`packages/server/src/local/routes/skills.ts`)
Base dir: `server.localConfig.dataDir || ~/.waggle`. Skills live in `{home}/skills/`, plugins in `{home}/plugins/`. On first run with an empty skills dir, **starter skills auto-install** (marker file `.starter-installed`). Every skill mutation (`POST`/`PUT`/`DELETE`) reloads `server.agentState.skills` in place and writes through `redactSkillContent` (strips secrets + user-home paths) and `computeSkillHash` (change detection). Skill writes apply path-traversal guards (`name` may not contain `..`, `/`, `\`, or spaces).
### Skills
| Method | Full path | Request | Response | Stream? |
|---|---|---|---|---|
| `GET` | `/api/skills` | — | `{ skills: [{name, length, preview(200ch)}], count, directory }` | no |
| `GET` | `/api/skills/:name` | path param | `{ name, content }` · 404 if missing · 400 on traversal | no |
| `POST` | `/api/skills` | `{ name, content }` | `{ ok, name, path }` · 400 on bad name | no |
| `POST` | `/api/skills/create` | `{ name, description, steps[], tools?, category? }` | `{ success, path, registered, skill:{...} }`. Generates via `generateSkillMarkdown`, kebab-cases name, records audit | no |
| `PUT` | `/api/skills/:name` | `{ content }` | `{ ok, name }` · 404 if missing | no |
| `DELETE` | `/api/skills/:name` | path param | `{ ok, name }` · 404 if missing | no |
| `GET` | `/api/skills/suggestions` | `?context=&topN=` | `{ suggestions: SkillRecommendation[], count }` · 400 if no context. Uses `SkillRecommender` | no |
| `GET` | `/api/skills/hash-status` | — | `server.skillHashStore.checkAll(...)` — which skills changed on disk | no |
| `POST` | `/api/skills/test` | `{ skillName, testInput? }` | Sandbox/dry-run: `{ skill:{...metadata}, wouldInject, wouldInjectLength, testPreview? }`. Parses frontmatter, shows what would be injected into the prompt without executing | no |
### Starter Pack & Capability Packs (built-in, from `@waggle/sdk`)
| Method | Full path | Request | Response |
|---|---|---|---|
| `POST` | `/api/skills/starter-pack` | — | `{ ok, installed[], count }` — install ALL starter skills |
| `GET` | `/api/skills/starter-pack/catalog` | — | `{ skills:[{id,name,description,family,familyLabel,state,isWorkflow}], families:[{id,label}] }`. `state` ∈ `active`/`installed`/`available` |
| `POST` | `/api/skills/starter-pack/:id` | path param | `{ ok, skill:{id,name,state} }` · 404/409 on missing/exists. Assesses trust + records audit |
| `GET` | `/api/skills/capability-packs/catalog` | — | `{ packs:[{...pack, skillStates, packState(available/incomplete/complete), installedCount, totalCount}] }` |
| `POST` | `/api/skills/capability-packs/:id` | path param | `{ ok, pack:{id,name}, installed[], skipped[], errors? }` — installs every skill in the pack |
The 17 known starter skills map to 7 capability **families** (`SKILL_FAMILIES`): writing, research, decision, planning, communication, code, creative. Three are multi-agent workflow skills (`WORKFLOW_SKILLS`): `research-team`, `review-pair`, `plan-execute`.
### Install Audit
| Method | Full path | Request | Response |
|---|---|---|---|
| `GET` | `/api/audit/installs` | `?limit=` (≤100) | `{ entries:[{id,timestamp,capabilityName,capabilityType,source,riskLevel,trustSource,approvalClass,action,initiator,detail}] }` |
### Plugins (managed by `PluginManager`)
| Method | Full path | Request | Response |
|---|---|---|---|
| `GET` | `/api/plugins` | — | `{ plugins[], count, directory }` |
| `POST` | `/api/plugins/install` | `{ sourceDir }` or `{ path }` | `{ ok, source }`. Hot-reloads via `pluginRuntimeManager.register/enable`. Called by `MarketplaceInstaller.notifyServer()` |
| `DELETE` | `/api/plugins/:name` | path param | `{ ok, name }` |
| `GET` | `/api/plugins/:name/tools` | path param | `{ pluginName, tools:[{name,description,parameters,hasImplementation,implPath,content}], toolsDir }` |
| `GET` | `/api/plugins/:name/tools/:toolName` | path | `{ exists, content, path }` — returns a generated template if no impl exists |
| `PUT` | `/api/plugins/:name/tools/:toolName` | `{ content }` | `{ ok, path, toolName }` — must `export` an `execute()` function; hot-reloads |
| `POST` | `/api/plugins/:name/tools` | `{ name, description, parameters? }` | `{ ok, tool, totalTools }` — appends tool decl to `plugin.json` · 409 if exists |
| `DELETE` | `/api/plugins/:name/tools/:toolName` | path | `{ ok, deleted }` · 404 if missing |
### Hooks (deny-rules at `~/.waggle/hooks.json`)
| Method | Full path | Request | Response |
|---|---|---|---|
| `GET` | `/api/hooks` | — | `{ rules:[{type,tools[],pattern}], total }` |
| `POST` | `/api/hooks` | `{ type:'deny', tools[], pattern }` | `{ ok, rules }` · 400 on bad shape |
| `DELETE` | `/api/hooks/:index` | path index | `{ ok, rules }` · 404 if out of range |
---
## 9. Marketplace Data Model (`packages/marketplace/src/types.ts`, `db.ts`)
SQLite DB at `~/.waggle/marketplace.db` via better-sqlite3 (WAL, `foreign_keys=ON`). FTS5 full-text index `packages_fts` joined on `packages.id = fts.rowid`. Auto-migrations on construct add `is_custom` and `sync_state` columns to `sources`. The `MarketplaceDB` constructor seeds the MCP registry **only** on non-empty DBs.
### Table: `sources` → `MarketplaceSource`
| Column | Type | Null | Meaning |
|---|---|---|---|
| `id` | number | no | PK |
| `name` | string | no | Internal key (e.g. `clawhub`) |
| `display_name` | string | no | UI label |
| `url` | string | no | Source URL |
| `source_type` | enum | no | `marketplace`/`registry`/`github_org`/`community_repo`/`curated_list`/`aggregator`/`npm_registry`/`official_marketplace`/`commercial_marketplace`/`tool`/`specification` |
| `platform` | string | no | e.g. `waggle` |
| `total_packages` | number | no | Cached count |
| `install_method` | enum | no | `npm`/`git_clone`/`download`/`api_fetch`/`cli`/`manual` |
| `api_endpoint` | string | yes | Sync endpoint |
| `description` | string | no | — |
| `last_synced_at` | string | yes | ISO |
| `is_custom` | boolean | no | User-added (vs built-in seed). Only `is_custom` sources are deletable |
| `sync_state` | TEXT(JSON) | yes | Resumable-pagination cursor (migration column) |
### Table: `packages` → `MarketplacePackage`
| Column | Type | Null | Meaning |
|---|---|---|---|
| `id` | number | no | PK |
| `source_id` | number | no | FK → sources |
| `name` | string | no | Unique within source (`UNIQUE(name, source_id)`) |
| `display_name` | string | no | UI name |
| `description` | string | no | — |
| `author` | string | no | — |
| `package_type` | enum | no | `skill`/`plugin`/`mcp_server`/`template`/`pack` |
| `waggle_install_type` | enum | no | `skill`/`plugin`/`mcp` — drives installer dispatch |
| `waggle_install_path` | string | no | e.g. `skills/x.md`, `plugins/x/`, `.mcp.json` |
| `version` | string | no | semver |
| `license` | string | yes | SPDX |
| `repository_url` | string | yes | — |
| `homepage_url` | string | yes | — |
| `downloads` | number | no | popularity sort |
| `stars` | number | no | popularity sort |
| `rating` | number | no | — |
| `rating_count` | number | no | — |
| `category` | string | no | one of `PACKAGE_CATEGORIES` ids |
| `subcategory` | string | yes | — |
| `install_manifest` | JSON | yes | `InstallManifest` (see below) — parsed on read |
| `platforms` | JSON string[] | no | — |
| `min_waggle_version` | string | yes | — |
| `dependencies` | JSON string[] | no | — |
| `packs` | JSON string[] | no | — |
| `created_at` / `updated_at` | string | no | ISO |
**Security columns** (added by the installer's `recordScanResult()`, optional; type `PackageSecurityColumns`): `security_status` (`unscanned`/`clean`/`low`/`medium`/`high`/`critical`), `security_score` (number), `last_scanned_at`, `content_hash`, `scan_engines` (JSON), `scan_findings` (JSON), `scan_blocked` (0/1). A package augmented with these is `ScannedPackage`.
### Table: `packs` + `pack_packages` → `MarketplacePack`
| Column | Type | Null | Meaning |
|---|---|---|---|
| `id` | number | no | PK |
| `slug` | string | no | Stable key (used by `/packs/:slug`) |
| `display_name` | string | no | — |
| `description` | string | no | — |
| `target_roles` | string | no | comma-sep role hints |
| `icon` | string | no | emoji |
| `priority` | enum | no | `core`/`recommended`/`optional` |
| `connectors_needed` | JSON string[] | no | — |
| `created_at` | string | no | — |
`pack_packages` is the many-to-many join (`pack_id`, `package_id`, `is_core`); `package_tags` exists for tag dedup.
### Table: `installations` → `Installation`
| Column | Type | Null | Meaning |
|---|---|---|---|
| `id` | number | no | PK |
| `package_id` | number | no | FK → packages |
| `installed_version` | string | no | — |
| `installed_at` | string | no | `datetime('now')` |
| `install_path` | string | no | where it landed |
| `status` | enum | no | `installed`/`updating`/`failed`/`uninstalled` |
| `config` | JSON | no | user settings — parsed on read |
`listInstallations()` returns the flat `InstalledPackageRow` (installation columns + `pkg_name`, `pkg_display_name`, `waggle_install_type`, `category`) — NOT a nested object.
### Table: `scan_history`
One row per scan (`package_id`, `scanned_at`, `overall_severity`, `security_score`, `content_hash`, `engines_used`, `findings`, `blocked`, `scan_duration_ms`, `triggered_by`).
### `InstallManifest` (stored as JSON on `packages.install_manifest`)
| Field | For | Meaning |
|---|---|---|
| `skill_url` / `skill_content` | skill | fetch URL or inline content |
| `plugin_manifest` (`PluginManifest`) / `git_url` | plugin | manifest to write / repo to clone |
| `mcp_config` (`McpServerConfig`) | mcp | `{ name, command, args[], env? }` |
| `npm_package` / `npm_args` | mcp/plugin | npm install target |
| `post_install` (`PostInstallHook[]`) | any | `run_command`/`create_file`/`append_config` |
`PluginManifest = { name, version, description, skills?[], mcpServers?[], settingsSchema?: Record<string, SettingField> }`; `SettingField = { type:'string'|'number'|'boolean', description, required?, default? }`.
### Search contract (`SearchOptions` → `SearchResult`)
`SearchOptions = { query?, type?, category?, pack?, source?, sort?: 'relevance'|'popular'|'recent'|'name', limit?=50, offset?=0 }`. Raw query is relaxed by `toFtsMatchQuery()` into an OR-of-prefix FTS5 expression (≥2-char tokens, capped at 24, `token*`); null ⇒ unfiltered listing fallback (never throws).
`SearchResult = { packages: MarketplacePackage[]; total; facets: { types, categories, sources: Record<string,number> }; installedCount }`.
---
## 10. Marketplace Installer (`packages/marketplace/src/installer.ts`)
`class MarketplaceInstaller(db, securityConfig?)`. Ensures `~/.waggle/skills/`, `~/.waggle/plugins/`, and `plugins/registry.json` exist on construct. Install flow:
1. Resolve package by id (404-style failure result if missing).
2. Skip if already installed and not `force`.
3. **SecurityGate pre-scan** of resolved content; record result to DB. If `scanResult.blocked` and not `forceInsecure` → return `success:false` with findings.
4. Dispatch on `waggle_install_type`:
- **skill** → write to `~/.waggle/skills/{name}.md` (inline content, `skill_url`, repo `SKILL.md`, or a generated stub), then `PUT /api/skills/{name}` to notify the server.
- **plugin** → `git clone` or `npm install` into `~/.waggle/plugins/{name}/`, write `plugin.json`, install bundled skills, update `registry.json`, run post-install hooks, then `POST /api/plugins/install`.
- **mcp** → optional global npm install, inject user settings into env vars, write into `.mcp.json` (`mcpServers[name]`).
5. On success, `recordInstallation(...)` and attach the scan result.
`InstallRequest = { packageId, installPath?, settings?, force?, forceInsecure? }`; `InstallResult = { success, packageId, packageName, installType, installPath, message, errors?, scanResult? }`. Pack install (`installPack(slug, {force?})`) loops package install and returns `PackInstallResult = { packSlug, packName, totalPackages, installed[], skipped[], failed[] }`. Also: `uninstall(id)`, `scanOnly(id) => ScanResult|null`, `getSecurityReport(id) => string`.
`MarketplaceInstaller.notifyServer()` posts to `WAGGLE_API_URL || http://localhost:3000`; failures are swallowed (files already on disk).
---
## 11. SecurityGate (`packages/marketplace/src/security.ts`) — the install gate
Four scan layers, run before any file is written. `block_threshold` defaults to **HIGH** (CRITICAL + HIGH block by default).
| Layer | Engine id | What |
|---|---|---|
| 1 | `gen_trust_hub` | Cloud URL pre-check (POST `https://ai.gendigital.com/api/scan/lookup`, 15s timeout, fail-open) |
| 2 | `cisco_skill_scanner` | Local deep scan of skill content via optional `skill-scanner` CLI / `cisco-scanner.ts` adapter |
| 3 | `mcp_guardian` | Pattern scan of MCP tool descriptions (optional `mcp-guardian` npm dep, else built-in `mcpPatternScan` fallback) |
| 4 | `waggle_heuristics` | Always-on regex rules `WAG-001..013` on content |
**Severity → score:** CRITICAL=0, HIGH=25, MEDIUM=60, LOW=85, CLEAN=100. Results SHA-256-hashed + cached for 24 h under `~/.waggle/security-cache/`.
`ScanResult = { package_name, package_type, scanned_at, overall_severity: Severity, security_score(0-100), findings: SecurityFinding[], engines_used: SecurityEngine[], content_hash, blocked, scan_duration_ms, ciscoScanResult? }`. `SecurityFinding = { rule_id, severity, category, title, description, location?, engine }`. `SecurityCategory` ∈ prompt_injection / data_exfiltration / malicious_code / privilege_escalation / suspicious_network / obfuscation / sensitive_path_access / tool_poisoning / cross_origin_escalation / rug_pull / untrusted_source. Built-in heuristics catch system-prompt manipulation, exfil commands, sensitive-path access (incl. `~/.waggle/*.mind`), code-exec patterns, credential harvesting, network beaconing, tool poisoning, hidden/zero-width content, supply-chain and cross-origin skill modification. `formatReport(result)` renders a human-readable text report.
---
## 12. Marketplace Sync (`packages/marketplace/src/sync.ts`)
`class MarketplaceSync(db, vaultLookup?)` → `syncAll(opts?: SyncOptions) => SyncResult[]`. Nine adapters, tried in priority order (first `canSync()` wins): `clawhub`, `skillsmp`, `lobehub`, `awesome-list`, `github-repo-content`, `npm-search`, `web-registry`, `github` (orgs/official), `generic`. Resumable pagination via `sources.sync_state` (429 → save cursor, return gracefully). After a full sync, `deduplicatePackages(db)` collapses normalized-name dupes (keeps highest `stars+downloads`). Vault keys for premium sources: `marketplace:source:{name}:api_key`.
`SyncOptions = { sources?, fullRefresh?, dryRun?, scanDuringSync?=false }`; `SyncResult = { source, added, updated, removed, errors[] }`. Exported helpers: `parseAwesomeListMarkdown`, `parseNpmSearchResults`, `normalizeName`, `deduplicatePackages`.
### Categories & MCP registry & enterprise packs
- `categories.ts`: 21 `PACKAGE_CATEGORIES` (`{id,name,icon,description}`) + `categorizePackage(name, desc)` keyword classifier + `recategorizeAll(db)`.
- `mcp-registry.ts`: `MCP_SERVERS: McpServerEntry[]` + `seedMcpServers(db)` (seeds curated MCP packages with real install manifests).
- `enterprise-packs.ts`: `ENTERPRISE_PACKS: EnterprisePack[]` (3 KVARK-conditional packs: enterprise-document-qa, compliance-workflow, knowledge-graph-enrichment) — only surfaced when KVARK is connected. `EnterprisePack = { slug, display_name, description, target_roles, icon, skills[], kvarkRequirements[] }`.
---
## 13. Marketplace HTTP API (`packages/server/src/local/routes/marketplace.ts`)
DB handle from `fastify.marketplace` (503 `Marketplace not available` if absent). **Tier gating** marks the upgrade triggers.
| Method | Full path | Request | Response | Tier gate | Stream? |
|---|---|---|---|---|---|
| `GET` | `/api/marketplace/search` | `?query=&type=&category=&pack=&source=&sort=&limit=&offset=` | `SearchResult` + each pkg annotated with `installed`, `scanStatus`(passed/failed/not_scanned/unavailable), `scanScore`; plus `categories` | — | no |
| `GET` | `/api/marketplace/packs` | — | `{ packs, total }` | — | no |
| `GET` | `/api/marketplace/packs/:slug` | path | `{ pack, packages[] }` · 404 | — | no |
| `GET` | `/api/marketplace/enterprise-packs` | — | `{ packs, total, kvarkRequired }` (empty + hint if KVARK not configured) | **ENTERPRISE** | no |
| `POST` | `/api/marketplace/install` | `{ packageId, installPath?, settings?, force?, forceInsecure? }` | `InstallResult` + `security:{severity,score,findingsCount,findings,warnings?}`; CRITICAL→403, HIGH→403 unless `force`, MEDIUM/LOW→proceeds-with-audit. 200 on success / 422 on failure | **PRO** | no |
| `POST` | `/api/marketplace/uninstall` | `{ packageId }` | `InstallResult` (200/422) | — | no |
| `GET` | `/api/marketplace/installed` | — | `{ installations: InstalledPackageRow[], total }` | — | no |
| `POST` | `/api/marketplace/security-check` | `{ packageId }` | `{ packageId, severity, score, blocked, enginesUsed, findingsCount, findings, durationMs, contentHash }` · 404 | — | no |
| `GET` | `/api/marketplace/sources` | — | `{ sources: (MarketplaceSource & {package_count})[], total }` | — | no |
| `POST` | `/api/marketplace/sources` | `{ name, url, displayName? }` | `201 { source, syncResult }`; auto-detects `source_type`, triggers an immediate sync · 400/409 | — | no |
| `DELETE` | `/api/marketplace/sources/:id` | path | `{ deleted, sourceId, name }` · 403 if built-in · 404 | — | no |
| `GET` | `/api/marketplace/categories` | — | `{ categories: PACKAGE_CATEGORIES, total }` | — | no |
| `POST` | `/api/marketplace/sync` | `{ sources? }` | `{ sourcesChecked, packagesAdded, packagesUpdated, errors[], details: SyncResult[] }`; emits a notification on new packages | — | no |
| `GET` | `/api/marketplace/security-status` | — | `{ ciscoScannerAvailable, jsSecurityGateVersion, totalScanned, totalPassed, totalFailed, hint? }` | — | no |
| `POST` | `/api/marketplace/publish` | `{ skillName }` | `201 { success, packageId, skillName, metadata, security }`; reads `~/.waggle/skills/{name}.md`, validates frontmatter (`validateSkillMd`), SecurityGate scan (403 if blocked), upserts under a `user-published` source | **PRO** | no |
The install/check routes construct a heuristics-only `SecurityGate` (cloud/cisco/guardian layers disabled at the route level for speed).
---
## 14. Wiki-Compiler Data Model (`packages/wiki-compiler/src/types.ts`, `state.ts`)
State lives in the **personal mind** SQLite DB (`@waggle/core` `MindDB`) — same DB as memory, not a separate file. `class CompilationState(db)` ensures two tables on construct.
### Table: `wiki_pages` → `PageRecord`
| Column | Type | Null | Meaning |
|---|---|---|---|
| `slug` | TEXT | no | PK (URL-safe) |
| `page_type` | TEXT | no | `entity`/`concept`/`synthesis`/`index`/`health` |
| `name` | TEXT | no | Display name |
| `content_hash` | TEXT | no | SHA-256(16-char) — change detection |
| `markdown` | TEXT | no | Full page body (migration-added column) |
| `frame_ids` | TEXT | no | JSON `number[]` of source frame IDs |
| `compiled_at` | TEXT | no | `datetime('now')` |
| `source_count` | INTEGER | no | distinct sources |
| `notion_page_id` | TEXT | yes | M-13 Notion export delta tracking (migration column) |
`upsertPage(...)` returns `{ action: 'created' | 'updated' | 'unchanged' }` — `unchanged` when `content_hash` matches (the incremental-compile no-op).
### Table: `wiki_watermark` → `CompilationWatermark`
Single-row (`id=1 CHECK`): `last_frame_id`, `last_compiled_at`, `pages_compiled`. `getMaxFrameId()` reads `MAX(id)` from `memory_frames`; `getFramesSince(id, limit)` pulls newer frames for the incremental "does this entity get mentioned in new frames?" check.
### Page frontmatter (`WikiPageFrontmatter`)
`{ type, name, entity_type?, confidence: number, sources: number, last_compiled: ISO, frame_ids: number[], related_entities: string[] }`. A `WikiPage` = `{ slug, frontmatter, markdown, contentHash }`.
---
## 15. Wiki Compilation (`packages/wiki-compiler/src/compiler.ts`)
`class WikiCompiler(kg, frames, search, state, config)` where `CompilerConfig = { synthesize: (prompt)=>Promise<string>; outputDir?='wiki'; minFramesPerPage?=2; maxFramesPerCall?=30; minConfidence?=0.3 }`.
| Method | Page type | How it builds |
|---|---|---|
| `compileEntityPage(entity)` | entity | HybridSearch on `entity.name`, gather frames + KG in/out relations; LLM via `entityPagePrompt`. Returns `null` if `< minFramesPerPage`. Confidence: >5 frames→0.9, >2→0.7, else 0.5 |
| `compileConceptPage(name)` | concept | Search on concept + `kg.searchEntities`; `conceptPagePrompt`. Confidence 0.85/0.6 |
| `compileSynthesisPage(topic)` | synthesis | Search 2× frames, group by `frame.source`; **needs ≥2 sources** else `null`. `synthesisPagePrompt` finds cross-source patterns/contradictions. slug = `synthesis-{topic}`. Confidence 0.85/0.65 |
| `compileIndex()` | index | Navigable catalog of all pages grouped by type with `[[wikilinks]]` |
| `compileHealth()` | (report) | See §16 |
| `compile({incremental?=true, concepts?})` | all | Orchestrates: entity pages (≤200 entities, skipping ones with no new frame mentions when incremental) → concept pages (`detectConcepts` or supplied) → synthesis pages → index → update watermark → health check |
`compile()` returns `CompilationResult = { pagesCreated, pagesUpdated, pagesUnchanged, entityPages[], conceptPages[], synthesisPages[], healthIssues, watermark, durationMs }`. Export helpers: `exportToMarkdown(): Map<slug,markdown>` and `exportToDirectory(dir)`. The three prompt builders live in `prompts.ts` and each instruct the LLM to cite frame IDs and output ONLY the body (frontmatter is added by `buildFrontmatter`).
---
## 16. Wiki Health Report (`compileHealth()` → `HealthReport`)
| Field | Type | Meaning |
|---|---|---|
| `totalEntities` | number | KG entity count |
| `totalFrames` | number | `frames.getStats().total` |
| `totalPages` | number | wiki_pages rows |
| `coverage` | number (0-1) | entity pages / compilable entities (entity with ≥1 relation OR type person/project); UI renders as % |
| `stalePageCount` | number | pages flagged `stale_page` |
| `issues` | `HealthIssue[]` | see below |
| `dataQualityScore` | number (0-100) | heuristic: entities(20) + frames(20/10) + pages(20/10) + 40 − 10×high-severity-issues |
| `compiledAt` | ISO | — |
`HealthIssue = { type: HealthIssueType, severity:'high'|'medium'|'low', description, entity?, frameIds?, suggestion? }`. `HealthIssueType` ∈ `contradiction` / `gap` / `orphan_entity` / `weak_confidence` / `stale_page` / `missing_page`. Detection: missing-page (KG entity with no page that has relations or is person/project), weak_confidence (<2 sources), orphan_entity (no relations at all), stale_page (compiled >30 days ago AND newer frames exist overall).
---
## 17. Wiki Synthesizer Resolution (`synthesizer.ts`)
`resolveSynthesizer(config?) => { synthesize: LLMSynthesizeFn; provider: 'anthropic'|'ollama'|'echo'; model }`. Priority chain:
1. **Anthropic** Haiku (`claude-haiku-4-5-20251001`) — if `ANTHROPIC_API_KEY` / `WAGGLE_ANTHROPIC_API_KEY` present and SDK importable.
2. **Ollama** — if `WAGGLE_OLLAMA_URL` reachable (`/api/tags` health check); model `WAGGLE_OLLAMA_MODEL || llama3.2`.
3. **Echo** fallback — no LLM; returns a structured stub summarizing frame content (so the wiki still renders without a key).
`SynthesizerConfig = { anthropicApiKey?, ollamaUrl?, ollamaModel?, maxTokens?=1500 }`.
---
## 18. Wiki HTTP API (`packages/server/src/local/routes/wiki.ts`)
All routes operate on `server.multiMind.personal` (the personal mind DB). **Critical UX gate:** `/compile` and `/health` return **503 `no_real_embedder`** when `embeddingProvider.getActiveProvider() === 'mock'` — a mock embedder produces zero-vector relevance, so the wiki refuses to compile rather than silently producing broken pages.
| Method | Full path | Request | Response | Stream? |
|---|---|---|---|---|
| `GET` | `/api/wiki/pages` | — | `PageRecord[]` (all pages) | no |
| `GET` | `/api/wiki/pages/:slug` | path | `PageRecord` · 404 | no |
| `GET` | `/api/wiki/pages/:slug/content` | path | `{ slug, markdown }` · 404 | no |
| `POST` | `/api/wiki/compile` | `{ mode?: 'incremental'\|'full', concepts?: string[] }` | `CompilationResult` + `{ llmProvider, llmModel }` · **503** if mock embedder | no |
| `GET` | `/api/wiki/health` | — | `HealthReport` · **503** if mock embedder | no |
| `GET` | `/api/wiki/watermark` | — | `CompilationWatermark` | no |
| `POST` | `/api/wiki/export/obsidian` | `{ outDir }` (absolute) | `ObsidianExportResult { outDir, filesWritten, indexPath, byType }` · 400/409/500 | no |
| `POST` | `/api/wiki/export/notion` | `{ rootPageUrl }` | `NotionExportStats { byType, pagesCreated, pagesUpdated, pagesUnchanged, pagesFailed, errors[] }` · 400/503/409/500. Needs `notion-wiki-token` in Vault | no |
**Obsidian export** (`adapters/obsidian.ts`) writes `{outDir}/_index.md` + `{outDir}/{entity|concept|synthesis}/{slug}.md`, rewriting `[[Display Name]]` → `[[slug|Display Name]]`; no LLM, no state mutation, idempotent. **Notion export** (`adapters/notion.ts`) creates child pages under a root page id (delta-tracked via `notion_page_id` + `content_hash` through `NotionStateHelpers`), converting Markdown to Notion blocks. None of the wiki routes stream.
---
## 19. The Paid-Tier Upgrade Trigger (`packages/shared/src/tiers.ts`)
`TierCapabilities.customSkills` and `connectorLimit` are the levers. Memory, Harvest, and the wiki remain free.
| Tier | `customSkills` | `connectorLimit` | `teamSkillLibrary` | Marketplace install/publish |
|---|---|---|---|---|
| TRIAL | true | -1 (unlimited) | true | allowed (all unlocked 15 days) |
| FREE | **false** | 5 | false | blocked (built-in skills only) |
| PRO | true | -1 | false | allowed (`requireTier('PRO')`) |
| TEAMS | true | -1 | true | allowed |
| ENTERPRISE | true | -1 | true | allowed + enterprise packs |
The marketplace routes enforce this directly: `POST /api/marketplace/install` and `POST /api/marketplace/publish` carry `preHandler: [requireTier('PRO')]`; `GET /api/marketplace/enterprise-packs` carries `requireTier('ENTERPRISE')`. Header copy in `tiers.ts`: *"Skills/connectors are the upgrade trigger."*
---
## 20. Frontend Integration Cheat-Sheet
- **Install Center / Skills UI** → drive from `GET /api/skills/starter-pack/catalog` (state per skill) + `GET /api/skills/capability-packs/catalog`; install via the `POST .../:id` variants; live skills via `GET /api/skills`; preview-without-install via `POST /api/skills/test`; in-context suggestions via `GET /api/skills/suggestions?context=`.
- **Marketplace browser** → `GET /api/marketplace/search` returns packages already annotated with `installed` + `scanStatus` + `scanScore` and the full category list; faceted filters map 1:1 to query params; `GET /api/marketplace/categories` and `/sources` populate filter chips. Show an upgrade modal on `403`/tier errors from `/install`.
- **Security UI** → `scanStatus` per package + `GET /api/marketplace/security-status` for the global banner; `POST /api/marketplace/security-check` for an on-demand scan; findings carry `severity`/`category`/`title`/`description`/`location`.
- **Wiki app** → list with `GET /api/wiki/pages`, body with `/pages/:slug/content`, build with `POST /api/wiki/compile`, quality dashboard with `GET /api/wiki/health` (handle the `503 no_real_embedder` state with an "add an embedding key" prompt), freshness with `GET /api/wiki/watermark`, and the two export buttons.