11 KiB
11 KiB
Waggle OS — Backend Map
This is the complete, source-grounded map of the Waggle OS backend, written so the frontend can be
rebuilt from scratch (e.g. in Lovable) against the existing, unchanged Fastify sidecar. The
mental model is simple and worth internalizing before anything else: Waggle is a single-page
"desktop OS" (window manager + dock + ~18 apps, opened by appId, not by URL) that is a thin
client — it renders state, streams responses over SSE, and POSTs user intent through one shared
adapter singleton to a local Node.js Fastify sidecar (http://127.0.0.1:3333). The sidecar is
the brain stem: it resolves the workspace session, builds the layered system prompt, runs the agent
tool-calling loop, executes tools through a governance middleware chain, and streams results back.
Two physically separate data stores sit behind it — per-workspace SQLite *.mind files (private
memory, knowledge graph, identity, awareness; the moat) and an optional team/cloud Postgres layer
(users, teams, tasks, governance). Everything is gated on two orthogonal axes: subscription tier
(does the feature exist?) and trust/autonomy (does this action need an approval click?). The
backend is the contract; the frontend is replaceable.
Recommended Reading Order
Read these in sequence for a clean ramp from concept → contracts → build playbook:
- 00-MENTAL-MODEL.md — the concept map and six-layer request lifecycle. Read this first; keep it in your head while reading everything else.
- DIAGRAMS/01-system-architecture.md — the cross-cutting runtime stack: shell → SPA → sidecar → packages → data stores → LLM router.
- DIAGRAMS/02-master-er.md — the two ER diagrams (SQLite memory layer + Postgres relational layer) and how they bridge.
- sections/03a-api-chat-agents.md … sections/03g-api-cloud-billing-kvark.md — the full HTTP/SSE/WS API contract, split by domain (start at 03a, the conversational core).
- sections/04-feature-map.md — the canonical app ↔ endpoint matrix: every screen and exactly which routes it calls.
- sections/05a-subsystem-agent-runtime.md … sections/05g-subsystem-skills-marketplace-wiki.md — the subsystem deep-dives explaining what the API actually does internally.
- 07-FRONTEND-REBUILD-GUIDE.md — the action-oriented, build-in-order playbook for the Lovable rebuild.
Full File Index
Top-level
| File | Description |
|---|---|
| 00-MENTAL-MODEL.md | The concept map: what Waggle OS is, the six-layer request lifecycle, and the two or three ideas everything hangs off. |
| 07-FRONTEND-REBUILD-GUIDE.md | Action-oriented Lovable rebuild playbook — build the adapter + ServiceProvider + Desktop shell first, then screens that call adapter methods. |
| AUDIT.md | Completeness audit cross-checking the written sections against ground truth (route files, schema tables, connectors); ~94% coverage with named gaps. |
| WAGGLE-BACKEND-VISUAL.html | Self-contained dark-themed visual one-pager of the whole backend map (Hive DS palette) — open in a browser for the diagrams. |
Diagrams (DIAGRAMS/)
| File | Description |
|---|---|
| DIAGRAMS/01-system-architecture.md | Full runtime stack as a layered flowchart: Tauri shell → React SPA → Fastify sidecar (:3333) → packages → SQLite/Postgres/Redis → LiteLLM → LLM providers. |
| DIAGRAMS/02-master-er.md | Two ER diagrams — per-workspace SQLite memory (*.mind, 14+2 tables) and team/cloud Postgres (20 tables) — sharing no FKs except users.mind_path. |
| DIAGRAMS/03-chat-turn-sequence.md | End-to-end sequence of one chat turn: POST /api/chat → prompt assembly → tool loop → 11-step tool middleware → SSE stream → cognify write-back. |
| DIAGRAMS/04-feature-api-map.md | Rebuild-contract flowchart mapping every OS app/overlay to the backend endpoint groups (03a–03g) it depends on. |
| DIAGRAMS/05-tier-gating.md | The two orthogonal gating axes — subscription tier (does the door exist?) vs trust/autonomy (does it need a key turn?), plus getEffectiveTier(). |
| DIAGRAMS/06-api-domains.md | Mind-map index of all 7 API domains with endpoint counts; local sidecar (:3333) vs cloud server (:3100), KVARK via in-process client only. |
Sections — Data Model (sections/02*)
| File | Description |
|---|---|
| sections/02a-data-model-memory.md | Per-workspace SQLite *.mind schema — frames, knowledge graph, identity, awareness, sessions; the canonical shape of everything the memory APIs return. |
| sections/02b-data-model-relational.md | Team/cloud Postgres + Drizzle schema (20 tables): users, teams, agents, tasks, jobs, governance — column names/types/FKs verbatim from source. |
| sections/02c-shared-types-tiers.md | The wire contract: @waggle/shared interfaces, enums, Zod request schemas, and the 5-tier capability matrix consumed by both sidecar and web. |
Sections — API Contracts (sections/03*)
| File | Description |
|---|---|
| sections/03a-api-chat-agents.md | Conversational core: POST /api/chat SSE stream, the event catalogue, mid-stream tool approvals, sessions CRUD/export, and slash commands. |
| sections/03b-api-memory.md | Memory subsystem API: recall/save frames, knowledge graph, wiki compile/read, harvest external AI exports, import, identity, documents, GDPR erasure. |
| sections/03c-api-workspace-team.md | Management plane: workspaces + templates, teams/members, personas, settings (models/budgets/autonomy/tier), user profile, and message pins. |
| sections/03d-api-marketplace-skills.md | Capability layer API: marketplace, skills, connectors, tools, OAuth, vault, and providers. |
| sections/03e-api-evolution-governance.md | "Self-improves + you stay in control": evolution runs, feedback, telemetry, EU AI Act compliance (PDF), cost tracking, capability status, approvals inbox. |
| sections/03f-api-realtime-ops.md | Real-time + ops: the 4 SSE streams (WaggleDance v1/v2, events, notifications), cron, offline queue, backup/restore, fleet, LLM proxies, browse, Telegram. |
| sections/03g-api-cloud-billing-kvark.md | The two HTTP surfaces, API-root discovery + auth/origin guards, Stripe billing flow (checkout → webhook → tier), KVARK client, and WebSocket channels. |
Sections — Feature Map (sections/04)
| File | Description |
|---|---|
| sections/04-feature-map.md | Canonical app↔endpoint matrix: every OS app, overlay, and page, the shared adapter/providers, and the exact backend endpoints each feature calls. |
Sections — Subsystem Deep-Dives (sections/05*)
| File | Description |
|---|---|
| sections/05a-subsystem-agent-runtime.md | The agent runtime: layered system-prompt assembly, the tool-calling loop, the tool middleware chain, and completion-time verification/file-write/skill gates. |
| sections/05b-subsystem-memory.md | The persistent memory engine (the moat): Identity → Awareness → Frames → Knowledge Graph, hybrid search, cognify write path, combined retrieval. |
| sections/05c-subsystem-harvest.md | Harvest ingestion pipeline: external AI exports + files/URLs → normalized UniversalImportItem → memory frames; powers the Memory app HarvestTab. |
| sections/05d-subsystem-evolution.md | Self-evolution loop: mine execution history → GEPA + EvolveSchema mutation → LLM-judge → safety gates → auditable EvolutionRun → user accept/reject. |
| sections/05e-subsystem-waggledance-aios.md | WaggleDance multi-agent coordination + the AI-OS arc: detect 7 external AI tools, install reversible hooks, launch with workspace context, stream signals back. |
| sections/05f-subsystem-capabilities-tiers.md | What an agent can/may do: capability discovery + acquisition, connectors-as-tools, MCP catalog, and the trust vs tier gates (incl. KVARK gating). |
| sections/05g-subsystem-skills-marketplace-wiki.md | Skills lifecycle, the marketplace catalog/installer/security gate, and the wiki-compiler — the PRO+ upgrade trigger (Memory/Harvest/wiki stay free). |
How to Feed This Into Lovable
For the rebuild, paste the contract-bearing docs directly into Lovable and keep the visual open on the side:
- Paste these as context (the binding contracts the new frontend must honor):
- 00-MENTAL-MODEL.md — so the model holds the desktop-OS / thin-client mental model.
- sections/02c-shared-types-tiers.md — the exact wire types, Zod request schemas, and tier capability matrix.
- sections/03a-api-chat-agents.md through sections/03g-api-cloud-billing-kvark.md — the full endpoint/SSE contract (paste the 03* sections relevant to the screens you're building).
- 07-FRONTEND-REBUILD-GUIDE.md — the build-in-order playbook (adapter + ServiceProvider + Desktop shell first, then screens).
- Open WAGGLE-BACKEND-VISUAL.html in a browser for the diagrams — it renders the architecture, ER, chat-turn, feature-API, and tier-gating views as one navigable one-pager, which is faster to skim than the raw Mermaid blocks.
Start by having Lovable build the adapter singleton + ServiceProvider + Desktop shell against
the sidecar at http://127.0.0.1:3333; everything else is screens that call adapter methods listed
in sections/04-feature-map.md.