Files
waggle-os/docs/redesign-warm-hive/pr4-recon/02-marketplace-backend.md
Oleg Maslov 0c3e2ead3b
Some checks failed
Installer Smoke / installer-smoke (push) Has been cancelled
moving
2026-09-02 10:10:29 +02:00

184 lines
23 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# PR4 Recon — Slice 2: Marketplace Backend + Routes
**Design contract:** SCREENS.md §09 (Marketplace — skills + connectors + MCP as one shelf, agent-searchable).
**PR4 scope (BUILD-PLAN.md §6, line 142):** "Marketplace + **shared install store ('sync')** (grid + agent-pick + inline-in-chat)" → `MarketplaceApp`, **new install store**, screen 09.
**Branch:** `feature/warm-hive-pr4`. **Mode:** READ ONLY (no source touched).
This doc maps the *existing* backend territory a PR4 "shared install store (sync)" would build ON. It does not propose the plan.
---
## 0. TL;DR for the synthesizer
There is **no single install store today**. The §09 contract assumes *one* store powering grid + agent-pick + inline-card, with type-aware micro-states (skill Add→Adding→Added, connector Connect→Signing in→Connected token→vault, MCP Enable→Enabling→Enabled) and a workspace-scoped "N in this workspace" count bar. The actual backend splits install state across **three independent persistence layers**, none of which agree on a shared shape, and **none of which are workspace-scoped**:
| Kind | "Is it installed?" source of truth | Install verb / route | Workspace-scoped? |
|---|---|---|---|
| **skill** / **plugin** / **mcp** (marketplace pkg) | `installations` table in `marketplace.db` (`status='installed'`), keyed by **numeric `package_id`** | `POST /api/marketplace/install` | **No** — global |
| **mcp** (runtime) | `<dataDir>/.mcp.json` ⋈ live `McpRuntime` server states, keyed by **string server name** | `POST /api/mcps/install` (delegates to marketplace) | **Partial** — a server carries one optional `workspaceId` tag (C19), but install presence is global |
| **connector** | **Vault** credential under `connector:{id}` (presence = "connected"), keyed by **string connector id** | `POST /api/connectors/:id/connect` (token→vault) | **No** — vault is single-user/global |
The install-audit trail (`install_audit` table) **is the only thing that already unifies all three** as a write-side feed — but it is an append-only *log*, not a queryable "installed set," and it has **no `workspace_id` column** (schema.ts:135-152). So the count-bar's "N in this workspace" claim has **no backing data today**; it would either need a new scoping column or be redefined as a global count.
The current FE (`MarketplaceApp.tsx`) already does federate-at-read across 6 domains but holds install state only in **local React `useState`** — installing patches `extensions[]` in place; there is no cross-view store, so "installing in any view reflects in all" (the §09 CRITICAL line) is **not satisfied** today.
---
## 1. Marketplace DB schema (`packages/marketplace/src/db.ts` + types.ts)
`MarketplaceDB` wraps a `better-sqlite3` handle at `~/.waggle/marketplace.db` (overridable; the server opens `<dataDir>/marketplace.db`, seeded by copy from `packages/marketplace/marketplace.db` — server/local/index.ts:457-506). WAL mode, FK on.
### Tables (inferred from queries — there is **no `CREATE TABLE` DDL in the TS source**; the schema ships pre-built inside the committed `marketplace.db` seed file)
- **`packages`** — the catalog. Columns referenced in code (db.ts, types.ts `MarketplacePackage`): `id` (PK), `source_id` (FK→sources), `name`, `display_name`, `description`, `author`, `package_type` (`skill|plugin|mcp_server|template|pack`), **`waggle_install_type`** (`skill|plugin|mcp` — the dispatch discriminant), `waggle_install_path`, `version`, `license`, `repository_url`, `homepage_url`, `downloads`, `stars`, `rating`, `rating_count`, `category`, `subcategory`, `install_manifest` (JSON), `platforms` (JSON), `min_waggle_version`, `dependencies` (JSON), `packs` (JSON), `created_at`, `updated_at`. **Plus security columns** added by the installer's `recordScanResult()` (types.ts:81-90 `PackageSecurityColumns`): `security_status`, `security_score`, `last_scanned_at`, `content_hash`, `scan_engines`, `scan_findings`, `scan_blocked`.
- **`packages_fts`** — FTS5 virtual table joined on `p.id = fts.rowid` for full-text search (db.ts:103). `rank` column used for relevance ordering.
- **`sources`** — catalog provenance. Columns (db.ts:271-283, types.ts `MarketplaceSource`): `id`, `name`, `display_name`, `url`, `source_type`, `platform`, `total_packages`, `install_method`, `api_endpoint`, `description`, `last_synced_at`, **`is_custom`** (auto-migrated, db.ts:57-67), **`sync_state`** (auto-migrated JSON for resumable sync, db.ts:69-78).
- **`packs`** — capability bundles. Columns (types.ts `MarketplacePack`): `id`, `slug`, `display_name`, `description`, `target_roles`, `icon`, `priority` (`core|recommended|optional`), `connectors_needed` (JSON), `created_at`.
- **`pack_packages`** — pack↔package join with `is_core` flag (db.ts:191-194).
- **`installations`** — **THE install-state source of truth for marketplace packages** (db.ts:350-404). Columns (types.ts `Installation`): `id`, `package_id` (FK), `installed_version`, `installed_at`, `install_path`, `status` (`installed|updating|failed|uninstalled`), `config` (JSON — secret VALUES redacted to `[redacted]`, keys kept; installer.ts:164-173). **No `workspace_id`.**
- **`scan_history`** — append-only security scan log (installer.ts:644-659): `package_id`, `scanned_at`, `overall_severity`, `security_score`, `content_hash`, `engines_used`, `findings`, `blocked`, `scan_duration_ms`, `triggered_by`.
### Install-state API on `MarketplaceDB` (the methods a sync store would call)
- `recordInstallation(packageId, version, installPath, config)` → inserts `installations` row, `status='installed'` (db.ts:350).
- `isInstalled(packageId): boolean``SELECT 1 FROM installations WHERE package_id=? AND status='installed'` (db.ts:390). **This is the per-package installed check.**
- `markUninstalled(packageId)` → flips status to `'uninstalled'` (db.ts:400).
- `listInstallations(): InstalledPackageRow[]` → joins `installations``packages` for `status='installed'`, ordered by `installed_at DESC` (db.ts:376).
- `getInstalledCount(): number`**global** count of `status='installed'` rows (db.ts:500). This is the only existing "count bar" primitive — **not workspace-aware**.
- `getPackage(id)` / `getPackageByName(name)` — by numeric id / string name.
- `search(SearchOptions): SearchResult` — FTS5 + faceted filters; returns `{ packages, total, facets:{types,categories,sources}, installedCount }` (db.ts:86-165). `installedCount` here is the global `getInstalledCount()`.
---
## 2. List/search/install/uninstall APIs (`packages/server/src/local/routes/marketplace.ts`)
Registered as `marketplaceRoutes(fastify)` (server/local/index.ts:95). Every route gates on `fastify.marketplace` (the decorated `MarketplaceDB|null`, index.ts:506); `requireDb()` returns **503** if absent.
| Method · Route | Purpose | Tier gate | Notes |
|---|---|---|---|
| `GET /api/marketplace/search` | FTS5 + facet search of the catalog | none | Annotates each pkg with `installed: db.isInstalled(pkg.id)`, `scanStatus`, `scanScore`; appends `categories: PACKAGE_CATEGORIES` (marketplace.ts:57-118). **This is the grid's read.** |
| `GET /api/marketplace` (bare alias) | S21 alias → injects into `/search`; accepts 6-domain `type` facet; returns honest empty `{federated:true}` for non-marketplace domains | none | `extend.ts:49-85`. **This is what the current FE `adapter.getMarketplace()` calls.** |
| `GET /api/marketplace/packs` · `/packs/:slug` | List packs / pack detail | none | marketplace.ts:123-146 |
| `GET /api/marketplace/enterprise-packs` | KVARK-gated packs | `requireTier('ENTERPRISE')` | marketplace.ts:152 |
| **`POST /api/marketplace/install`** | **Install a package by numeric `packageId`** | `requireTier('PRO')` | marketplace.ts:181-432. SecurityGate pre-scan (heuristics-only) → severity gating (CRITICAL=403 always; HIGH=403 unless `force`; MEDIUM/LOW proceed) → `MarketplaceInstaller.install()` → audit rows. Returns `{...InstallResult, security:{...}}`, **200** on success / **422** on fail / **403** on block. |
| `POST /api/marketplace/uninstall` | Uninstall by `packageId` | none | marketplace.ts:437-470. For MCP packages ALSO tears down the live runtime + `<dataDir>/.mcp.json` (else it resurrects at boot). |
| `GET /api/marketplace/installed` | `listInstallations()` | none | marketplace.ts:475 |
| `POST /api/marketplace/security-check` | Scan-only by id | none | marketplace.ts:486 |
| `GET/POST/DELETE /api/marketplace/sources` | Source CRUD + sync-on-add | none | marketplace.ts:525-641 |
| `GET /api/marketplace/categories` | `PACKAGE_CATEGORIES` taxonomy (21 cats) | none | marketplace.ts:646 — feeds the category filter |
| `POST /api/marketplace/sync` | Manual sync from sources | none | marketplace.ts:655 |
| `GET /api/marketplace/security-status` | Cisco scanner availability + aggregate scan counts | none | marketplace.ts:712 |
| `POST /api/marketplace/publish` | Publish a local skill into the catalog | `requireTier('PRO')` | marketplace.ts:764 |
**Tier note for the §09 "one-click install":** `POST /api/marketplace/install` is **PRO-gated**. The skill "Add (instant)" micro-state in the contract collides with PRO-gating unless the install path for FREE-tier skills differs. The MCP path (`/api/mcps/install`) is also PRO. Connector connect (`/api/connectors/:id/connect`) is **ungated**.
---
## 3. Installer flow per type (`packages/marketplace/src/installer.ts`)
`MarketplaceInstaller.install(request: InstallRequest)` (installer.ts:81) is the **single dispatch entrypoint** for skill/plugin/mcp. Flow:
1. `db.getPackage(packageId)` → 404-shaped result if missing.
2. Idempotency: if `!force && db.isInstalled(pkg.id)` → returns success "already installed" (installer.ts:96).
3. **Security gate** (`SecurityGate.scan`) on resolved content → `recordScanResult()` → if `blocked && !forceInsecure` returns blocked result (installer.ts:116-130).
4. **Dispatch on `pkg.waggle_install_type`** (installer.ts:137):
- **`skill`** → `installSkill()` (installer.ts:309): writes `~/.waggle/skills/{name}.md` from `manifest.skill_content` / `skill_url` / repo `SKILL.md` / generated stub; then `PUT /api/skills/{name}` notify. **Instant** (matches §09 "Add→Added instant").
- **`plugin`** → `installPlugin()` (installer.ts:367): mkdir `~/.waggle/plugins/{name}/`, git-clone or `npm install`, write `plugin.json`, install bundled skills, update `registry.json`, run post-install hooks, `POST /api/plugins/install` notify. Slow / multi-step.
- **`mcp`** → `installMcp()` (installer.ts:489): optional `npm install -g`, apply settings to env, write the server entry into `<dataDir>/.mcp.json` (`mcpConfigPath()` = `WAGGLE_DATA_DIR/.mcp.json`, installer.ts:47). **Does NOT start the runtime** — that is the `/api/mcps/install` route's job (mcps.ts:277-294). (§09 "Enable→Enabling→Enabled" = install + runtime start.)
5. On success: `db.recordInstallation()` (with redacted setting keys) and attaches `scanResult`.
`uninstall(packageId)` (installer.ts:256) mirror-dispatches: `uninstallSkill` (rm file + DELETE notify), `uninstallPlugin` (rmdir + registry + DELETE), `uninstallMcp` (remove from `.mcp.json`), then `db.markUninstalled()`.
**Notify pattern:** the installer best-effort POSTs to `API_BASE` (`WAGGLE_API_URL` || `http://localhost:3000`, installer.ts:52) — fire-and-forget, swallows failure (installer.ts:764). Note the default port 3000 vs the running dev sidecar; relevant if a sync store relied on the notify hook firing.
---
## 4. The three install-state stores in detail (the "sync" problem)
The §09 CRITICAL line — *"one store powers the grid, the agent picks, AND the inline card … installing in any view reflects in all"* — has no backend equivalent. The three stores:
### 4a. Marketplace packages (skill/plugin/mcp) — `installations` table
- Truth: `MarketplaceDB.isInstalled(packageId: number)`. Keyed by **numeric package id**.
- Read surfaces: `/api/marketplace/search` (annotates `installed`), `/api/marketplace/installed`.
### 4b. MCP runtime — `<dataDir>/.mcp.json` ⋈ `McpRuntime` (`routes/mcps.ts`)
- Truth: `GET /api/mcps` (mcps.ts:143) = `MCP_CATALOG` (from `@waggle/shared`) ⋈ persisted `.mcp.json` entries ⋈ live `runtime.getServerStates()`. Keyed by **string server name**. `installed = name ∈ (persisted runtime)`.
- Statuses: `installed | running | error | stopped` (mcps.ts:67 `toInstanceStatus`).
- `POST /api/mcps/install` (mcps.ts:189) **delegates to `/api/marketplace/install`** via `fastify.inject`, then registers + `start()`s the runtime server (8s budget). So an MCP "install" touches BOTH 4a and 4b. **This is the closest thing to a working cross-store sync** — but it is MCP-specific and one-directional.
- **Workspace tag:** `PATCH /api/mcps/:id/permissions` sets a single `workspaceId` (C19 — "single-workspace scoping v1", mcps.ts:535). This is the ONLY place a "workspace" appears in install state, and it is a *scope filter on tool exposure*, not an install-presence scoping.
### 4c. Connectors — Vault (`routes/connectors.ts`)
- Truth: presence of a vault credential under `connector:{id}` (`fastify.vault.getConnectorCredential(id)`). "Connected" = credential exists & not expired.
- Read: `GET /api/connectors``connectorRegistry.getDefinitions()` (a STATIC catalog of all connectors) + `GET /api/connectors/:id/health` for live `connected|disconnected|expired|error` status.
- Install verb: `POST /api/connectors/:id/connect` — stores token→vault (matches §09 "token → vault"), re-inits the connector, writes audit. **Ungated.** Disconnect/revoke purge vault + OAuth tokens.
- **Connectors are NOT in `marketplace.db` at all.** They never appear in `installations`/`isInstalled`. The §09 grid showing connectors alongside skills/MCP must federate from `/api/connectors`.
### 4d. The unifying write-side feed — `install_audit` (the one cross-type thing that exists)
- `fastify.auditStore` (`@waggle/core` `install-audit.ts`) writes to table `install_audit` (schema.ts:135-152). All three stores already write here on install/uninstall/connect/revoke (marketplace.ts, mcps.ts `recordMcpAudit`, connectors.ts `recordConnectorAudit`).
- **Read:** `GET /api/extend/audit` (extend.ts:88) — ONE shared feed across `native|skill|plugin|mcp|connector|marketplace` with `?type=` / `?capability=` filters. The `AuditStore` API: `getRecent(limit)`, `getRecentByType(type, limit)`, `getByCapability(name)`.
- **CRITICAL GAP for the count bar:** `install_audit` has **NO `workspace_id` column** (schema.ts:135-152 — confirmed; `ai_interactions`/`execution_traces` DO have it at schema.ts:174/219, install_audit does not). It is an append-only event log, not a "current installed set." A "N in this workspace" count therefore has **zero backing data** today.
---
## 5. The install-audit trail (what's already auditable)
`RecordAuditInput` fields (used by all three route files): `capabilityName`, `capabilityType` (`native|skill|plugin|mcp|connector|marketplace`), `source`, `riskLevel` (`low|medium|high|critical`), `trustSource` (`builtin|starter_pack|local_user|third_party_verified|third_party_unverified|unknown|security-gate`), `approvalClass` (`standard|elevated|critical|blocked`), `action` (`proposed|approved|installed|rejected|failed|blocked|uninstalled`), `initiator` (`agent|user|system`), `detail`. CHECK constraints in schema.ts:143-150 enforce these — **adding a value (e.g. a `synced` action or a `workspace_id`) is a migration + CHECK change**, and the file warns these drifted once and crashed `acquire_capability` (schema.ts:139-142). Treat the audit vocabulary as a contract.
The marketplace install route writes audit rows for CRITICAL-block / HIGH-block / HIGH-force-override / MEDIUM / LOW / blocked-by-installer / forceInsecure-override (marketplace.ts:222-415) — a thorough trust trail, all global.
---
## 6. Exact integration points a PR4 "sync store" would hook into
**Reads (catalog + installed-state):**
- `GET /api/marketplace/search` (grid; annotates `installed` per pkg) — or the bare `GET /api/marketplace?type=` alias the FE already uses.
- `GET /api/marketplace/installed``MarketplaceDB.listInstallations()`.
- `GET /api/mcps` (catalog ⋈ persisted ⋈ runtime; `installed` + `status`).
- `GET /api/connectors` + `GET /api/connectors/:id/health` (connector connected-state).
- `GET /api/marketplace/categories` (the All/Skills/Connectors/MCP filter taxonomy).
- `GET /api/extend/audit` (the existing cross-type unified feed — the natural read for a "what's installed across everything" view, modulo it being a log).
**Install verbs (the three one-click flows):**
- skill / plugin → `POST /api/marketplace/install` `{ packageId }` (PRO). FE adapter: `adapter.installMarketplacePackage(packageId)``/api/marketplace/install` (adapter.ts:1339).
- mcp → `POST /api/mcps/install` `{ mcpId, settings, force }` (PRO) — delegates to marketplace install + runtime start; uninstall via `POST /api/mcps/:id/revoke`.
- connector → `POST /api/connectors/:id/connect` `{ token|apiKey }` (token→vault); disconnect `POST /api/connectors/:id/disconnect`, strong revoke `POST /api/connectors/:id/revoke`.
**Installer functions (package layer, if the store goes below the routes):**
- `MarketplaceInstaller.install(InstallRequest)` / `.uninstall(packageId)` / `.installPack(slug)` (installer.ts:81/256/210).
- `MarketplaceDB.isInstalled(id)` / `recordInstallation(...)` / `markUninstalled(id)` / `getInstalledCount()` / `listInstallations()` (db.ts).
**Server decoration:** `fastify.marketplace: MarketplaceDB | null` (index.ts:506); `fastify.connectorRegistry`, `fastify.vault`, `fastify.auditStore`, `fastify.agentState.mcpRuntime` are the sibling singletons a unified store would coordinate.
**Current FE consumer to refactor:** `apps/web/src/components/os/apps/MarketplaceApp.tsx` (417 LOC) — already federates 6 domains at read (skill via `getMarketplace`, packs, mcp via `getMcps`+`getMarketplace`, persona, connector via `getConnectors`, model, template) but holds install state in **local `useState extensions[]`**; install just patches the row (`installed: true`) — **no shared store, no cross-view propagation.** View-model federation already lives in `apps/web/src/lib/extension-catalog.ts`.
---
## 7. Key deltas vs the §09 contract (territory, not plan)
1. **No single install store.** Three stores (marketplace `installations` / `.mcp.json`+runtime / vault), three key types (numeric id / string name / string id), three install verbs, three tiers (PRO / PRO / ungated). The §09 "one store … reflects in all" is the central build.
2. **No workspace scoping of install state.** `installations` and `install_audit` have no `workspace_id`; vault is global; only MCP carries a single optional `workspaceId` *tool-scope* tag. The "N in this workspace" count bar has no data source — needs either a new scoping dimension or redefinition to a global count.
3. **No agent-suggestion endpoint.** The §09 "agent-suggestion box (connector + skill + tool, each with a why)" has no backing route. FTS search (`/api/marketplace/search`) + the agent's `acquire_capability` path (the verbose-`need` FTS handling in db.ts:91-99 / `toFtsMatchQuery`) is the nearest substrate, but a "recommend one of each kind with a reason" composite does not exist.
4. **No inline-in-chat picker route.** Variation B (mid-conversation connector offer with vault-aware approval) reuses the same install verbs but has no dedicated surfacing/approval endpoint; the connector connect flow + the existing approval/confirmation machinery (`routes/approval.ts`, agent `confirmation.ts`) are the substrate.
5. **Connectors absent from `marketplace.db`.** Any unified grid must federate connectors from `/api/connectors`, not from the marketplace catalog — they share no row shape with `MarketplacePackage`.
6. **Audit feed is a log, not a set.** `GET /api/extend/audit` unifies *events* across types but cannot answer "what is currently installed in workspace W" without scanning + reducing; the count bar wants a live set.
---
## 8. Risks / sharp edges for a builder
- **Schema DDL is invisible.** The `packages`/`sources`/`packs`/`installations`/`scan_history` tables exist only inside the committed `marketplace.db` seed binary — there is no `CREATE TABLE` in TS. Adding a `workspace_id` to `installations` means a runtime `ALTER TABLE` migration in `MarketplaceDB.migrateSchema()` (the established pattern, db.ts:57-78), not an edit to a schema file.
- **`install_audit` CHECK constraints are a hard contract.** Adding a `synced` action or any new vocabulary needs a coordinated `runMigrations()` + CHECK rewrite; schema.ts explicitly records a prior drift that crashed `acquire_capability`.
- **PRO-gating vs "instant Add".** Skill install is PRO-gated server-side; the §09 instant micro-state must reconcile with `requireTier('PRO')` (or rely on a different non-gated skill path).
- **MCP install is the only working cross-store coupling** (`/api/mcps/install` writes both `installations` and `.mcp.json`+runtime, and revoke keeps marketplace `installed` honest, mcps.ts:509-519). A unified store should mirror this bidirectional bookkeeping for the other types, or it will drift (the FE will show "installed" after a revoke, etc.).
- **OSS-sync constraint (§7.5):** `install_audit` DDL inside `mind/schema.ts` is OSS-excluded (Waggle governance). A `workspace_id` addition there has nowhere to land on the public mirror — fine for the monorepo, but flag it as a curated-strip item.
- **Three-store consistency under failure.** The installer's `notify` hook is fire-and-forget to `localhost:3000` (not the running sidecar port); a sync store cannot rely on the notify callback to invalidate caches.
---
## 9. Open questions for the founder/lead
1. **Workspace scoping:** Is "N in this workspace" a real per-workspace install set (requires a new `workspace_id` on installs + per-workspace activation model), or a relabeled global count for v1? This is the single biggest decision — it determines whether the sync store needs a new data dimension across all three backends.
2. **Skill install tier:** Should the §09 instant skill "Add" stay PRO-gated, or is there a FREE skill-install lane? (CLAUDE.md moat: skills/connectors are the upgrade trigger — so PRO-gating may be intentional and the "Add" CTA should show the upgrade nudge instead of installing.)
3. **Agent-suggestion box:** Build a new composite recommend endpoint (one connector + one skill + one tool + "why"), or compose it client-side from three `search` calls + an LLM rationale? No backing route exists either way.
4. **Sync store location:** A FE-only store (React context / Zustand over the existing routes) vs a new server-side unified `/api/install-state` read that reduces all three stores into one shape. The §09 "one store reflects in all" is achievable purely client-side IF every view subscribes to it; a server read is only needed for the count bar's correctness across reloads.
5. **Inline-in-chat (Variation B):** Reuse `/api/connectors/:id/connect` + existing approval machinery, or a dedicated in-chat install/approval contract? Decide before building the chat surface.