moving

docs/redesign-warm-hive/pr4-recon/02-marketplace-backend.md

@@ -0,0 +1,183 @@
+# PR4 Recon — Slice 2: Marketplace Backend + Routes
+
+**Design contract:** SCREENS.md §09 (Marketplace — skills + connectors + MCP as one shelf, agent-searchable).
+**PR4 scope (BUILD-PLAN.md §6, line 142):** "Marketplace + **shared install store ('sync')** (grid + agent-pick + inline-in-chat)" → `MarketplaceApp`, **new install store**, screen 09.
+**Branch:** `feature/warm-hive-pr4`. **Mode:** READ ONLY (no source touched).
+
+This doc maps the *existing* backend territory a PR4 "shared install store (sync)" would build ON. It does not propose the plan.
+
+---
+
+## 0. TL;DR for the synthesizer
+
+There is **no single install store today**. The §09 contract assumes *one* store powering grid + agent-pick + inline-card, with type-aware micro-states (skill Add→Adding→Added, connector Connect→Signing in→Connected token→vault, MCP Enable→Enabling→Enabled) and a workspace-scoped "N in this workspace" count bar. The actual backend splits install state across **three independent persistence layers**, none of which agree on a shared shape, and **none of which are workspace-scoped**:
+
+| Kind | "Is it installed?" source of truth | Install verb / route | Workspace-scoped? |
+|---|---|---|---|
+| **skill** / **plugin** / **mcp** (marketplace pkg) | `installations` table in `marketplace.db` (`status='installed'`), keyed by **numeric `package_id`** | `POST /api/marketplace/install` | **No** — global |
+| **mcp** (runtime) | `<dataDir>/.mcp.json` ⋈ live `McpRuntime` server states, keyed by **string server name** | `POST /api/mcps/install` (delegates to marketplace) | **Partial** — a server carries one optional `workspaceId` tag (C19), but install presence is global |
+| **connector** | **Vault** credential under `connector:{id}` (presence = "connected"), keyed by **string connector id** | `POST /api/connectors/:id/connect` (token→vault) | **No** — vault is single-user/global |
+
+The install-audit trail (`install_audit` table) **is the only thing that already unifies all three** as a write-side feed — but it is an append-only *log*, not a queryable "installed set," and it has **no `workspace_id` column** (schema.ts:135-152). So the count-bar's "N in this workspace" claim has **no backing data today**; it would either need a new scoping column or be redefined as a global count.
+
+The current FE (`MarketplaceApp.tsx`) already does federate-at-read across 6 domains but holds install state only in **local React `useState`** — installing patches `extensions[]` in place; there is no cross-view store, so "installing in any view reflects in all" (the §09 CRITICAL line) is **not satisfied** today.
+
+---
+
+## 1. Marketplace DB schema (`packages/marketplace/src/db.ts` + types.ts)
+
+`MarketplaceDB` wraps a `better-sqlite3` handle at `~/.waggle/marketplace.db` (overridable; the server opens `<dataDir>/marketplace.db`, seeded by copy from `packages/marketplace/marketplace.db` — server/local/index.ts:457-506). WAL mode, FK on.
+
+### Tables (inferred from queries — there is **no `CREATE TABLE` DDL in the TS source**; the schema ships pre-built inside the committed `marketplace.db` seed file)
+
+- **`packages`** — the catalog. Columns referenced in code (db.ts, types.ts `MarketplacePackage`): `id` (PK), `source_id` (FK→sources), `name`, `display_name`, `description`, `author`, `package_type` (`skill|plugin|mcp_server|template|pack`), **`waggle_install_type`** (`skill|plugin|mcp` — the dispatch discriminant), `waggle_install_path`, `version`, `license`, `repository_url`, `homepage_url`, `downloads`, `stars`, `rating`, `rating_count`, `category`, `subcategory`, `install_manifest` (JSON), `platforms` (JSON), `min_waggle_version`, `dependencies` (JSON), `packs` (JSON), `created_at`, `updated_at`. **Plus security columns** added by the installer's `recordScanResult()` (types.ts:81-90 `PackageSecurityColumns`): `security_status`, `security_score`, `last_scanned_at`, `content_hash`, `scan_engines`, `scan_findings`, `scan_blocked`.
+- **`packages_fts`** — FTS5 virtual table joined on `p.id = fts.rowid` for full-text search (db.ts:103). `rank` column used for relevance ordering.
+- **`sources`** — catalog provenance. Columns (db.ts:271-283, types.ts `MarketplaceSource`): `id`, `name`, `display_name`, `url`, `source_type`, `platform`, `total_packages`, `install_method`, `api_endpoint`, `description`, `last_synced_at`, **`is_custom`** (auto-migrated, db.ts:57-67), **`sync_state`** (auto-migrated JSON for resumable sync, db.ts:69-78).
+- **`packs`** — capability bundles. Columns (types.ts `MarketplacePack`): `id`, `slug`, `display_name`, `description`, `target_roles`, `icon`, `priority` (`core|recommended|optional`), `connectors_needed` (JSON), `created_at`.
+- **`pack_packages`** — pack↔package join with `is_core` flag (db.ts:191-194).
+- **`installations`** — **THE install-state source of truth for marketplace packages** (db.ts:350-404). Columns (types.ts `Installation`): `id`, `package_id` (FK), `installed_version`, `installed_at`, `install_path`, `status` (`installed|updating|failed|uninstalled`), `config` (JSON — secret VALUES redacted to `[redacted]`, keys kept; installer.ts:164-173). **No `workspace_id`.**
+- **`scan_history`** — append-only security scan log (installer.ts:644-659): `package_id`, `scanned_at`, `overall_severity`, `security_score`, `content_hash`, `engines_used`, `findings`, `blocked`, `scan_duration_ms`, `triggered_by`.
+
+### Install-state API on `MarketplaceDB` (the methods a sync store would call)
+
+- `recordInstallation(packageId, version, installPath, config)` → inserts `installations` row, `status='installed'` (db.ts:350).
+- `isInstalled(packageId): boolean` → `SELECT 1 FROM installations WHERE package_id=? AND status='installed'` (db.ts:390). **This is the per-package installed check.**
+- `markUninstalled(packageId)` → flips status to `'uninstalled'` (db.ts:400).
+- `listInstallations(): InstalledPackageRow[]` → joins `installations` ⋈ `packages` for `status='installed'`, ordered by `installed_at DESC` (db.ts:376).
+- `getInstalledCount(): number` → **global** count of `status='installed'` rows (db.ts:500). This is the only existing "count bar" primitive — **not workspace-aware**.
+- `getPackage(id)` / `getPackageByName(name)` — by numeric id / string name.
+- `search(SearchOptions): SearchResult` — FTS5 + faceted filters; returns `{ packages, total, facets:{types,categories,sources}, installedCount }` (db.ts:86-165). `installedCount` here is the global `getInstalledCount()`.
+
+---
+
+## 2. List/search/install/uninstall APIs (`packages/server/src/local/routes/marketplace.ts`)
+
+Registered as `marketplaceRoutes(fastify)` (server/local/index.ts:95). Every route gates on `fastify.marketplace` (the decorated `MarketplaceDB|null`, index.ts:506); `requireDb()` returns **503** if absent.
+
+| Method · Route | Purpose | Tier gate | Notes |
+|---|---|---|---|
+| `GET /api/marketplace/search` | FTS5 + facet search of the catalog | none | Annotates each pkg with `installed: db.isInstalled(pkg.id)`, `scanStatus`, `scanScore`; appends `categories: PACKAGE_CATEGORIES` (marketplace.ts:57-118). **This is the grid's read.** |
+| `GET /api/marketplace` (bare alias) | S21 alias → injects into `/search`; accepts 6-domain `type` facet; returns honest empty `{federated:true}` for non-marketplace domains | none | `extend.ts:49-85`. **This is what the current FE `adapter.getMarketplace()` calls.** |
+| `GET /api/marketplace/packs` · `/packs/:slug` | List packs / pack detail | none | marketplace.ts:123-146 |
+| `GET /api/marketplace/enterprise-packs` | KVARK-gated packs | `requireTier('ENTERPRISE')` | marketplace.ts:152 |
+| **`POST /api/marketplace/install`** | **Install a package by numeric `packageId`** | `requireTier('PRO')` | marketplace.ts:181-432. SecurityGate pre-scan (heuristics-only) → severity gating (CRITICAL=403 always; HIGH=403 unless `force`; MEDIUM/LOW proceed) → `MarketplaceInstaller.install()` → audit rows. Returns `{...InstallResult, security:{...}}`, **200** on success / **422** on fail / **403** on block. |
+| `POST /api/marketplace/uninstall` | Uninstall by `packageId` | none | marketplace.ts:437-470. For MCP packages ALSO tears down the live runtime + `<dataDir>/.mcp.json` (else it resurrects at boot). |
+| `GET /api/marketplace/installed` | `listInstallations()` | none | marketplace.ts:475 |
+| `POST /api/marketplace/security-check` | Scan-only by id | none | marketplace.ts:486 |
+| `GET/POST/DELETE /api/marketplace/sources` | Source CRUD + sync-on-add | none | marketplace.ts:525-641 |
+| `GET /api/marketplace/categories` | `PACKAGE_CATEGORIES` taxonomy (21 cats) | none | marketplace.ts:646 — feeds the category filter |
+| `POST /api/marketplace/sync` | Manual sync from sources | none | marketplace.ts:655 |
+| `GET /api/marketplace/security-status` | Cisco scanner availability + aggregate scan counts | none | marketplace.ts:712 |
+| `POST /api/marketplace/publish` | Publish a local skill into the catalog | `requireTier('PRO')` | marketplace.ts:764 |
+
+**Tier note for the §09 "one-click install":** `POST /api/marketplace/install` is **PRO-gated**. The skill "Add (instant)" micro-state in the contract collides with PRO-gating unless the install path for FREE-tier skills differs. The MCP path (`/api/mcps/install`) is also PRO. Connector connect (`/api/connectors/:id/connect`) is **ungated**.
+
+---
+
+## 3. Installer flow per type (`packages/marketplace/src/installer.ts`)
+
+`MarketplaceInstaller.install(request: InstallRequest)` (installer.ts:81) is the **single dispatch entrypoint** for skill/plugin/mcp. Flow:
+1. `db.getPackage(packageId)` → 404-shaped result if missing.
+2. Idempotency: if `!force && db.isInstalled(pkg.id)` → returns success "already installed" (installer.ts:96).
+3. **Security gate** (`SecurityGate.scan`) on resolved content → `recordScanResult()` → if `blocked && !forceInsecure` returns blocked result (installer.ts:116-130).
+4. **Dispatch on `pkg.waggle_install_type`** (installer.ts:137):
+   - **`skill`** → `installSkill()` (installer.ts:309): writes `~/.waggle/skills/{name}.md` from `manifest.skill_content` / `skill_url` / repo `SKILL.md` / generated stub; then `PUT /api/skills/{name}` notify. **Instant** (matches §09 "Add→Added instant").
+   - **`plugin`** → `installPlugin()` (installer.ts:367): mkdir `~/.waggle/plugins/{name}/`, git-clone or `npm install`, write `plugin.json`, install bundled skills, update `registry.json`, run post-install hooks, `POST /api/plugins/install` notify. Slow / multi-step.
+   - **`mcp`** → `installMcp()` (installer.ts:489): optional `npm install -g`, apply settings to env, write the server entry into `<dataDir>/.mcp.json` (`mcpConfigPath()` = `WAGGLE_DATA_DIR/.mcp.json`, installer.ts:47). **Does NOT start the runtime** — that is the `/api/mcps/install` route's job (mcps.ts:277-294). (§09 "Enable→Enabling→Enabled" = install + runtime start.)
+5. On success: `db.recordInstallation()` (with redacted setting keys) and attaches `scanResult`.
+
+`uninstall(packageId)` (installer.ts:256) mirror-dispatches: `uninstallSkill` (rm file + DELETE notify), `uninstallPlugin` (rmdir + registry + DELETE), `uninstallMcp` (remove from `.mcp.json`), then `db.markUninstalled()`.
+
+**Notify pattern:** the installer best-effort POSTs to `API_BASE` (`WAGGLE_API_URL` || `http://localhost:3000`, installer.ts:52) — fire-and-forget, swallows failure (installer.ts:764). Note the default port 3000 vs the running dev sidecar; relevant if a sync store relied on the notify hook firing.
+
+---
+
+## 4. The three install-state stores in detail (the "sync" problem)
+
+The §09 CRITICAL line — *"one store powers the grid, the agent picks, AND the inline card … installing in any view reflects in all"* — has no backend equivalent. The three stores:
+
+### 4a. Marketplace packages (skill/plugin/mcp) — `installations` table
+- Truth: `MarketplaceDB.isInstalled(packageId: number)`. Keyed by **numeric package id**.
+- Read surfaces: `/api/marketplace/search` (annotates `installed`), `/api/marketplace/installed`.
+
+### 4b. MCP runtime — `<dataDir>/.mcp.json` ⋈ `McpRuntime` (`routes/mcps.ts`)
+- Truth: `GET /api/mcps` (mcps.ts:143) = `MCP_CATALOG` (from `@waggle/shared`) ⋈ persisted `.mcp.json` entries ⋈ live `runtime.getServerStates()`. Keyed by **string server name**. `installed = name ∈ (persisted ∪ runtime)`.
+- Statuses: `installed | running | error | stopped` (mcps.ts:67 `toInstanceStatus`).
+- `POST /api/mcps/install` (mcps.ts:189) **delegates to `/api/marketplace/install`** via `fastify.inject`, then registers + `start()`s the runtime server (8s budget). So an MCP "install" touches BOTH 4a and 4b. **This is the closest thing to a working cross-store sync** — but it is MCP-specific and one-directional.
+- **Workspace tag:** `PATCH /api/mcps/:id/permissions` sets a single `workspaceId` (C19 — "single-workspace scoping v1", mcps.ts:535). This is the ONLY place a "workspace" appears in install state, and it is a *scope filter on tool exposure*, not an install-presence scoping.
+
+### 4c. Connectors — Vault (`routes/connectors.ts`)
+- Truth: presence of a vault credential under `connector:{id}` (`fastify.vault.getConnectorCredential(id)`). "Connected" = credential exists & not expired.
+- Read: `GET /api/connectors` → `connectorRegistry.getDefinitions()` (a STATIC catalog of all connectors) + `GET /api/connectors/:id/health` for live `connected|disconnected|expired|error` status.
+- Install verb: `POST /api/connectors/:id/connect` — stores token→vault (matches §09 "token → vault"), re-inits the connector, writes audit. **Ungated.** Disconnect/revoke purge vault + OAuth tokens.
+- **Connectors are NOT in `marketplace.db` at all.** They never appear in `installations`/`isInstalled`. The §09 grid showing connectors alongside skills/MCP must federate from `/api/connectors`.
+
+### 4d. The unifying write-side feed — `install_audit` (the one cross-type thing that exists)
+- `fastify.auditStore` (`@waggle/core` `install-audit.ts`) writes to table `install_audit` (schema.ts:135-152). All three stores already write here on install/uninstall/connect/revoke (marketplace.ts, mcps.ts `recordMcpAudit`, connectors.ts `recordConnectorAudit`).
+- **Read:** `GET /api/extend/audit` (extend.ts:88) — ONE shared feed across `native|skill|plugin|mcp|connector|marketplace` with `?type=` / `?capability=` filters. The `AuditStore` API: `getRecent(limit)`, `getRecentByType(type, limit)`, `getByCapability(name)`.
+- **CRITICAL GAP for the count bar:** `install_audit` has **NO `workspace_id` column** (schema.ts:135-152 — confirmed; `ai_interactions`/`execution_traces` DO have it at schema.ts:174/219, install_audit does not). It is an append-only event log, not a "current installed set." A "N in this workspace" count therefore has **zero backing data** today.
+
+---
+
+## 5. The install-audit trail (what's already auditable)
+
+`RecordAuditInput` fields (used by all three route files): `capabilityName`, `capabilityType` (`native|skill|plugin|mcp|connector|marketplace`), `source`, `riskLevel` (`low|medium|high|critical`), `trustSource` (`builtin|starter_pack|local_user|third_party_verified|third_party_unverified|unknown|security-gate`), `approvalClass` (`standard|elevated|critical|blocked`), `action` (`proposed|approved|installed|rejected|failed|blocked|uninstalled`), `initiator` (`agent|user|system`), `detail`. CHECK constraints in schema.ts:143-150 enforce these — **adding a value (e.g. a `synced` action or a `workspace_id`) is a migration + CHECK change**, and the file warns these drifted once and crashed `acquire_capability` (schema.ts:139-142). Treat the audit vocabulary as a contract.
+
+The marketplace install route writes audit rows for CRITICAL-block / HIGH-block / HIGH-force-override / MEDIUM / LOW / blocked-by-installer / forceInsecure-override (marketplace.ts:222-415) — a thorough trust trail, all global.
+
+---
+
+## 6. Exact integration points a PR4 "sync store" would hook into
+
+**Reads (catalog + installed-state):**
+- `GET /api/marketplace/search` (grid; annotates `installed` per pkg) — or the bare `GET /api/marketplace?type=` alias the FE already uses.
+- `GET /api/marketplace/installed` → `MarketplaceDB.listInstallations()`.
+- `GET /api/mcps` (catalog ⋈ persisted ⋈ runtime; `installed` + `status`).
+- `GET /api/connectors` + `GET /api/connectors/:id/health` (connector connected-state).
+- `GET /api/marketplace/categories` (the All/Skills/Connectors/MCP filter taxonomy).
+- `GET /api/extend/audit` (the existing cross-type unified feed — the natural read for a "what's installed across everything" view, modulo it being a log).
+
+**Install verbs (the three one-click flows):**
+- skill / plugin → `POST /api/marketplace/install` `{ packageId }` (PRO). FE adapter: `adapter.installMarketplacePackage(packageId)` → `/api/marketplace/install` (adapter.ts:1339).
+- mcp → `POST /api/mcps/install` `{ mcpId, settings, force }` (PRO) — delegates to marketplace install + runtime start; uninstall via `POST /api/mcps/:id/revoke`.
+- connector → `POST /api/connectors/:id/connect` `{ token|apiKey }` (token→vault); disconnect `POST /api/connectors/:id/disconnect`, strong revoke `POST /api/connectors/:id/revoke`.
+
+**Installer functions (package layer, if the store goes below the routes):**
+- `MarketplaceInstaller.install(InstallRequest)` / `.uninstall(packageId)` / `.installPack(slug)` (installer.ts:81/256/210).
+- `MarketplaceDB.isInstalled(id)` / `recordInstallation(...)` / `markUninstalled(id)` / `getInstalledCount()` / `listInstallations()` (db.ts).
+
+**Server decoration:** `fastify.marketplace: MarketplaceDB | null` (index.ts:506); `fastify.connectorRegistry`, `fastify.vault`, `fastify.auditStore`, `fastify.agentState.mcpRuntime` are the sibling singletons a unified store would coordinate.
+
+**Current FE consumer to refactor:** `apps/web/src/components/os/apps/MarketplaceApp.tsx` (417 LOC) — already federates 6 domains at read (skill via `getMarketplace`, packs, mcp via `getMcps`+`getMarketplace`, persona, connector via `getConnectors`, model, template) but holds install state in **local `useState extensions[]`**; install just patches the row (`installed: true`) — **no shared store, no cross-view propagation.** View-model federation already lives in `apps/web/src/lib/extension-catalog.ts`.
+
+---
+
+## 7. Key deltas vs the §09 contract (territory, not plan)
+
+1. **No single install store.** Three stores (marketplace `installations` / `.mcp.json`+runtime / vault), three key types (numeric id / string name / string id), three install verbs, three tiers (PRO / PRO / ungated). The §09 "one store … reflects in all" is the central build.
+2. **No workspace scoping of install state.** `installations` and `install_audit` have no `workspace_id`; vault is global; only MCP carries a single optional `workspaceId` *tool-scope* tag. The "N in this workspace" count bar has no data source — needs either a new scoping dimension or redefinition to a global count.
+3. **No agent-suggestion endpoint.** The §09 "agent-suggestion box (connector + skill + tool, each with a why)" has no backing route. FTS search (`/api/marketplace/search`) + the agent's `acquire_capability` path (the verbose-`need` FTS handling in db.ts:91-99 / `toFtsMatchQuery`) is the nearest substrate, but a "recommend one of each kind with a reason" composite does not exist.
+4. **No inline-in-chat picker route.** Variation B (mid-conversation connector offer with vault-aware approval) reuses the same install verbs but has no dedicated surfacing/approval endpoint; the connector connect flow + the existing approval/confirmation machinery (`routes/approval.ts`, agent `confirmation.ts`) are the substrate.
+5. **Connectors absent from `marketplace.db`.** Any unified grid must federate connectors from `/api/connectors`, not from the marketplace catalog — they share no row shape with `MarketplacePackage`.
+6. **Audit feed is a log, not a set.** `GET /api/extend/audit` unifies *events* across types but cannot answer "what is currently installed in workspace W" without scanning + reducing; the count bar wants a live set.
+
+---
+
+## 8. Risks / sharp edges for a builder
+
+- **Schema DDL is invisible.** The `packages`/`sources`/`packs`/`installations`/`scan_history` tables exist only inside the committed `marketplace.db` seed binary — there is no `CREATE TABLE` in TS. Adding a `workspace_id` to `installations` means a runtime `ALTER TABLE` migration in `MarketplaceDB.migrateSchema()` (the established pattern, db.ts:57-78), not an edit to a schema file.
+- **`install_audit` CHECK constraints are a hard contract.** Adding a `synced` action or any new vocabulary needs a coordinated `runMigrations()` + CHECK rewrite; schema.ts explicitly records a prior drift that crashed `acquire_capability`.
+- **PRO-gating vs "instant Add".** Skill install is PRO-gated server-side; the §09 instant micro-state must reconcile with `requireTier('PRO')` (or rely on a different non-gated skill path).
+- **MCP install is the only working cross-store coupling** (`/api/mcps/install` writes both `installations` and `.mcp.json`+runtime, and revoke keeps marketplace `installed` honest, mcps.ts:509-519). A unified store should mirror this bidirectional bookkeeping for the other types, or it will drift (the FE will show "installed" after a revoke, etc.).
+- **OSS-sync constraint (§7.5):** `install_audit` DDL inside `mind/schema.ts` is OSS-excluded (Waggle governance). A `workspace_id` addition there has nowhere to land on the public mirror — fine for the monorepo, but flag it as a curated-strip item.
+- **Three-store consistency under failure.** The installer's `notify` hook is fire-and-forget to `localhost:3000` (not the running sidecar port); a sync store cannot rely on the notify callback to invalidate caches.
+
+---
+
+## 9. Open questions for the founder/lead
+
+1. **Workspace scoping:** Is "N in this workspace" a real per-workspace install set (requires a new `workspace_id` on installs + per-workspace activation model), or a relabeled global count for v1? This is the single biggest decision — it determines whether the sync store needs a new data dimension across all three backends.
+2. **Skill install tier:** Should the §09 instant skill "Add" stay PRO-gated, or is there a FREE skill-install lane? (CLAUDE.md moat: skills/connectors are the upgrade trigger — so PRO-gating may be intentional and the "Add" CTA should show the upgrade nudge instead of installing.)
+3. **Agent-suggestion box:** Build a new composite recommend endpoint (one connector + one skill + one tool + "why"), or compose it client-side from three `search` calls + an LLM rationale? No backing route exists either way.
+4. **Sync store location:** A FE-only store (React context / Zustand over the existing routes) vs a new server-side unified `/api/install-state` read that reduces all three stores into one shape. The §09 "one store reflects in all" is achievable purely client-side IF every view subscribes to it; a server read is only needed for the count bar's correctness across reloads.
+5. **Inline-in-chat (Variation B):** Reuse `/api/connectors/:id/connect` + existing approval machinery, or a dedicated in-chat install/approval contract? Decide before building the chat surface.