waggle-os/docs/backend-map/sections/05g-subsystem-skills-marketplace-wiki.md

# Subsystem 05g — Skills Lifecycle · Marketplace · Wiki-Compiler

**Purpose.** This subsystem covers everything a user installs, authors, recommends, retires, browses-and-buys, and compiles-into-knowledge. It spans three packages — `@waggle/agent` (skill lifecycle helpers), `@waggle/marketplace` (the package catalog + installer + security gate), and `@waggle/wiki-compiler` (turns memory frames into an interlinked wiki) — all exposed to the frontend through Fastify routes under `/api/skills/*`, `/api/plugins/*`, `/api/marketplace/*`, and `/api/wiki/*`. **Strategically, skills + connectors are the paid-tier upgrade trigger:** custom skills and the marketplace are gated to PRO and above (`customSkills: false` on FREE, `requireTier('PRO')` on install/publish), while Memory + Harvest + the wiki stay free.

---

## 1. Mental Model

A **skill** is a Markdown file (`~/.waggle/skills/{name}.md`) with optional YAML frontmatter. Its content is injected into the agent's system prompt. Skills have a full lifecycle: authored/templated → recommended in context → usage-tracked → auto-retired when idle → hot-reloaded when the file changes on disk.

The **marketplace** is a local SQLite catalog (`~/.waggle/marketplace.db`) of three installable package kinds — `skill`, `plugin`, `mcp` — synced from ~30 external sources (GitHub, npm, ClawHub, SkillsMP, LobeHub, awesome-lists, web registries). Installing a package runs a multi-layer **SecurityGate** scan, then writes files to `~/.waggle/skills/`, `~/.waggle/plugins/`, or `.mcp.json`.

The **wiki-compiler** reads the personal memory substrate (FrameStore + KnowledgeGraph + HybridSearch) and produces five page types (entity / concept / synthesis / index / health) as Markdown, persisted to a `wiki_pages` SQLite table, with incremental compilation via a frame-ID watermark and exporters to Obsidian and Notion.

```mermaid
flowchart TD
  subgraph Skills["Skill Lifecycle (@waggle/agent)"]
    SC[skill-creator.ts<br/>generateSkillMarkdown / detectWorkflowPattern]
    SF[skill-frontmatter.ts<br/>parseSkillFrontmatter / scopes]
    SR[skill-recommender.ts<br/>SkillRecommender.recommend]
    SU[skill-usage.ts<br/>recordSkillUsage JSON store]
    SRet[skill-retirement.ts<br/>retireStaleSkills]
    SW[skill-watcher.ts<br/>watchSkillDirectory hot-reload]
    DISK[(~/.waggle/skills/*.md)]
    SC --> DISK
    DISK --> SF
    DISK --> SR
    DISK <--> SW
    SU --> SRet
    DISK --> SRet
  end

  subgraph MP["Marketplace (@waggle/marketplace)"]
    MDB[(marketplace.db<br/>sources/packages/packs/installations/scan_history)]
    SYNC[MarketplaceSync<br/>9 adapters]
    INST[MarketplaceInstaller]
    GATE[SecurityGate<br/>4 scan layers]
    SYNC --> MDB
    MDB --> INST
    INST --> GATE
    INST --> DISK
    INST --> MCPJSON[(.mcp.json)]
    INST --> PLUG[(~/.waggle/plugins/)]
  end

  subgraph Wiki["Wiki Compiler (@waggle/wiki-compiler)"]
    WC[WikiCompiler]
    WS[CompilationState<br/>wiki_pages + watermark]
    SYN[resolveSynthesizer<br/>anthropic→ollama→echo]
    MIND[(personal.mind:<br/>KnowledgeGraph/FrameStore/HybridSearch)]
    MIND --> WC
    WC --> SYN
    WC --> WS
    WS --> OBS[Obsidian export]
    WS --> NOT[Notion export]
  end

  API[Fastify routes<br/>/api/skills /api/plugins<br/>/api/marketplace /api/wiki] --> Skills
  API --> MP
  API --> Wiki
```

---

## 2. Skill Frontmatter & Scopes (`packages/agent/src/skill-frontmatter.ts`)

Skills may begin with a `---`-delimited YAML-ish block. The parser is hand-rolled (no YAML dependency): it handles top-level `key: value` pairs plus a nested `permissions:` block.

### `SkillFrontmatter`

| Field | Type | Nullable | Meaning |
|---|---|---|---|
| `name` | `string` | yes | Display name |
| `description` | `string` | yes | One-line description |
| `scope` | `SkillScope` | yes | Where the skill is available. Defaults to `'personal'` when omitted |
| `promoted_from` | `SkillScope[]` | yes | Audit trail of prior scopes; appended on each promotion, never rewritten |
| `permissions` | `Partial<{fileSystem, network, codeExecution, externalServices, secrets, browserAutomation: boolean}>` | yes | Declared permission flags |

### `SkillScope` & promotion chain

`type SkillScope = 'personal' | 'workspace' | 'team' | 'enterprise'`. The ordered constant `SKILL_SCOPE_ORDER` drives one-step-at-a-time promotion (demotion is NOT supported).

| Export | Signature | Behavior |
|---|---|---|
| `parseSkillFrontmatter` | `(content: string) => ParsedSkill` | Returns `{ frontmatter, body }`. If no leading `---` or no closing `\n---`, returns empty frontmatter + full content as body |
| `nextScope` | `(current: SkillScope) => SkillScope \| null` | Next scope up, or `null` at `enterprise` |
| `serializeFrontmatter` | `(fm: SkillFrontmatter, body: string) => string` | Rebuilds a `SKILL.md` string preserving scope + `promoted_from`; used by `promote_skill` to write the change to disk |

`ParsedSkill` = `{ frontmatter: SkillFrontmatter; body: string }`.

> Note: the **server-side** skill loader (`loadSkills` in `prompt-loader.ts`) does NOT parse frontmatter — it loads each `.md` file as `{ name, content }` (trimmed). Frontmatter parsing is used by promotion/validation paths, not by the prompt-injection path.

---

## 3. Skill Creation (`packages/agent/src/skill-creator.ts`)

### `SkillTemplate`

| Field | Type | Meaning |
|---|---|---|
| `name` | `string` | Human name (kebab-cased on output) |
| `description` | `string` | Description line |
| `triggerPatterns` | `string[]` | Phrases that should activate the skill |
| `steps` | `string[]` | Ordered procedure |
| `tools` | `string[]` | Tools the skill uses |
| `category` | `string` | Free-form category |

| Export | Signature | Behavior |
|---|---|---|
| `generateSkillMarkdown` | `(template: SkillTemplate) => string` | Emits a valid `SKILL.md` with `---name/description---` frontmatter, then `# Title`, optional `## Trigger Patterns`, `## Steps` (numbered), `## Tools Used`, `## Category`. Name is kebab-cased. Output is compatible with `validateSkillMd` (`@waggle/sdk`) and `parseSkillFrontmatter` |
| `detectWorkflowPattern` | `(messages: Array<{role, content, toolsUsed?}>) => SkillTemplate \| null` | Mines session history for a repeatable tool sequence. Requires ≥6 messages and ≥3 tools; finds a 3-5-tool window repeated ≥2×; infers name (`tool-then-tool-then-tool`), description, trigger patterns (first 3 user msgs truncated to 80 chars), and a category via `inferCategory` (keyword map → research/coding/knowledge/writing/planning/general) |

---

## 4. Skill Recommendation (`packages/agent/src/skill-recommender.ts`)

A keyword/synonym/bigram + TF-IDF scorer that suggests installed skills relevant to a conversation context. No embeddings — pure lexical.

### Types

```
SkillRecommendation = { skillName: string; reason: string; relevanceScore: number /* 0-1 */ }
SkillRecommenderDeps = { getSkills: () => Array<{name, content}>; activeSkills?: string[] }
```

`class SkillRecommender(deps)` → `recommend(context: string, topN = 3): SkillRecommendation[]`.

**Scoring weights** (multiplied by an IDF weight `log(totalDocs/df)+1`, rarer terms score higher):

| Signal | Weight |
|---|---|
| Exact keyword in skill name | `3×` |
| Synonym match in name | `2×` |
| Exact keyword in content (× small TF boost, capped) | `1×` |
| Synonym match in content | `0.7×` |
| Bigram (phrase) overlap in combined text | `+0.5` flat per bigram |

Score normalized against `keywords.length * 3`, capped at 1.0, rounded to 3 decimals; results `< 0.05` dropped. Active skills (`deps.activeSkills`) are excluded. There are 13 hardcoded `SYNONYM_CLUSTERS` (review/code/write/research/plan/decide/risk/meeting/brainstorm/task/team/explain/retrospective) and a STOP_WORDS set. The `reason` string is human-readable, e.g. `Skill name matches your topic: "review"`.

---

## 5. Skill Usage Tracking (`packages/agent/src/skill-usage.ts`)

A standalone JSON sidecar store at `~/.waggle/skill-usage.json` (NOT SQLite). Atomic writes (`.{pid}.tmp` + rename). Missing/corrupt file ⇒ `{}`.

**Schema:** `Record<skillStem, { lastUsedAt: ISOstring; count: number }>` (`SkillUsageIndex` / `SkillUsageEntry`).

| Export | Signature | Behavior |
|---|---|---|
| `getSkillUsagePath` | `(waggleHome) => string` | `{waggleHome}/skill-usage.json` |
| `loadSkillUsage` | `(waggleHome) => SkillUsageIndex` | `{}` on any FS/parse failure |
| `saveSkillUsage` | `(waggleHome, index) => void` | Atomic write; mkdirs home if absent |
| `recordSkillUsage` | `(waggleHome, skillName, nowFn?) => SkillUsageEntry` | Sets `lastUsedAt=now`, increments `count` |
| `forgetSkillUsage` | `(waggleHome, skillName) => void` | Deletes the entry (called after retirement) |

---

## 6. Skill Retirement / Decay (`packages/agent/src/skill-retirement.ts`)

Walks `~/.waggle/skills/` and **moves** (not deletes) skills idle longer than `maxIdleDays` (default **90**) into `~/.waggle/skills-archive/` with a timestamp-prefixed filename. Recoverable; clears the usage entry on success.

**Retirement rules:** only personal-scope skills auto-retire (team/enterprise are co-owned → admin-only; workspace skills are left alone). "Last activity" = `skill-usage.json` timestamp, falling back to the file's mtime (so a never-used skill's decay window starts at install time). On stat error, assumed fresh (no over-retirement).

```
RetireOptions = { maxIdleDays?=90; dryRun?=false; now?: ()=>Date; improvementSignals?: ImprovementSignalStore }
RetireReport = { scanned: number; retired: string[]; archived: string[]/*abs paths*/;
                 skipped: Array<{name, reason}>; dryRun: boolean }
```

`retireStaleSkills(waggleHome, opts?) => RetireReport`. Best-effort per file (never throws on one bad file). When `improvementSignals` is provided, emits a `workflow_pattern` signal `retire:{skillName}` for observability.

---

## 7. Skill Hot-Reload Watcher (`packages/agent/src/skill-watcher.ts`)

Uses Node's built-in `fs.watch()` (not chokidar — keeps the Tauri binary small) on a single directory, filtered to `.md` direct children, with a **150 ms debounce** (editors emit 2-3 events per save). Creates the dir if missing; silently no-ops if `fs.watch` throws on the platform.

```
SkillWatcherOptions = { onChange: (changedFiles: string[]) => void; debounceMs?=150 }
SkillWatcherHandle = { close(): void /* idempotent */ }
watchSkillDirectory(dir, opts) => SkillWatcherHandle
```

Callback errors are swallowed so they never kill the watcher.

---

## 8. Skills/Plugins/Hooks HTTP API (`packages/server/src/local/routes/skills.ts`)

Base dir: `server.localConfig.dataDir || ~/.waggle`. Skills live in `{home}/skills/`, plugins in `{home}/plugins/`. On first run with an empty skills dir, **starter skills auto-install** (marker file `.starter-installed`). Every skill mutation (`POST`/`PUT`/`DELETE`) reloads `server.agentState.skills` in place and writes through `redactSkillContent` (strips secrets + user-home paths) and `computeSkillHash` (change detection). Skill writes apply path-traversal guards (`name` may not contain `..`, `/`, `\`, or spaces).

### Skills

| Method | Full path | Request | Response | Stream? |
|---|---|---|---|---|
| `GET` | `/api/skills` | — | `{ skills: [{name, length, preview(200ch)}], count, directory }` | no |
| `GET` | `/api/skills/:name` | path param | `{ name, content }` · 404 if missing · 400 on traversal | no |
| `POST` | `/api/skills` | `{ name, content }` | `{ ok, name, path }` · 400 on bad name | no |
| `POST` | `/api/skills/create` | `{ name, description, steps[], tools?, category? }` | `{ success, path, registered, skill:{...} }`. Generates via `generateSkillMarkdown`, kebab-cases name, records audit | no |
| `PUT` | `/api/skills/:name` | `{ content }` | `{ ok, name }` · 404 if missing | no |
| `DELETE` | `/api/skills/:name` | path param | `{ ok, name }` · 404 if missing | no |
| `GET` | `/api/skills/suggestions` | `?context=&topN=` | `{ suggestions: SkillRecommendation[], count }` · 400 if no context. Uses `SkillRecommender` | no |
| `GET` | `/api/skills/hash-status` | — | `server.skillHashStore.checkAll(...)` — which skills changed on disk | no |
| `POST` | `/api/skills/test` | `{ skillName, testInput? }` | Sandbox/dry-run: `{ skill:{...metadata}, wouldInject, wouldInjectLength, testPreview? }`. Parses frontmatter, shows what would be injected into the prompt without executing | no |

### Starter Pack & Capability Packs (built-in, from `@waggle/sdk`)

| Method | Full path | Request | Response |
|---|---|---|---|
| `POST` | `/api/skills/starter-pack` | — | `{ ok, installed[], count }` — install ALL starter skills |
| `GET` | `/api/skills/starter-pack/catalog` | — | `{ skills:[{id,name,description,family,familyLabel,state,isWorkflow}], families:[{id,label}] }`. `state` ∈ `active`/`installed`/`available` |
| `POST` | `/api/skills/starter-pack/:id` | path param | `{ ok, skill:{id,name,state} }` · 404/409 on missing/exists. Assesses trust + records audit |
| `GET` | `/api/skills/capability-packs/catalog` | — | `{ packs:[{...pack, skillStates, packState(available/incomplete/complete), installedCount, totalCount}] }` |
| `POST` | `/api/skills/capability-packs/:id` | path param | `{ ok, pack:{id,name}, installed[], skipped[], errors? }` — installs every skill in the pack |

The 17 known starter skills map to 7 capability **families** (`SKILL_FAMILIES`): writing, research, decision, planning, communication, code, creative. Three are multi-agent workflow skills (`WORKFLOW_SKILLS`): `research-team`, `review-pair`, `plan-execute`.

### Install Audit

| Method | Full path | Request | Response |
|---|---|---|---|
| `GET` | `/api/audit/installs` | `?limit=` (≤100) | `{ entries:[{id,timestamp,capabilityName,capabilityType,source,riskLevel,trustSource,approvalClass,action,initiator,detail}] }` |

### Plugins (managed by `PluginManager`)

| Method | Full path | Request | Response |
|---|---|---|---|
| `GET` | `/api/plugins` | — | `{ plugins[], count, directory }` |
| `POST` | `/api/plugins/install` | `{ sourceDir }` or `{ path }` | `{ ok, source }`. Hot-reloads via `pluginRuntimeManager.register/enable`. Called by `MarketplaceInstaller.notifyServer()` |
| `DELETE` | `/api/plugins/:name` | path param | `{ ok, name }` |
| `GET` | `/api/plugins/:name/tools` | path param | `{ pluginName, tools:[{name,description,parameters,hasImplementation,implPath,content}], toolsDir }` |
| `GET` | `/api/plugins/:name/tools/:toolName` | path | `{ exists, content, path }` — returns a generated template if no impl exists |
| `PUT` | `/api/plugins/:name/tools/:toolName` | `{ content }` | `{ ok, path, toolName }` — must `export` an `execute()` function; hot-reloads |
| `POST` | `/api/plugins/:name/tools` | `{ name, description, parameters? }` | `{ ok, tool, totalTools }` — appends tool decl to `plugin.json` · 409 if exists |
| `DELETE` | `/api/plugins/:name/tools/:toolName` | path | `{ ok, deleted }` · 404 if missing |

### Hooks (deny-rules at `~/.waggle/hooks.json`)

| Method | Full path | Request | Response |
|---|---|---|---|
| `GET` | `/api/hooks` | — | `{ rules:[{type,tools[],pattern}], total }` |
| `POST` | `/api/hooks` | `{ type:'deny', tools[], pattern }` | `{ ok, rules }` · 400 on bad shape |
| `DELETE` | `/api/hooks/:index` | path index | `{ ok, rules }` · 404 if out of range |

---

## 9. Marketplace Data Model (`packages/marketplace/src/types.ts`, `db.ts`)

SQLite DB at `~/.waggle/marketplace.db` via better-sqlite3 (WAL, `foreign_keys=ON`). FTS5 full-text index `packages_fts` joined on `packages.id = fts.rowid`. Auto-migrations on construct add `is_custom` and `sync_state` columns to `sources`. The `MarketplaceDB` constructor seeds the MCP registry **only** on non-empty DBs.

### Table: `sources` → `MarketplaceSource`

| Column | Type | Null | Meaning |
|---|---|---|---|
| `id` | number | no | PK |
| `name` | string | no | Internal key (e.g. `clawhub`) |
| `display_name` | string | no | UI label |
| `url` | string | no | Source URL |
| `source_type` | enum | no | `marketplace`/`registry`/`github_org`/`community_repo`/`curated_list`/`aggregator`/`npm_registry`/`official_marketplace`/`commercial_marketplace`/`tool`/`specification` |
| `platform` | string | no | e.g. `waggle` |
| `total_packages` | number | no | Cached count |
| `install_method` | enum | no | `npm`/`git_clone`/`download`/`api_fetch`/`cli`/`manual` |
| `api_endpoint` | string | yes | Sync endpoint |
| `description` | string | no | — |
| `last_synced_at` | string | yes | ISO |
| `is_custom` | boolean | no | User-added (vs built-in seed). Only `is_custom` sources are deletable |
| `sync_state` | TEXT(JSON) | yes | Resumable-pagination cursor (migration column) |

### Table: `packages` → `MarketplacePackage`

| Column | Type | Null | Meaning |
|---|---|---|---|
| `id` | number | no | PK |
| `source_id` | number | no | FK → sources |
| `name` | string | no | Unique within source (`UNIQUE(name, source_id)`) |
| `display_name` | string | no | UI name |
| `description` | string | no | — |
| `author` | string | no | — |
| `package_type` | enum | no | `skill`/`plugin`/`mcp_server`/`template`/`pack` |
| `waggle_install_type` | enum | no | `skill`/`plugin`/`mcp` — drives installer dispatch |
| `waggle_install_path` | string | no | e.g. `skills/x.md`, `plugins/x/`, `.mcp.json` |
| `version` | string | no | semver |
| `license` | string | yes | SPDX |
| `repository_url` | string | yes | — |
| `homepage_url` | string | yes | — |
| `downloads` | number | no | popularity sort |
| `stars` | number | no | popularity sort |
| `rating` | number | no | — |
| `rating_count` | number | no | — |
| `category` | string | no | one of `PACKAGE_CATEGORIES` ids |
| `subcategory` | string | yes | — |
| `install_manifest` | JSON | yes | `InstallManifest` (see below) — parsed on read |
| `platforms` | JSON string[] | no | — |
| `min_waggle_version` | string | yes | — |
| `dependencies` | JSON string[] | no | — |
| `packs` | JSON string[] | no | — |
| `created_at` / `updated_at` | string | no | ISO |

**Security columns** (added by the installer's `recordScanResult()`, optional; type `PackageSecurityColumns`): `security_status` (`unscanned`/`clean`/`low`/`medium`/`high`/`critical`), `security_score` (number), `last_scanned_at`, `content_hash`, `scan_engines` (JSON), `scan_findings` (JSON), `scan_blocked` (0/1). A package augmented with these is `ScannedPackage`.

### Table: `packs` + `pack_packages` → `MarketplacePack`

| Column | Type | Null | Meaning |
|---|---|---|---|
| `id` | number | no | PK |
| `slug` | string | no | Stable key (used by `/packs/:slug`) |
| `display_name` | string | no | — |
| `description` | string | no | — |
| `target_roles` | string | no | comma-sep role hints |
| `icon` | string | no | emoji |
| `priority` | enum | no | `core`/`recommended`/`optional` |
| `connectors_needed` | JSON string[] | no | — |
| `created_at` | string | no | — |

`pack_packages` is the many-to-many join (`pack_id`, `package_id`, `is_core`); `package_tags` exists for tag dedup.

### Table: `installations` → `Installation`

| Column | Type | Null | Meaning |
|---|---|---|---|
| `id` | number | no | PK |
| `package_id` | number | no | FK → packages |
| `installed_version` | string | no | — |
| `installed_at` | string | no | `datetime('now')` |
| `install_path` | string | no | where it landed |
| `status` | enum | no | `installed`/`updating`/`failed`/`uninstalled` |
| `config` | JSON | no | user settings — parsed on read |

`listInstallations()` returns the flat `InstalledPackageRow` (installation columns + `pkg_name`, `pkg_display_name`, `waggle_install_type`, `category`) — NOT a nested object.

### Table: `scan_history`

One row per scan (`package_id`, `scanned_at`, `overall_severity`, `security_score`, `content_hash`, `engines_used`, `findings`, `blocked`, `scan_duration_ms`, `triggered_by`).

### `InstallManifest` (stored as JSON on `packages.install_manifest`)

| Field | For | Meaning |
|---|---|---|
| `skill_url` / `skill_content` | skill | fetch URL or inline content |
| `plugin_manifest` (`PluginManifest`) / `git_url` | plugin | manifest to write / repo to clone |
| `mcp_config` (`McpServerConfig`) | mcp | `{ name, command, args[], env? }` |
| `npm_package` / `npm_args` | mcp/plugin | npm install target |
| `post_install` (`PostInstallHook[]`) | any | `run_command`/`create_file`/`append_config` |

`PluginManifest = { name, version, description, skills?[], mcpServers?[], settingsSchema?: Record<string, SettingField> }`; `SettingField = { type:'string'|'number'|'boolean', description, required?, default? }`.

### Search contract (`SearchOptions` → `SearchResult`)

`SearchOptions = { query?, type?, category?, pack?, source?, sort?: 'relevance'|'popular'|'recent'|'name', limit?=50, offset?=0 }`. Raw query is relaxed by `toFtsMatchQuery()` into an OR-of-prefix FTS5 expression (≥2-char tokens, capped at 24, `token*`); null ⇒ unfiltered listing fallback (never throws).

`SearchResult = { packages: MarketplacePackage[]; total; facets: { types, categories, sources: Record<string,number> }; installedCount }`.

---

## 10. Marketplace Installer (`packages/marketplace/src/installer.ts`)

`class MarketplaceInstaller(db, securityConfig?)`. Ensures `~/.waggle/skills/`, `~/.waggle/plugins/`, and `plugins/registry.json` exist on construct. Install flow:

1. Resolve package by id (404-style failure result if missing).
2. Skip if already installed and not `force`.
3. **SecurityGate pre-scan** of resolved content; record result to DB. If `scanResult.blocked` and not `forceInsecure` → return `success:false` with findings.
4. Dispatch on `waggle_install_type`:
   - **skill** → write to `~/.waggle/skills/{name}.md` (inline content, `skill_url`, repo `SKILL.md`, or a generated stub), then `PUT /api/skills/{name}` to notify the server.
   - **plugin** → `git clone` or `npm install` into `~/.waggle/plugins/{name}/`, write `plugin.json`, install bundled skills, update `registry.json`, run post-install hooks, then `POST /api/plugins/install`.
   - **mcp** → optional global npm install, inject user settings into env vars, write into `.mcp.json` (`mcpServers[name]`).
5. On success, `recordInstallation(...)` and attach the scan result.

`InstallRequest = { packageId, installPath?, settings?, force?, forceInsecure? }`; `InstallResult = { success, packageId, packageName, installType, installPath, message, errors?, scanResult? }`. Pack install (`installPack(slug, {force?})`) loops package install and returns `PackInstallResult = { packSlug, packName, totalPackages, installed[], skipped[], failed[] }`. Also: `uninstall(id)`, `scanOnly(id) => ScanResult|null`, `getSecurityReport(id) => string`.

`MarketplaceInstaller.notifyServer()` posts to `WAGGLE_API_URL || http://localhost:3000`; failures are swallowed (files already on disk).

---

## 11. SecurityGate (`packages/marketplace/src/security.ts`) — the install gate

Four scan layers, run before any file is written. `block_threshold` defaults to **HIGH** (CRITICAL + HIGH block by default).

| Layer | Engine id | What |
|---|---|---|
| 1 | `gen_trust_hub` | Cloud URL pre-check (POST `https://ai.gendigital.com/api/scan/lookup`, 15s timeout, fail-open) |
| 2 | `cisco_skill_scanner` | Local deep scan of skill content via optional `skill-scanner` CLI / `cisco-scanner.ts` adapter |
| 3 | `mcp_guardian` | Pattern scan of MCP tool descriptions (optional `mcp-guardian` npm dep, else built-in `mcpPatternScan` fallback) |
| 4 | `waggle_heuristics` | Always-on regex rules `WAG-001..013` on content |

**Severity → score:** CRITICAL=0, HIGH=25, MEDIUM=60, LOW=85, CLEAN=100. Results SHA-256-hashed + cached for 24 h under `~/.waggle/security-cache/`.

`ScanResult = { package_name, package_type, scanned_at, overall_severity: Severity, security_score(0-100), findings: SecurityFinding[], engines_used: SecurityEngine[], content_hash, blocked, scan_duration_ms, ciscoScanResult? }`. `SecurityFinding = { rule_id, severity, category, title, description, location?, engine }`. `SecurityCategory` ∈ prompt_injection / data_exfiltration / malicious_code / privilege_escalation / suspicious_network / obfuscation / sensitive_path_access / tool_poisoning / cross_origin_escalation / rug_pull / untrusted_source. Built-in heuristics catch system-prompt manipulation, exfil commands, sensitive-path access (incl. `~/.waggle/*.mind`), code-exec patterns, credential harvesting, network beaconing, tool poisoning, hidden/zero-width content, supply-chain and cross-origin skill modification. `formatReport(result)` renders a human-readable text report.

---

## 12. Marketplace Sync (`packages/marketplace/src/sync.ts`)

`class MarketplaceSync(db, vaultLookup?)` → `syncAll(opts?: SyncOptions) => SyncResult[]`. Nine adapters, tried in priority order (first `canSync()` wins): `clawhub`, `skillsmp`, `lobehub`, `awesome-list`, `github-repo-content`, `npm-search`, `web-registry`, `github` (orgs/official), `generic`. Resumable pagination via `sources.sync_state` (429 → save cursor, return gracefully). After a full sync, `deduplicatePackages(db)` collapses normalized-name dupes (keeps highest `stars+downloads`). Vault keys for premium sources: `marketplace:source:{name}:api_key`.

`SyncOptions = { sources?, fullRefresh?, dryRun?, scanDuringSync?=false }`; `SyncResult = { source, added, updated, removed, errors[] }`. Exported helpers: `parseAwesomeListMarkdown`, `parseNpmSearchResults`, `normalizeName`, `deduplicatePackages`.

### Categories & MCP registry & enterprise packs

- `categories.ts`: 21 `PACKAGE_CATEGORIES` (`{id,name,icon,description}`) + `categorizePackage(name, desc)` keyword classifier + `recategorizeAll(db)`.
- `mcp-registry.ts`: `MCP_SERVERS: McpServerEntry[]` + `seedMcpServers(db)` (seeds curated MCP packages with real install manifests).
- `enterprise-packs.ts`: `ENTERPRISE_PACKS: EnterprisePack[]` (3 KVARK-conditional packs: enterprise-document-qa, compliance-workflow, knowledge-graph-enrichment) — only surfaced when KVARK is connected. `EnterprisePack = { slug, display_name, description, target_roles, icon, skills[], kvarkRequirements[] }`.

---

## 13. Marketplace HTTP API (`packages/server/src/local/routes/marketplace.ts`)

DB handle from `fastify.marketplace` (503 `Marketplace not available` if absent). **Tier gating** marks the upgrade triggers.

| Method | Full path | Request | Response | Tier gate | Stream? |
|---|---|---|---|---|---|
| `GET` | `/api/marketplace/search` | `?query=&type=&category=&pack=&source=&sort=&limit=&offset=` | `SearchResult` + each pkg annotated with `installed`, `scanStatus`(passed/failed/not_scanned/unavailable), `scanScore`; plus `categories` | — | no |
| `GET` | `/api/marketplace/packs` | — | `{ packs, total }` | — | no |
| `GET` | `/api/marketplace/packs/:slug` | path | `{ pack, packages[] }` · 404 | — | no |
| `GET` | `/api/marketplace/enterprise-packs` | — | `{ packs, total, kvarkRequired }` (empty + hint if KVARK not configured) | **ENTERPRISE** | no |
| `POST` | `/api/marketplace/install` | `{ packageId, installPath?, settings?, force?, forceInsecure? }` | `InstallResult` + `security:{severity,score,findingsCount,findings,warnings?}`; CRITICAL→403, HIGH→403 unless `force`, MEDIUM/LOW→proceeds-with-audit. 200 on success / 422 on failure | **PRO** | no |
| `POST` | `/api/marketplace/uninstall` | `{ packageId }` | `InstallResult` (200/422) | — | no |
| `GET` | `/api/marketplace/installed` | — | `{ installations: InstalledPackageRow[], total }` | — | no |
| `POST` | `/api/marketplace/security-check` | `{ packageId }` | `{ packageId, severity, score, blocked, enginesUsed, findingsCount, findings, durationMs, contentHash }` · 404 | — | no |
| `GET` | `/api/marketplace/sources` | — | `{ sources: (MarketplaceSource & {package_count})[], total }` | — | no |
| `POST` | `/api/marketplace/sources` | `{ name, url, displayName? }` | `201 { source, syncResult }`; auto-detects `source_type`, triggers an immediate sync · 400/409 | — | no |
| `DELETE` | `/api/marketplace/sources/:id` | path | `{ deleted, sourceId, name }` · 403 if built-in · 404 | — | no |
| `GET` | `/api/marketplace/categories` | — | `{ categories: PACKAGE_CATEGORIES, total }` | — | no |
| `POST` | `/api/marketplace/sync` | `{ sources? }` | `{ sourcesChecked, packagesAdded, packagesUpdated, errors[], details: SyncResult[] }`; emits a notification on new packages | — | no |
| `GET` | `/api/marketplace/security-status` | — | `{ ciscoScannerAvailable, jsSecurityGateVersion, totalScanned, totalPassed, totalFailed, hint? }` | — | no |
| `POST` | `/api/marketplace/publish` | `{ skillName }` | `201 { success, packageId, skillName, metadata, security }`; reads `~/.waggle/skills/{name}.md`, validates frontmatter (`validateSkillMd`), SecurityGate scan (403 if blocked), upserts under a `user-published` source | **PRO** | no |

The install/check routes construct a heuristics-only `SecurityGate` (cloud/cisco/guardian layers disabled at the route level for speed).

---

## 14. Wiki-Compiler Data Model (`packages/wiki-compiler/src/types.ts`, `state.ts`)

State lives in the **personal mind** SQLite DB (`@waggle/core` `MindDB`) — same DB as memory, not a separate file. `class CompilationState(db)` ensures two tables on construct.

### Table: `wiki_pages` → `PageRecord`

| Column | Type | Null | Meaning |
|---|---|---|---|
| `slug` | TEXT | no | PK (URL-safe) |
| `page_type` | TEXT | no | `entity`/`concept`/`synthesis`/`index`/`health` |
| `name` | TEXT | no | Display name |
| `content_hash` | TEXT | no | SHA-256(16-char) — change detection |
| `markdown` | TEXT | no | Full page body (migration-added column) |
| `frame_ids` | TEXT | no | JSON `number[]` of source frame IDs |
| `compiled_at` | TEXT | no | `datetime('now')` |
| `source_count` | INTEGER | no | distinct sources |
| `notion_page_id` | TEXT | yes | M-13 Notion export delta tracking (migration column) |

`upsertPage(...)` returns `{ action: 'created' | 'updated' | 'unchanged' }` — `unchanged` when `content_hash` matches (the incremental-compile no-op).

### Table: `wiki_watermark` → `CompilationWatermark`

Single-row (`id=1 CHECK`): `last_frame_id`, `last_compiled_at`, `pages_compiled`. `getMaxFrameId()` reads `MAX(id)` from `memory_frames`; `getFramesSince(id, limit)` pulls newer frames for the incremental "does this entity get mentioned in new frames?" check.

### Page frontmatter (`WikiPageFrontmatter`)

`{ type, name, entity_type?, confidence: number, sources: number, last_compiled: ISO, frame_ids: number[], related_entities: string[] }`. A `WikiPage` = `{ slug, frontmatter, markdown, contentHash }`.

---

## 15. Wiki Compilation (`packages/wiki-compiler/src/compiler.ts`)

`class WikiCompiler(kg, frames, search, state, config)` where `CompilerConfig = { synthesize: (prompt)=>Promise<string>; outputDir?='wiki'; minFramesPerPage?=2; maxFramesPerCall?=30; minConfidence?=0.3 }`.

| Method | Page type | How it builds |
|---|---|---|
| `compileEntityPage(entity)` | entity | HybridSearch on `entity.name`, gather frames + KG in/out relations; LLM via `entityPagePrompt`. Returns `null` if `< minFramesPerPage`. Confidence: >5 frames→0.9, >2→0.7, else 0.5 |
| `compileConceptPage(name)` | concept | Search on concept + `kg.searchEntities`; `conceptPagePrompt`. Confidence 0.85/0.6 |
| `compileSynthesisPage(topic)` | synthesis | Search 2× frames, group by `frame.source`; **needs ≥2 sources** else `null`. `synthesisPagePrompt` finds cross-source patterns/contradictions. slug = `synthesis-{topic}`. Confidence 0.85/0.65 |
| `compileIndex()` | index | Navigable catalog of all pages grouped by type with `[[wikilinks]]` |
| `compileHealth()` | (report) | See §16 |
| `compile({incremental?=true, concepts?})` | all | Orchestrates: entity pages (≤200 entities, skipping ones with no new frame mentions when incremental) → concept pages (`detectConcepts` or supplied) → synthesis pages → index → update watermark → health check |

`compile()` returns `CompilationResult = { pagesCreated, pagesUpdated, pagesUnchanged, entityPages[], conceptPages[], synthesisPages[], healthIssues, watermark, durationMs }`. Export helpers: `exportToMarkdown(): Map<slug,markdown>` and `exportToDirectory(dir)`. The three prompt builders live in `prompts.ts` and each instruct the LLM to cite frame IDs and output ONLY the body (frontmatter is added by `buildFrontmatter`).

---

## 16. Wiki Health Report (`compileHealth()` → `HealthReport`)

| Field | Type | Meaning |
|---|---|---|
| `totalEntities` | number | KG entity count |
| `totalFrames` | number | `frames.getStats().total` |
| `totalPages` | number | wiki_pages rows |
| `coverage` | number (0-1) | entity pages / compilable entities (entity with ≥1 relation OR type person/project); UI renders as % |
| `stalePageCount` | number | pages flagged `stale_page` |
| `issues` | `HealthIssue[]` | see below |
| `dataQualityScore` | number (0-100) | heuristic: entities(20) + frames(20/10) + pages(20/10) + 40 − 10×high-severity-issues |
| `compiledAt` | ISO | — |

`HealthIssue = { type: HealthIssueType, severity:'high'|'medium'|'low', description, entity?, frameIds?, suggestion? }`. `HealthIssueType` ∈ `contradiction` / `gap` / `orphan_entity` / `weak_confidence` / `stale_page` / `missing_page`. Detection: missing-page (KG entity with no page that has relations or is person/project), weak_confidence (<2 sources), orphan_entity (no relations at all), stale_page (compiled >30 days ago AND newer frames exist overall).

---

## 17. Wiki Synthesizer Resolution (`synthesizer.ts`)

`resolveSynthesizer(config?) => { synthesize: LLMSynthesizeFn; provider: 'anthropic'|'ollama'|'echo'; model }`. Priority chain:

1. **Anthropic** Haiku (`claude-haiku-4-5-20251001`) — if `ANTHROPIC_API_KEY` / `WAGGLE_ANTHROPIC_API_KEY` present and SDK importable.
2. **Ollama** — if `WAGGLE_OLLAMA_URL` reachable (`/api/tags` health check); model `WAGGLE_OLLAMA_MODEL || llama3.2`.
3. **Echo** fallback — no LLM; returns a structured stub summarizing frame content (so the wiki still renders without a key).

`SynthesizerConfig = { anthropicApiKey?, ollamaUrl?, ollamaModel?, maxTokens?=1500 }`.

---

## 18. Wiki HTTP API (`packages/server/src/local/routes/wiki.ts`)

All routes operate on `server.multiMind.personal` (the personal mind DB). **Critical UX gate:** `/compile` and `/health` return **503 `no_real_embedder`** when `embeddingProvider.getActiveProvider() === 'mock'` — a mock embedder produces zero-vector relevance, so the wiki refuses to compile rather than silently producing broken pages.

| Method | Full path | Request | Response | Stream? |
|---|---|---|---|---|
| `GET` | `/api/wiki/pages` | — | `PageRecord[]` (all pages) | no |
| `GET` | `/api/wiki/pages/:slug` | path | `PageRecord` · 404 | no |
| `GET` | `/api/wiki/pages/:slug/content` | path | `{ slug, markdown }` · 404 | no |
| `POST` | `/api/wiki/compile` | `{ mode?: 'incremental'\|'full', concepts?: string[] }` | `CompilationResult` + `{ llmProvider, llmModel }` · **503** if mock embedder | no |
| `GET` | `/api/wiki/health` | — | `HealthReport` · **503** if mock embedder | no |
| `GET` | `/api/wiki/watermark` | — | `CompilationWatermark` | no |
| `POST` | `/api/wiki/export/obsidian` | `{ outDir }` (absolute) | `ObsidianExportResult { outDir, filesWritten, indexPath, byType }` · 400/409/500 | no |
| `POST` | `/api/wiki/export/notion` | `{ rootPageUrl }` | `NotionExportStats { byType, pagesCreated, pagesUpdated, pagesUnchanged, pagesFailed, errors[] }` · 400/503/409/500. Needs `notion-wiki-token` in Vault | no |

**Obsidian export** (`adapters/obsidian.ts`) writes `{outDir}/_index.md` + `{outDir}/{entity|concept|synthesis}/{slug}.md`, rewriting `[[Display Name]]` → `[[slug|Display Name]]`; no LLM, no state mutation, idempotent. **Notion export** (`adapters/notion.ts`) creates child pages under a root page id (delta-tracked via `notion_page_id` + `content_hash` through `NotionStateHelpers`), converting Markdown to Notion blocks. None of the wiki routes stream.

---

## 19. The Paid-Tier Upgrade Trigger (`packages/shared/src/tiers.ts`)

`TierCapabilities.customSkills` and `connectorLimit` are the levers. Memory, Harvest, and the wiki remain free.

| Tier | `customSkills` | `connectorLimit` | `teamSkillLibrary` | Marketplace install/publish |
|---|---|---|---|---|
| TRIAL | true | -1 (unlimited) | true | allowed (all unlocked 15 days) |
| FREE | **false** | 5 | false | blocked (built-in skills only) |
| PRO | true | -1 | false | allowed (`requireTier('PRO')`) |
| TEAMS | true | -1 | true | allowed |
| ENTERPRISE | true | -1 | true | allowed + enterprise packs |

The marketplace routes enforce this directly: `POST /api/marketplace/install` and `POST /api/marketplace/publish` carry `preHandler: [requireTier('PRO')]`; `GET /api/marketplace/enterprise-packs` carries `requireTier('ENTERPRISE')`. Header copy in `tiers.ts`: *"Skills/connectors are the upgrade trigger."*

---

## 20. Frontend Integration Cheat-Sheet

- **Install Center / Skills UI** → drive from `GET /api/skills/starter-pack/catalog` (state per skill) + `GET /api/skills/capability-packs/catalog`; install via the `POST .../:id` variants; live skills via `GET /api/skills`; preview-without-install via `POST /api/skills/test`; in-context suggestions via `GET /api/skills/suggestions?context=`.
- **Marketplace browser** → `GET /api/marketplace/search` returns packages already annotated with `installed` + `scanStatus` + `scanScore` and the full category list; faceted filters map 1:1 to query params; `GET /api/marketplace/categories` and `/sources` populate filter chips. Show an upgrade modal on `403`/tier errors from `/install`.
- **Security UI** → `scanStatus` per package + `GET /api/marketplace/security-status` for the global banner; `POST /api/marketplace/security-check` for an on-demand scan; findings carry `severity`/`category`/`title`/`description`/`location`.
- **Wiki app** → list with `GET /api/wiki/pages`, body with `/pages/:slug/content`, build with `POST /api/wiki/compile`, quality dashboard with `GET /api/wiki/health` (handle the `503 no_real_embedder` state with an "add an embedding key" prompt), freshness with `GET /api/wiki/watermark`, and the two export buttons.