oleg.m/waggle-os
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
b20b138fe485817e8d69d0c60e376e3c53af149a
waggle-os/docs/specs/WIKI-COMPILER-SPEC.md
Oleg Maslov 0c3e2ead3b
Some checks failed
Installer Smoke / installer-smoke (push) Has been cancelled
Details
moving
2026-09-02 10:10:29 +02:00

71 KiB
Raw Blame History

Wiki Compiler: Waggle x Karpathy Synthesis Engine

Status: Spec Draft Author: Marko Markovic / Egzakta Group Date: 2026-04-13 Codename: Hive Mind Compiler (HMC)

1. Executive Summary

Andrej Karpathy published the LLM Wiki pattern (April 2026) — a system where LLMs incrementally build and maintain a persistent, interlinked markdown wiki from raw sources. Within days it amassed massive community adoption, with GBrain (Garry Tan / YC, 5,400 stars in 24 hours), Rowboat, Basic Memory, and dozens of implementations following.

The thesis of this document: Karpathy's wiki is the READ side of personal knowledge. Waggle's memory engine is the WRITE side. Neither is complete alone. Combined, they create the first full-cycle personal knowledge OS — a system where AI agents automatically accumulate structured memory AND compile it into a human-readable, interlinked, compounding wiki.

Dual-track delivery:

  1. Waggle Feature — Wiki Compiler as a native workspace capability
  2. Open-Source Product — Standalone "Hive Mind" MCP server + CLI

2. Strategic Context

2.1 Why This Matters for Waggle

Waggle's memory engine (FrameStore + HybridSearch + KnowledgeGraph + Identity + Awareness) is technically superior to every competitor in the space. But it has a critical gap: the memory is invisible to the user. Frames in SQLite are powerful for agents, opaque to humans. Users can't see the state of what they know, how ideas connect, what's changed over time.

The wiki compiler layer solves this. It gives users a window into their own knowledge — beautiful, browsable, interlinked markdown pages that update automatically as memory accumulates. This is the emotional hook that turns a technical feature into a product story.

2.2 Why This Matters for KVARK

Karpathy's gist explicitly mentions: "Business/team: an internal wiki maintained by LLMs, fed by Slack threads, meeting transcripts, project documents." This is the KVARK enterprise play:

Your institutional knowledge, automatically compiled from every conversation your people have with AI, into a living wiki that both your agents and your people can read. On your infrastructure, behind your perimeter.

The Teams tier ($49/seat) + shared team memory already exists. The wiki compiler turns it into a visible, browsable institutional knowledge base — the thing enterprise buyers actually want to show their CFO.

2.3 Marketing Hook

The Karpathy signal is enormous. His gist spawned an entire ecosystem in days. Launching an open-source product that explicitly extends his thesis with production-grade infrastructure (real search, knowledge graph, team sharing, multi-workspace) rides this wave with technical credibility.

Positioning: "Karpathy showed you the pattern. We built the engine."

3. Competitive Landscape (April 2026)

Product Architecture Search KG Team Wiki Output Scale
Karpathy's LLM Wiki Flat markdown files index.md (breaks ~500 docs) Backlinks only No Yes (IS the wiki) ~100 sources
GBrain (Garry Tan) Markdown + PGLite/pgvector Hybrid (pgvector + keyword) Entity pages (markdown) No Compiled truth pages 10k+ files
Mem0 Vector + Graph store Semantic + entity links Yes (graph DB) Cloud only No Large
Hindsight Embedded Postgres + pgvector 4-strategy parallel + reranking Entity graph No No Medium
Zep / Graphiti Temporal knowledge graph Entity + temporal queries Yes (Neo4j/FalkorDB) Cloud No Large
Cognee KG + Vector Graph + vector combined Yes (Kuzu) No No Medium
Basic Memory Flat markdown files MCP-based search No No Yes (IS the store) Small
Letta Tiered (core/archival/recall) Agent-managed No No No Medium
Rowboat Typed MD entities + graph Multi-source Yes (typed entities) No Briefings Medium
Waggle Memory SQLite + FTS5 + sqlite-vec Hybrid (BM25 + vector + RRF) Yes (typed entities + relations) Yes (team sync) No Large

Gap Analysis

Nobody has all of:

  • Real hybrid search at scale (BM25 + vector + RRF)
  • Typed knowledge graph with confidence scoring
  • Multi-workspace isolation + cross-workspace search
  • Team memory sharing
  • Multi-source harvest (ChatGPT, Claude, Gemini, etc.)
  • Identity + Awareness layers
  • AND a compiled wiki output layer

Waggle has everything except the wiki output. Adding it creates a category of one.

GBrain: Closest Competitor

GBrain launched 3 days before this spec. It has:

  • Markdown storage + PGLite/pgvector
  • Hybrid search
  • Entity pages (compiled truth + timeline)
  • "Dream cycles" (overnight consolidation)
  • 37 operations, MCP server, CLI, HTTP

But independent code review found that flagship features like compiled truth rewriting and dream cycles are markdown instruction documents that guide AI agents, not executable code. The search is pgvector-only (no FTS5/BM25). No team sharing. No workspace isolation. No temporal knowledge graph. No harvest pipeline. No awareness/identity layers.

Waggle's memory engine is genuinely deeper. The wiki compiler would make that visible.

4. Architecture

4.1 Layer Model

                     +----- HUMAN READS -----+
                     |                        |
Layer 6:  Wiki       |  Interlinked markdown  |  ← NEW: Wiki Compiler
          Output     |  Entity/concept/topic  |
                     |  pages, timelines,     |
                     |  contradictions, gaps   |
                     +------------------------+
                              ↑ compiles from
                     +------------------------+
Layer 5:  Agent      |  Behavioral spec,      |  ← Waggle (exists)
          Intel      |  personas, tool filter  |
                     +------------------------+
                              ↑ uses
                     +------------------------+
Layer 4:  Knowledge  |  Typed entities,       |  ← Waggle KG (exists)
          Graph      |  typed relations,      |
                     |  confidence scores     |
                     +------------------------+
                              ↑ extracted from
                     +------------------------+
Layer 3:  Indexed    |  FTS5 + sqlite-vec     |  ← Waggle HybridSearch
          Memory     |  BM25 + vector + RRF   |     (exists)
                     +------------------------+
                              ↑ indexes
                     +------------------------+
Layer 2:  Raw        |  I/P/B-Frames          |  ← Waggle FrameStore
          Memory     |  in SQLite             |     (exists)
                     +------------------------+
                              ↑ harvested from
                     +------------------------+
Layer 1:  Identity   |  Who you are, what     |  ← Waggle Identity +
          & Context  |  you're doing now      |     Awareness (exists)
                     +------------------------+
                              ↑ derived from
                     +------------------------+
Layer 0:  Raw        |  Conversations, docs,  |  ← Harvest pipeline
          Sources    |  exports, PDFs         |     (exists)
                     +------------------------+

4.2 Wiki Compiler Core

The compiler is a new module that reads from Layers 2-4 and writes Layer 6.

┌─────────────────────────────────────────────────────────┐
│                    WIKI COMPILER                         │
│                                                         │
│  Inputs:                                                │
│    - FrameStore (all frames, with metadata)             │
│    - KnowledgeGraph (entities, relations, confidence)   │
│    - Identity + Awareness (context)                     │
│    - Previous wiki state (for incremental updates)      │
│    - Wiki Schema (compilation rules per workspace)      │
│                                                         │
│  Process:                                               │
│    1. Identify what changed since last compilation      │
│    2. Determine affected pages                          │
│    3. For each affected page:                           │
│       a. Gather relevant frames (via HybridSearch)      │
│       b. Gather related entities (via KG traversal)     │
│       c. Read current page content (if exists)          │
│       d. LLM: synthesize/update the page               │
│       e. LLM: update cross-references                  │
│       f. Write page with frontmatter + citations        │
│    4. Update index.md                                   │
│    5. Update compilation log                            │
│                                                         │
│  Outputs:                                               │
│    - Markdown files (entity pages, concept pages,       │
│      topic summaries, timeline, contradictions)         │
│    - index.md (navigable catalog)                       │
│    - log.md (compilation history)                       │
│    - health.md (gaps, orphans, stale claims)            │
│                                                         │
└─────────────────────────────────────────────────────────┘

4.3 Page Types

Type Content Generated From Update Trigger
Entity page Person, project, org, tech — compiled truth + timeline KG entity + related frames New frame mentioning entity
Concept page Topic synthesis — what we know, sources, evolution Frames matching concept + KG relations New frame on topic
Overview High-level summary of entire workspace knowledge All frames + KG summary Periodic / on-demand
Comparison Side-by-side analysis (e.g., Tool A vs Tool B) Frames mentioning both + KG relations On-demand or lint
Timeline Chronological evolution of knowledge All frames ordered by time Any new frame
Contradictions Claims that conflict across sources KG confidence + frame comparison Lint pass
Gaps Topics mentioned but under-documented KG orphan analysis + frame coverage Lint pass
Index Catalog of all pages with summaries All pages Every compilation

4.4 Page Format

---
type: entity
entity_type: project
name: "Project Alpha"
confidence: 0.87
sources: 14
last_compiled: 2026-04-13T10:30:00Z
frame_ids: [42, 67, 103, 155, 201, ...]
related_entities: ["John Smith", "React", "Q2 Launch"]
---

# Project Alpha

## Summary
[LLM-generated synthesis of all knowledge about this entity]

## Key Facts
- **Started:** 2026-02-15 (from frame #42)
- **Lead:** John Smith (from frame #67, confidence: 0.95)
- **Stack:** React + Node.js (from frames #103, #155)
- **Status:** In development, targeting Q2 launch (from frame #201)

## Timeline
| Date | Event | Source |
|------|-------|--------|
| 2026-02-15 | Project initiated | frame #42 |
| 2026-03-01 | Architecture finalized | frame #67 |
| 2026-03-20 | MVP complete | frame #155 |

## Relations
- [[John Smith]] — project lead (confidence: 0.95)
- [[React]] — primary frontend framework
- [[Q2 Launch]] — target milestone

## Open Questions
- Budget allocation not yet documented
- No test strategy frames found

## Contradictions
- Frame #103 says "TypeScript only" but Frame #155 mentions "some JavaScript"
  → Confidence: 0.6, needs resolution

4.5 Compilation Modes

1. Full Compilation

  • Reads ALL frames + entire KG
  • Generates all pages from scratch
  • Expensive (many LLM calls), used for initial build or reset
  • Token budget: O(frames * avg_frame_size) for context, O(pages) for generation

2. Incremental Compilation

  • Tracks last_compiled_frame_id watermark
  • Only processes frames newer than watermark
  • Determines affected entities/concepts from new frames
  • Updates only affected pages
  • 10-100x cheaper than full compilation

3. Lint Pass

  • No new frames processed
  • Scans existing pages for:
    • Contradictions (frames with conflicting claims on same entity)
    • Stale claims (old frames superseded by newer ones)
    • Orphan pages (no inbound links from other pages)
    • Missing pages (entities in KG with no wiki page)
    • Weak confidence (pages based on single low-confidence frame)
    • Gap suggestions ("you should investigate X")
  • Writes health.md with findings

4. On-Demand Query Compilation

  • User asks a question
  • System searches frames + KG for answer
  • If answer is substantial, offers to compile it as a new page
  • "This comparison could be useful later — save as wiki page?"
  • Karpathy's insight: answers should compound into the wiki

4.6 Wiki Schema (Governance)

Each workspace can have a wiki-schema.md that tells the compiler how to operate — analogous to Karpathy's CLAUDE.md governance file.

# Wiki Schema: Sales Pipeline Workspace

## Page conventions
- Entity pages use "compiled truth + timeline" format
- Concept pages start with a 2-sentence TL;DR
- All claims must cite frame IDs
- Confidence below 0.5 triggers a "Needs Verification" badge

## Entity types to compile
- person (always): full entity pages with relations
- organization (always): entity pages
- deal (always): entity pages with status tracking
- technology (on-demand): only if 3+ frames mention it

## Compilation schedule
- Incremental: after every 10 new frames
- Lint: weekly
- Full: manual only

## Output format
- Obsidian-compatible markdown (wikilinks: [[Page Name]])
- Frontmatter: YAML with type, confidence, sources, frame_ids
- Images: reference from workspace files directory

5. Universal Source Pipeline — The Full Second Brain

5.1 The Gap

Currently Waggle's harvest pipeline only ingests AI conversation exports (ChatGPT, Claude, Gemini JSON). Karpathy's system ingests anything — articles, papers, PDFs, podcast notes, journal entries, web clips, meeting transcripts, book chapters. GBrain adds Gmail, Calendar, and Twilio.

To be a true second brain, the ingest layer must accept any knowledge source a human encounters. The wiki compiler then synthesizes ALL of it — not just what you said to AI, but what you read, heard, watched, and thought.

5.2 Source Adapter Architecture

┌─────────────────────────────────────────────────────────┐
│              UNIVERSAL SOURCE PIPELINE                    │
│                                                         │
│  Every adapter implements:                              │
│    parse(input) → SourceItem[]                          │
│                                                         │
│  SourceItem = {                                         │
│    title: string                                        │
│    content: string          (extracted text, max 4000)  │
│    source: string           (adapter ID)                │
│    sourceUrl?: string       (original URL/path)         │
│    createdAt?: string       (original date if known)    │
│    metadata?: {                                         │
│      entities?: { name, type }[]                        │
│      tags?: string[]                                    │
│      author?: string                                    │
│      contentType: 'article' | 'paper' | 'transcript'   │
│                   | 'note' | 'book' | 'conversation'   │
│                   | 'email' | 'code' | 'image'         │
│    }                                                    │
│  }                                                      │
└─────────────────────────────────────────────────────────┘

5.3 Source Adapters — Full Catalog

Tier 1: Ship with v1 (extends existing harvest)

Adapter Input What It Extracts
chatgpt JSON export Conversations → frames (exists)
claude JSON export Conversations → frames (exists)
claude-code Session JSON Code sessions → frames (exists)
gemini JSON export Conversations → frames (exists)
universal Generic JSON Best-effort extraction (exists)
markdown .md file or folder Sections → frames, headings → entities
plaintext .txt file Paragraphs → frames
pdf .pdf file Pages → text → frames (via pdf-parse)
url Web URL Fetch → readability extract → markdown → frames
obsidian-vault Folder of .md files Bulk import, preserve wikilinks as KG relations

Tier 2: High-value integrations

Adapter Input What It Extracts
epub .epub file Chapters → frames, characters/themes → entities
youtube YouTube URL Transcript → frames (via yt-dlp or API)
podcast Audio URL / file Whisper transcription → frames
email-mbox .mbox export Threads → frames, contacts → entities
notion-export Notion ZIP export Pages → frames, databases → entities
confluence-export Confluence export Pages → frames
slack-export Slack JSON export Channels/threads → frames, users → entities
rss-feed RSS/Atom URL Periodic poll → new articles → frames
zotero Zotero export Papers + notes + annotations → frames
kindle-highlights Kindle export Highlights + notes → frames per book

Tier 3: Live connectors (via MCP or API)

Adapter Input What It Extracts
gmail-mcp Gmail MCP server Emails → frames (real-time via MCP)
calendar-mcp Google Calendar MCP Events → awareness items
slack-mcp Slack MCP server Messages → frames (real-time)
notion-mcp Notion MCP server Page changes → frames (real-time)
github-mcp GitHub MCP server Issues, PRs, discussions → frames
linear-mcp Linear MCP server Issues, projects → frames
meeting-transcript Granola / Fireflies / Otter Meeting notes → frames, action items → awareness
voice-memo Audio file Whisper → text → frames

5.4 Ingest Flow

Raw Source
    │
    ▼
┌──────────┐     ┌───────────┐     ┌──────────────┐
│ Adapter  │────▶│ SourceItem│────▶│ Frame        │
│ .parse() │     │ []        │     │ Store        │
└──────────┘     └───────────┘     └──────┬───────┘
                                          │
                      ┌───────────────────┼───────────────────┐
                      ▼                   ▼                   ▼
               ┌────────────┐    ┌──────────────┐    ┌──────────────┐
               │ HybridSearch│    │ KnowledgeGraph│    │ Wiki Compiler│
               │ .indexFrame()│    │ .createEntity()│    │ (incremental)│
               └────────────┘    └──────────────┘    └──────────────┘

For every source item:

  1. Create I-Frame in FrameStore (with dedup)
  2. Index in HybridSearch (FTS5 + vector)
  3. Extract entities → create/update in KnowledgeGraph
  4. Trigger incremental wiki compilation (if enabled)

5.5 Smart Ingest: LLM-Assisted Entity Extraction

For Tier 1, entity extraction is rule-based (headings, @mentions, etc.). For Tier 2+, the ingest pipeline optionally runs an LLM pass:

Source text → LLM prompt:
  "Extract entities (people, organizations, projects, technologies,
   concepts) and their relationships from this text. Return JSON."
→ Entities + relations → KnowledgeGraph

This is where GEPA pays off — entity extraction is a perfect task for Haiku-class models with optimized prompts. Cost: ~$0.001 per source item.

5.6 Obsidian Vault Import (Killer Onramp)

Karpathy's audience uses Obsidian. Many already have vaults with hundreds of notes. The obsidian-vault adapter is the killer onramp:

hive-mind ingest ~/Documents/MyVault --source obsidian-vault

What it does:

  1. Walks the vault directory
  2. For each .md file:
    • Parse YAML frontmatter → metadata
    • Parse wikilinks [[Page Name]] → KG relations
    • Parse sections → individual frames
    • Parse tags #tag → frame metadata
  3. Preserve the vault's link graph as KG relations
  4. Result: user's existing Obsidian vault becomes searchable, KG-indexed, and ready for wiki compilation

This means: import your existing second brain, get a better one back.

5.7 The Full Second Brain Vision

With the universal source pipeline, the system becomes:

 ┌─── EVERYTHING YOU ENCOUNTER ───────────────────────┐
 │                                                     │
 │  AI conversations (ChatGPT, Claude, Gemini, etc.)  │
 │  Articles you read (web clips, RSS)                 │
 │  Papers you study (PDFs, Zotero)                    │
 │  Books you read (Kindle, epub)                      │
 │  Meetings you attend (transcripts)                  │
 │  Emails you receive (Gmail, mbox)                   │
 │  Code you write (GitHub, Claude Code)               │
 │  Notes you take (Obsidian, Notion)                  │
 │  Podcasts you listen to (transcripts)               │
 │  Slack threads you participate in                   │
 │  Voice memos you record                             │
 │                                                     │
 └──────────────────┬──────────────────────────────────┘
                    │
                    ▼
 ┌─── UNIVERSAL SOURCE PIPELINE ──────────────────────┐
 │  Adapters → SourceItems → Frames → KG → Vectors   │
 └──────────────────┬──────────────────────────────────┘
                    │
                    ▼
 ┌─── YOUR SECOND BRAIN ─────────────────────────────┐
 │                                                     │
 │  FrameStore: every fact, decision, insight          │
 │  KnowledgeGraph: every person, project, concept     │
 │  HybridSearch: find anything instantly              │
 │  Identity: who you are evolves over time            │
 │  Awareness: what you're focused on right now        │
 │                                                     │
 └──────────────────┬──────────────────────────────────┘
                    │
                    ▼
 ┌─── COMPILED WIKI ─────────────────────────────────┐
 │                                                     │
 │  Entity pages: people, projects, orgs you know      │
 │  Concept pages: topics you've explored              │
 │  Timeline: how your knowledge evolved               │
 │  Contradictions: where sources disagree             │
 │  Gaps: what you should investigate next             │
 │  Filed answers: insights you've generated           │
 │                                                     │
 │  Browsable in Obsidian, Waggle UI, or any editor   │
 │                                                     │
 └─────────────────────────────────────────────────────┘

This is Vannevar Bush's Memex (1945), finally realized.

Bush imagined: "a device in which an individual stores all his books, records, and communications, and which is mechanized so that it may be consulted with exceeding speed and flexibility." He called the connections between documents "associative trails."

Karpathy cited Bush explicitly. But Bush couldn't solve who does the maintenance. Karpathy said "the LLM handles that." We say: "the LLM handles that, AND the LLM itself uses it, AND it works across every source you encounter, AND it works for your whole team."

5.8 Source Priority for Implementation

Phase Sources Rationale
Phase 1 Existing 5 AI adapters + markdown + plaintext + pdf + url Covers 80% of Karpathy's use case
Phase 2 obsidian-vault + epub + youtube Killer onramp + high engagement sources
Phase 3 email-mbox + slack-export + notion-export Business/team sources
Phase 4 Live MCP connectors (Gmail, Slack, Notion, GitHub) Real-time second brain

Phase 1 turns hive-mind into a full second brain for researchers. Phase 2 captures the Obsidian/Karpathy community. Phase 3 enables the team/enterprise story. Phase 4 makes it live and always-current.

5.9 Auto-Ingest via MCP Connectors — Zero-Effort Second Brain

Waggle already has 148+ MCP servers in its connector catalog. The insight: these aren't just tools for agents to use — they're live source feeds for the second brain.

Once a user connects their Gmail MCP, Slack MCP, Notion MCP, GitHub MCP, etc., the source pipeline can automatically pull new content on a schedule and ingest it into the memory engine. No manual file drops. No export/import dance. Set up once, then everything flows in.

┌─── CONNECTED MCP SERVERS ──────────────────────────┐
│                                                     │
│  Gmail MCP ─────────────┐                           │
│  Slack MCP ─────────────┤                           │
│  Notion MCP ────────────┤    ┌───────────────────┐  │
│  GitHub MCP ────────────┼───▶│ Auto-Ingest       │  │
│  Linear MCP ────────────┤    │ Scheduler         │  │
│  Google Calendar MCP ───┤    │                   │  │
│  Confluence MCP ────────┤    │ Poll interval:    │  │
│  Fireflies MCP ─────────┘    │ per-source config │  │
│                              └────────┬──────────┘  │
│                                       │             │
└───────────────────────────────────────┼─────────────┘
                                        │
                                        ▼
                              ┌──────────────────┐
                              │ Universal Source  │
                              │ Pipeline         │
                              │                  │
                              │ → Frames         │
                              │ → KG entities    │
                              │ → Vector index   │
                              │ → Wiki compile   │
                              └──────────────────┘

How it works:

MCP Server What Gets Ingested Schedule
Gmail New emails matching filter rules (important, starred, specific labels) Every 30 min
Slack Messages in configured channels, DMs with AI tools Every 15 min
Notion Updated pages in watched databases/workspaces Every 1 hour
GitHub New issues, PR discussions, commit messages in watched repos Every 1 hour
Linear Issue updates, project changes Every 1 hour
Google Calendar Meeting events + linked transcripts Every 30 min
Confluence Updated pages in watched spaces Every 2 hours
Fireflies/Granola New meeting transcripts After each meeting

User configuration (Settings > Auto-Ingest):

auto_ingest:
  gmail:
    enabled: true
    filter: "label:important OR is:starred"
    interval_minutes: 30
  slack:
    enabled: true
    channels: ["#engineering", "#product", "#ai-updates"]
    interval_minutes: 15
  github:
    enabled: true
    repos: ["waggle-os/waggle-os", "egzakta/kvark"]
    types: ["issues", "pull_requests", "discussions"]
    interval_minutes: 60
  notion:
    enabled: true
    databases: ["Product Roadmap", "Meeting Notes"]
    interval_minutes: 60

The result: Your second brain fills itself. You wake up, open Waggle, and your wiki has already been updated with yesterday's emails, Slack threads, meeting transcripts, and GitHub activity. Overnight compilation (dream cycles) synthesized it all into updated entity pages, timeline entries, and gap reports.

Enterprise scaling: For Teams/KVARK, auto-ingest runs server-side. Every team member's connected sources feed into the shared team wiki. Institutional knowledge accumulates automatically — no one has to "maintain the wiki."

5A. EU AI Act Compliance Layer — The Second Brain as Audit Trail

5A.1 The Insight

The EU AI Act (full application: August 2, 2026) requires enterprises to maintain comprehensive audit trails, logging, transparency, and human oversight for AI systems. The penalties: up to EUR 35M or 7% of global revenue.

The second brain IS the compliance artifact. The same system that makes you productive also makes you auditable. Every AI interaction, decision, and source — automatically captured, searchable, citation-linked, and compiled into a browsable wiki that auditors can review.

Nobody else has this. Credo AI ($25K+/yr) and Holistic AI (six figures/yr) are bolt-on governance tools that track policies but not conversations. Claude's new Compliance API captures usage logs but not the knowledge context. Waggle IS the AI workspace AND the governance layer — compliance is native, not bolted on.

5A.2 Article-by-Article Mapping

AI Act Article Requirement How Hive Mind / Waggle Satisfies It
Art. 12: Record-keeping Automatic logging of all events, inputs, outputs over system lifetime FrameStore captures every AI interaction as timestamped frames. HarvestSourceStore tracks all ingestion events. SessionStore groups frames by session. Wiki log.md provides human-readable chronological record.
Art. 13: Transparency Operations sufficiently transparent for deployers to interpret output Compiled wiki IS the transparency layer. Every wiki page cites source frames. KG shows entity relationships. Identity layer shows who the user is. Model attribution per frame (which LLM generated what).
Art. 14: Human oversight Appropriate human-machine interface for effective oversight Wiki = human-readable view of all AI knowledge. Health.md flags contradictions, gaps, stale claims. Approval gates for high-impact actions. Read-only personas prevent unauthorized changes. Lint operation = systematic oversight check.
Art. 19: Log retention Logs retained minimum 6 months FrameStore retains permanently (SQLite). Wiki pages carry full history. Export capability for regulatory submission.
Art. 26: Deployer obligations Monitor operation, keep auto-generated logs, ensure input relevance Auto-ingest captures all inputs. Cost tracking per interaction. Awareness layer tracks active context. Wiki timeline shows evolution over time.
Art. 50: Transparency (GPAI) AI-generated content must be machine-readable as such Every frame tagged with source type (user_stated / tool_verified / agent_inferred / system). Wiki frontmatter identifies compilation as LLM-generated. KG confidence scores distinguish verified from inferred claims.

5A.3 Compliance Wiki Pages (Auto-Generated)

The wiki compiler can generate compliance-specific page types:

Page Type Content AI Act Article
audit-log.md Chronological record of all AI interactions, decisions, model versions Art. 12
model-registry.md Which LLM models were used, when, for what, with what parameters Art. 13, 50
decision-trail.md For each significant decision: what was decided, what frames informed it, what alternatives existed Art. 14
data-provenance.md For each source: origin, ingest date, adapter used, frames generated, entity count Art. 10
risk-assessment.md Per-workspace risk classification (auto-suggested from template type) Art. 9
human-oversight-log.md All approval/denial/override events by human operators Art. 14
contradiction-report.md All detected contradictions between sources, resolution status Art. 13

5A.4 The Compliance Export

For regulatory review, the system can export:

compliance-export-2026-Q2/
├── audit-log.json              # Machine-readable full event log
├── audit-log.pdf               # Human-readable formatted version
├── model-registry.json         # All models used with metadata
├── decision-trails/            # Per-decision evidence packages
│   ├── decision-001.json
│   └── decision-001.pdf
├── data-provenance.json        # Complete source chain
├── risk-assessment.pdf         # Current risk classification
├── human-oversight-log.json    # Approval/deny/override events
├── wiki/                       # Full compiled wiki snapshot
│   ├── index.md
│   ├── entities/
│   ├── concepts/
│   └── compliance/
└── metadata.json               # Export metadata, timestamps, signatures

5A.5 The KVARK Enterprise Pitch

This is where the funnel tightens to KVARK:

"Every AI interaction your organization has — across email, Slack, meetings, code reviews, customer calls — automatically captured in a living knowledge base. Your people get a second brain that makes them productive. Your compliance team gets an audit trail that satisfies the AI Act. Your auditors get a browsable wiki they can actually understand.

And with KVARK, all of this runs on YOUR infrastructure. Your data never leaves your perimeter. Full sovereign deployment. Complete governance."

Market timing: August 2, 2026 deadline. Enterprises are scrambling. AI governance market: $340M (2025) → $492M (2026) → $1B+ (2030), 28% CAGR. Every company using AI in the EU needs this. Nobody else offers productivity

  • compliance in one system.

5A.6 Compliance as a Feature, Not a Tax

The critical UX principle: compliance should be invisible to the user.

The user works normally — chats with AI, reads articles, attends meetings, writes code. The source pipeline captures everything. The wiki compiler synthesizes it. The compliance pages are generated automatically alongside the regular wiki. The user never thinks about compliance.

Only when an auditor asks "show me your AI governance" does the user open the compliance tab and export. Everything is already there.

This is "compliance by default" — the second hook (alongside Memory Harvest) for enterprise adoption.

5B. Privacy Architecture — Local-First, User-Controlled

5B.1 Core Principle

Your second brain is YOUR brain. No cloud. No telemetry. No phone-home. Every byte stays on your machine unless you explicitly choose to share it.

This isn't just a feature — it's the trust foundation. If users don't trust that their emails, Slack DMs, meeting transcripts, and AI conversations are private, they won't connect their sources. Privacy is the prerequisite for the entire system working.

5B.2 Privacy Tiers

Tier Data Location Sharing Encryption Use Case
Personal (default) Local device only (~/.waggle/) Never leaves device Optional (SQLCipher) Free/Pro users
Team Shared Team server (MinIO/S3) Explicitly promoted frames only Encrypted at rest + in transit Teams tier
Enterprise Sovereign Customer infrastructure (KVARK) Governed by RBAC + policies Full encryption + audit KVARK tier

5B.3 Privacy Controls

Per-Source Privacy Settings:

sources:
  gmail:
    privacy: private          # never shared to team wiki
    pii_filter: redact        # strip emails, phones, SSN
    retention: 1y             # auto-delete after 1 year
  slack:
    privacy: team-eligible    # can be promoted to team wiki
    pii_filter: flag          # flag but don't redact
    channels_excluded: ["#random", "#social"]
  meetings:
    privacy: private
    pii_filter: redact
    transcript_retention: 6mo

PII Detection & Filtering on Ingest:

  • Email addresses → [EMAIL]
  • Phone numbers → [PHONE]
  • Credit card numbers → [CC]
  • Social Security / ID numbers → [SSN]
  • API keys / tokens → [SECRET]
  • Passwords → [PASSWORD]
  • Custom patterns (user-defined regex)

Configurable modes:

  1. Redact — replace PII with placeholders before storing
  2. Flag — store but mark as sensitive (excluded from wiki compilation)
  3. Pass-through — store as-is (user accepts risk)
  4. Ask — prompt user on first encounter of each PII type

Frame-Level Privacy:

interface FramePrivacy {
  visibility: 'private' | 'team-eligible' | 'team-shared';
  pii_detected: boolean;
  pii_types?: string[];      // ['email', 'phone', 'api_key']
  redacted: boolean;
  source_privacy: string;     // inherited from source config
  promoted_by?: string;       // who promoted to team (audit trail)
  promoted_at?: string;       // when
}

5B.4 Right to Forget

Users must be able to delete any source and ALL derived data:

Delete source "Gmail import 2026-03-15"
  → Delete all frames from this source
    → Delete all KG entities only sourced from these frames
      → Delete all wiki pages that relied solely on these frames
        → Re-compile affected wiki pages that had mixed sources
          → Update index.md

Cascade delete with source tracking. Every frame knows its source. Every wiki page knows its frames. The chain is traceable and deletable.

5B.5 Data Portability

Everything is open formats:

  • Memory: SQLite database (standard, readable by any tool)
  • Wiki: Markdown files (editable in any editor)
  • KG: JSON export of entities + relations
  • Full export: ZIP of everything, ready to move to another system

No vendor lock-in. This is critical for trust AND for GDPR Art. 20 (right to data portability).

5B.6 GDPR Compliance Mapping

GDPR Article Requirement How We Comply
Art. 5(1)(c) Data minimization Per-source filters, PII redaction, retention limits
Art. 6 Lawful basis User explicitly connects each source (consent)
Art. 13-14 Information provision Transparency dashboard: "What data do I have?"
Art. 15 Right of access Full data export in open formats
Art. 17 Right to erasure Cascade delete per source
Art. 20 Data portability SQLite + Markdown + JSON export
Art. 25 Data protection by design Local-first, no cloud default, encryption option
Art. 32 Security SQLCipher encryption at rest, no network transmission

5B.7 Zero-Telemetry Pledge (Open-Source Product)

For hive-mind (open-source):

  • ZERO telemetry. No analytics. No crash reports. No usage tracking.
  • ZERO network calls except to user-configured LLM API and user-connected MCP servers.
  • ZERO cloud dependencies. Everything runs locally. SQLite, not Postgres.
  • Auditable. Open source. Anyone can verify.

This is the trust moat. When users compare hive-mind to GBrain (PGLite + potential cloud), Mem0 (cloud option), or any SaaS — we're the one they can verify won't leak their data.

5C. Complete Source Coverage + Custom Adapters

5C.1 Full AI System Harvest Coverage

Current adapters only cover 5 systems. The complete list of AI systems people use in 2026 that need harvest adapters:

Tier 1 — Must-Have (ship with v1):

AI System Export Method Adapter Status
ChatGPT JSON export (Settings > Data controls) EXISTS
Claude JSON export (Settings > Account) EXISTS
Claude Code Session JSON EXISTS
Gemini JSON export (Google Takeout) EXISTS
Perplexity GDPR request → JSON, or browser extension scrape NEW
Microsoft Copilot account.microsoft.com/privacy/copilot NEW
GitHub Copilot Chat VS Code chat history (SQLite in extension dir) NEW

Tier 2 — High Value:

AI System Export Method Notes
Cursor SQLite session DB (~/.cursor/) Popular AI code editor
Windsurf Session files (~/.windsurf/) Growing AI code editor
Grok (X) Account data export X/Twitter AI
DeepSeek Chat export Chinese AI, growing Western usage
Mistral (Le Chat) Chat export EU-based AI
NotebookLM Google Takeout Google's research AI
Poe Account export Multi-model AI platform
v0 (Vercel) Session history Design/code AI
Replit AI Workspace export Code generation
Bolt / Lovable Session export Code generation platforms

Tier 3 — Niche but Requested:

AI System Export Method
Character.AI GDPR request
Pi (Inflection) Account export
Cohere Coral API history
Together AI Usage logs
Groq Playground Session history

5C.2 Microsoft Stack — Full Coverage

Microsoft is THE enterprise stack. One microsoft-graph MCP connector covers everything via the unified Graph API:

Service What Gets Ingested Graph API Endpoint
Outlook Emails (filtered by rules) /me/messages
Calendar Events + attendees + notes /me/events
Teams Chats 1:1 and group messages /me/chats/messages
Teams Channels Channel messages in joined teams /teams/{id}/channels/{id}/messages
Teams Meetings Meeting transcripts (if available) /me/onlineMeetings/{id}/transcripts
OneDrive Document content (Word, Excel, PDF) /me/drive/items
SharePoint Site pages, document libraries /sites/{id}/pages
OneNote Notebooks, sections, pages /me/onenote/pages
Planner Tasks, plans, buckets /me/planner/tasks
To Do Task lists, tasks /me/todo/lists

Authentication: OAuth 2.0 via Microsoft identity platform. User grants delegated permissions. No admin consent required for personal data. Enterprise admin can pre-authorize via Entra ID for team-wide deployment.

Implementation: Use the existing microsoft-365 MCP server from our catalog, or the official Anthropic M365 connector. Both use Microsoft Graph. Wire as an auto-ingest source with per-service toggle and filter rules.

5C.3 Manual Source Input

Not everything comes from a connector. Users need frictionless manual input:

In Waggle Desktop UI:

  • Quick Add (keyboard shortcut: Ctrl+Shift+N): Floating input field. Type or paste anything. Auto-detects: URL → fetch and ingest, file path → read and ingest, plain text → save as frame.
  • File Drop Zone: Drag & drop files onto any workspace. Supports: PDF, markdown, text, Word (.docx), images (OCR via LLM vision), audio (Whisper transcription).
  • Clip from Browser: Browser extension that sends current page to Waggle as a source. One click. Like Obsidian Web Clipper but for your second brain.
  • Share Target: Register as a system share target on Windows/macOS. Share from any app → ingests into active workspace.

In CLI:

# Ingest a file
hive-mind add paper.pdf
hive-mind add meeting-notes.md
hive-mind add ~/Documents/report.docx

# Ingest a URL
hive-mind add https://arxiv.org/abs/2401.12345

# Ingest pasted text
echo "Key decision: we're going with React for the frontend" | hive-mind add -

# Ingest a whole directory
hive-mind add ./research-papers/ --recursive

# Ingest with metadata
hive-mind add paper.pdf --tags "ml,transformers" --importance critical

In MCP (for AI agents):

Tool: ingest_source
  content: "string or file path or URL"
  type_hint: "pdf" | "url" | "markdown" | "text" | "auto"
  importance: "critical" | "important" | "normal"
  tags: ["optional", "tags"]
  workspace: "optional workspace ID"

5C.4 Custom Adapters — Extensible by Design

Users and the community must be able to add their own source adapters:

Adapter Interface:

interface SourceAdapter {
  /** Unique adapter ID */
  id: string;

  /** Human-readable name */
  displayName: string;

  /** Supported input types */
  accepts: ('file' | 'url' | 'json' | 'text' | 'directory')[];

  /** File extensions this adapter handles (e.g., ['.pdf', '.epub']) */
  extensions?: string[];

  /** Parse input into source items */
  parse(input: unknown, options?: AdapterOptions): SourceItem[];

  /** Optional: detect if this adapter can handle the input */
  canHandle?(input: unknown): boolean;
}

interface AdapterOptions {
  /** Maximum content length per item */
  maxContentLength?: number;
  /** Extract entities via LLM */
  extractEntities?: boolean;
  /** PII filtering mode */
  piiFilter?: 'redact' | 'flag' | 'pass' | 'ask';
  /** Custom metadata to attach */
  metadata?: Record<string, unknown>;
}

Custom Adapter Registration:

# Register a custom adapter from a JS/TS file
hive-mind adapters add ./my-custom-adapter.ts

# List registered adapters
hive-mind adapters list

# Remove a custom adapter
hive-mind adapters remove my-custom-id

Community Adapter Registry:

  • Published as npm packages: hive-mind-adapter-*
  • Install: npm install hive-mind-adapter-jira
  • Auto-discovered on startup
  • Community-maintained, reviewed for security

5C.5 Custom Automations

Beyond adapters, users need custom automation triggers:

Webhook Ingest:

# Start local webhook server
hive-mind webhooks start --port 9876

# POST from any tool (Zapier, n8n, Make, custom scripts)
curl -X POST http://localhost:9876/ingest \
  -H "Content-Type: application/json" \
  -d '{"content": "Meeting decision: launch Q3", "source": "custom"}'

File Watch:

# Watch a directory for new files → auto-ingest
automations:
  - type: file-watch
    path: ~/Downloads/research/
    pattern: "*.pdf"
    adapter: pdf
    workspace: research-project
    on_ingest: compile  # auto-compile wiki after ingest

Schedule:

# Run on a schedule
automations:
  - type: cron
    schedule: "0 6 * * *"        # every day at 6 AM
    action: compile               # compile wiki
  - type: cron
    schedule: "0 22 * * *"       # every night at 10 PM
    action: lint                  # health check
  - type: cron
    schedule: "*/30 * * * *"     # every 30 minutes
    action: ingest-mcp            # pull from connected MCP sources

5C.6 UX — Killer Easy Setup

The entire source configuration must be accessible to non-technical users.

Onboarding Flow (Waggle Desktop):

Step 1: "Welcome to your Second Brain"
  → Brief 10-second animation showing the concept

Step 2: "Connect your AI tools" (checkboxes)
  [✓] ChatGPT    → "Upload your export (Settings > Data controls > Export)"
  [✓] Claude     → "Upload your export (Settings > Account > Export)"
  [ ] Perplexity → "Request your data (Settings > Privacy > Request data)"
  [ ] Copilot    → "Download from account.microsoft.com/privacy"
  [Skip for now]

Step 3: "Connect your work tools" (OAuth one-click)
  [Connect Gmail]      → OAuth popup → done
  [Connect Slack]      → OAuth popup → done
  [Connect Notion]     → OAuth popup → done
  [Connect Microsoft 365] → OAuth popup → done
  [Skip for now]

Step 4: "Import existing knowledge" (optional)
  [Import Obsidian Vault] → folder picker → done
  [Import Files]          → drag & drop zone → done
  [Skip for now]

Step 5: "Privacy settings"
  ○ Maximum privacy (all PII redacted, no team sharing)
  ○ Balanced (PII flagged, team sharing opt-in per source)
  ○ Open (no filtering, full team sharing)
  [Customize per source] → advanced settings

Step 6: "Your second brain is ready"
  → First compilation starts
  → "Check back in 10 minutes for your first wiki"

CLI Setup:

$ hive-mind init

  Welcome to Hive Mind — your second brain engine.

  ? Choose a schema template:
    ❯ Personal (journal, articles, podcasts, self-improvement)
      Research (papers, articles, notes, thesis development)
      Business (meetings, emails, projects, team knowledge)
      Custom (start from scratch)

  ? Set privacy level:
    ❯ Maximum (all PII redacted, local only)
      Balanced (PII flagged, sharing opt-in)
      Open (no filtering)

  ? Connect sources now? (you can add more later)
    [y/N]: y

  Initialized in ~/.hive-mind/
  Run 'hive-mind add <file-or-url>' to start ingesting.
  Run 'hive-mind compile' to build your wiki.

6. Dual-Track Delivery Plan

Track A: Waggle Feature (packages/wiki-compiler)

The wiki compiler as a native Waggle workspace capability.

Integration points:

  • New tab in MemoryApp: "Wiki" — browse compiled pages
  • New agent tool: compile_wiki — trigger compilation
  • New agent tool: search_wiki — search compiled pages
  • Settings: compilation schedule, schema editing
  • FilesApp: wiki pages appear as browsable markdown
  • Workspace export: "Export as Obsidian vault" (wiki + raw sources)

Technical implementation:

  • New package: packages/wiki-compiler/
  • Depends on: @waggle/core (FrameStore, HybridSearch, KnowledgeGraph)
  • Output: markdown files in ~/.waggle/workspaces/{id}/wiki/
  • LLM calls: via the existing agent infrastructure (orchestrator)
  • Compilation state: SQLite table tracking watermarks, page hashes

Waggle-specific advantages:

  • Per-workspace wikis with workspace-specific schemas
  • Cross-workspace wiki search (leveraging MultiMindCache)
  • Team wiki (shared team memory → compiled team wiki)
  • Identity-aware compilation ("compile with context of who I am")
  • GEPA optimization for cheaper compilation with smaller models

Track B: Open-Source Product (hive-mind-mcp)

A standalone MCP server + CLI that implements the full wiki compiler pattern independently of Waggle's desktop app.

Product name: hive-mind (working title)

Positioning: "Karpathy showed you the pattern. GBrain gave you markdown files. Hive Mind gives you the engine — real search, knowledge graph, compiled wiki, team sharing. Open source."

Architecture:

hive-mind/
├── packages/
│   ├── core/          # Memory engine (extracted from @waggle/core)
│   │   ├── frame-store.ts
│   │   ├── hybrid-search.ts
│   │   ├── knowledge-graph.ts
│   │   ├── identity.ts
│   │   ├── awareness.ts
│   │   └── session-store.ts
│   ├── compiler/      # Wiki compiler (NEW)
│   │   ├── compiler.ts        # Core compilation loop
│   │   ├── page-generators/   # Per-type page generators
│   │   ├── linker.ts          # Cross-reference resolver
│   │   ├── linter.ts          # Health check / contradiction finder
│   │   └── schema-parser.ts   # Wiki schema governance
│   ├── harvest/       # Source adapters
│   │   ├── chatgpt.ts
│   │   ├── claude.ts
│   │   ├── gemini.ts
│   │   └── universal.ts
│   ├── mcp-server/    # MCP server (stdio transport)
│   │   ├── tools/     # 20+ MCP tools
│   │   └── resources/ # MCP resources
│   └── cli/           # CLI interface
│       ├── ingest.ts
│       ├── compile.ts
│       ├── search.ts
│       ├── lint.ts
│       └── export.ts
├── schemas/           # Example wiki schemas for different domains
│   ├── research.md
│   ├── business.md
│   ├── personal.md
│   └── reading.md
├── docs/
└── README.md

MCP Tools (exposed to Claude Code / Claude Desktop / any MCP client):

Category Tool Description
Memory save_memory Save a fact/decision/preference
Memory recall_memory Hybrid search across all memory
Knowledge search_entities Search knowledge graph
Knowledge save_entity Create/update entity
Knowledge create_relation Link entities
Wiki compile_wiki Trigger incremental compilation
Wiki compile_full Full wiki rebuild
Wiki compile_page Compile a single page
Wiki lint_wiki Run health check
Wiki search_wiki Search compiled pages
Wiki get_page Read a compiled wiki page
Wiki file_answer Save a query answer as a wiki page
Harvest harvest_import Import from AI systems
Harvest harvest_sources List registered sources
Identity get_identity / set_identity User profile
Awareness get_awareness / set_awareness Current context
Workspace list_workspaces / create_workspace Manage workspaces
Export export_obsidian Export as Obsidian vault

CLI Interface:

# Initialize a new brain
hive-mind init --schema research

# Ingest a source
hive-mind ingest paper.pdf --source research
hive-mind ingest chatgpt-export.json --source chatgpt

# Compile the wiki
hive-mind compile              # incremental
hive-mind compile --full       # full rebuild
hive-mind compile --page "Project Alpha"  # single page

# Search
hive-mind search "what do we know about React performance"
hive-mind search --entities "John Smith"

# Health check
hive-mind lint
hive-mind lint --fix  # auto-fix what's possible

# Export
hive-mind export --format obsidian --output ./my-vault

Why open source?

  1. Rides the Karpathy wave with genuine technical depth
  2. Community adoption → contributors → ecosystem
  3. Every hive-mind user is a potential Waggle convert (needs desktop app for full experience) or KVARK lead (needs team/enterprise features)
  4. MIT license, same as GBrain — no friction
  5. Memory engine quality speaks for itself vs. GBrain's markdown instructions

7. Revised Execution Plan (post-evaluation, April 13)

Key insight from live test: The wiki compiler isn't the bottleneck — data quality is. personal.mind has 80% test noise, the KG has 80% misclassified entities, and only 8 relations across 2,734 entities. Building a compiler on noisy data produces noisy pages. Fix the input first.

v1 — BUILD (3-4 sessions)

Session A: Data Foundation

  • Wipe personal.mind test pollution (E2E/benchmark frames + garbage entities)
  • KG cleanup: delete entities where name is common noun + type is "person"
  • KG dedup: merge entities with same normalized name + type
  • Improve Claude Code harvest adapter: extract DECISIONS and DIRECTIONS from user messages in session JSONL, not just plans/rules
  • Add markdown + pdf + url source adapters to harvest pipeline
  • Add ingest_source MCP tool

Session B: Core Compiler

  • Create packages/wiki-compiler/ with types + interfaces
  • Compilation state tracking (SQLite: watermarks, page hashes)
  • compileEntityPage() — entity → gather frames + KG → LLM synthesis → markdown
  • compileConceptPage() — topic → search → LLM synthesis → markdown
  • compileSynthesisPage() — cross-source pattern detection (the killer feature)
  • compileIndex() — navigable catalog with summaries
  • Incremental compilation (only new frames since watermark)
  • compileHealth() — contradictions, gaps, orphans, data quality

Session C: MCP + Wire

  • Add wiki tools to memory-mcp: compile_wiki, search_wiki, get_page, lint_wiki, file_answer
  • Add wiki resources: memory://wiki/index, memory://wiki/page/{name}
  • Wire compilation to LLM (Haiku default, Ollama fallback)
  • Test: "compile my wiki" → "search wiki for X" → browse pages

v1 — TEST (the real validation)

Test Protocol — Marko's Real Second Brain

PHASE 1: Clean Slate
  - Wipe all E2E test data from personal.mind
  - Verify: 0 frames, 0 entities, clean database
  - Run Waggle OS normally for real work (not testing)
  - After the build session(s), personal.mind will have REAL frames
    from actual Waggle usage during development

PHASE 2: Harvest All AI Accounts (Marko's real data)

  Source                  Export Method                          Est. Data
  ─────────────────────── ────────────────────────────────────── ──────────
  Claude Code sessions    JSONL files (~200MB, 20+ sessions,     HIGH
                          2-3 months of history)
  Claude.ai (claude.com)  Settings > Account > Export data        HIGH
  Claude Cowork           Export from Cowork interface             MEDIUM
  ChatGPT                 Settings > Data controls > Export        MEDIUM
  Perplexity              Settings > Privacy > Request data        MEDIUM
                          (GDPR export → JSON)
  X.AI (Grok)             Account data export                     LOW-MED
  Google Gemini            Google Takeout                          MEDIUM
  Google NotebookLM        Google Takeout                          LOW-MED
  Genspark                 Account export / GDPR request           LOW

  Total estimated: 500-2000 frames of REAL knowledge

PHASE 3: Microsoft Graph (real business data)

  Connect Marko's Microsoft 365 via Graph API:
  - Outlook: filtered to important/starred emails
  - Teams: key channels only (not noise)
  - Calendar: meetings with notes
  - OneDrive/SharePoint: key documents
  - Apply PII filtering: redact mode for emails, flag for Teams

PHASE 4: Compile + Evaluate

  - Run full wiki compilation on all harvested data
  - Generate entity pages, concept pages, synthesis pages
  - Generate health report
  - Evaluate:
    [ ] Does the Waggle OS page match reality?
    [ ] Does Marko's page capture real working patterns?
    [ ] Do cross-session synthesis pages reveal new insights?
    [ ] Are entity relations meaningful (not noise)?
    [ ] Is the health report actionable?
    [ ] Would Marko read this wiki daily?
    [ ] Hit rate target: 80%+ pages are genuinely useful

PHASE 5: Iterate

  - Fix issues found in evaluation
  - Re-compile
  - If hit rate >= 80%: proceed to v2
  - If hit rate < 80%: diagnose and fix before continuing

v2 — SCALE (post-test, estimated 4-6 sessions)

Only proceed after v1 test validates the output quality.

v2a: Waggle UI

  • New "Wiki" tab in MemoryApp (markdown renderer + wikilink navigation)
  • Compile button + progress indicator
  • Health dashboard
  • "Open in Obsidian" export button

v2b: Additional Harvest Adapters

  • Perplexity adapter (GDPR JSON export)
  • Microsoft Copilot adapter
  • GitHub Copilot Chat adapter (VS Code SQLite)
  • Cursor / Windsurf session adapters
  • Grok (X.AI) adapter
  • Genspark adapter

v2c: Auto-Ingest via MCP

  • Microsoft Graph auto-ingest (Outlook, Teams, Calendar)
  • Gmail MCP auto-ingest
  • Slack MCP auto-ingest
  • Configurable poll intervals + filter rules
  • Privacy controls: per-source PII filtering, default OFF

v2d: Open-Source Package

  • Extract into standalone hive-mind repo
  • CLI interface
  • README with Karpathy attribution
  • npm publish

v2e: Compliance Layer

  • Auto-generated compliance pages (audit-log, model-registry, decision-trail)
  • Compliance export package (JSON + PDF)
  • Wire to Waggle Teams/KVARK tier

8. Technical Decisions

7.1 LLM for Compilation

The compiler needs an LLM to synthesize frames into wiki pages. Options:

Approach Pros Cons
Via Waggle orchestrator Reuses existing infrastructure, cost tracking, model routing Coupled to Waggle
Direct API call Simple, standalone-friendly Need own key management
GEPA-optimized Haiku/Sonnet quality at fraction of Opus cost Needs GEPA implementation
Local (Ollama) Free, private, offline Slower, lower quality

Decision: Support all four. Default to direct API call for standalone, orchestrator for Waggle integration. GEPA as optimization layer. Ollama as free fallback.

7.2 Compilation Cost Model

Rough estimates for a workspace with 500 frames, 50 entities:

Operation LLM Calls Input Tokens Output Tokens Cost (Haiku)
Full compile (50 pages) 50 ~500K ~100K ~$0.15
Incremental (5 pages) 5 ~50K ~10K ~$0.015
Lint pass 10 ~200K ~20K ~$0.04
File answer as page 1 ~10K ~2K ~$0.003

With GEPA (Haiku + prompt optimization): 3-5x cheaper than raw Opus. This makes compilation economically viable even for free-tier users if we use Haiku with optimized prompts.

7.3 Citation Chain

Every claim in a wiki page MUST cite its source frame(s):

- **Status:** In development (frame #201, 2026-03-20)

Frame IDs in frontmatter frame_ids array enable:

  • Click-through from wiki page → original frame
  • Confidence calculation (more frames = higher confidence)
  • Staleness detection (oldest frame age)
  • Contradiction detection (conflicting frames on same claim)

7.4 Wikilink Format

Use Obsidian-compatible wikilinks: [[Page Name]]

Cross-references generated automatically by the linker:

  1. Scan page content for known entity names
  2. Insert wikilinks where entities are mentioned
  3. Maintain backlink index (which pages link to which)

7.5 Storage Location

~/.waggle/workspaces/{id}/
├── workspace.mind          # SQLite (frames, KG, search index)
├── workspace.json          # Workspace config
├── wiki/                   # Compiled wiki output (NEW)
│   ├── index.md
│   ├── log.md
│   ├── health.md
│   ├── entities/
│   │   ├── john-smith.md
│   │   ├── project-alpha.md
│   │   └── ...
│   ├── concepts/
│   │   ├── react-performance.md
│   │   └── ...
│   ├── topics/
│   │   ├── q2-launch-plan.md
│   │   └── ...
│   └── queries/            # Filed answers
│       ├── comparison-react-vs-vue.md
│       └── ...
├── files/                  # Virtual filesystem
└── sessions/

9. Differentiation Matrix

vs. Karpathy's LLM Wiki

Dimension Karpathy Hive Mind
Search index.md (breaks ~500 docs) FTS5 + sqlite-vec + RRF (10K+ docs)
Knowledge structure Backlinks between pages Typed KG with confidence scores
Source input Manual (Obsidian Web Clipper) Auto-harvest from 5+ AI systems
Multi-topic One wiki per folder Multi-workspace with isolation
Team No Team memory sync + team wiki
Identity No Identity + Awareness layers
Temporal Pages updated in place Frame timestamps + temporal queries
Scale ceiling ~100-500 sources Thousands (SQLite + vector index)
Output Wiki IS the system Wiki is a compiled VIEW of deeper memory

vs. GBrain

Dimension GBrain Hive Mind
Storage PGLite + markdown files SQLite + FTS5 + sqlite-vec
Search pgvector semantic only Hybrid (BM25 + vector + RRF)
KG Entity pages (markdown) Typed entities + relations + confidence
Compilation Markdown instructions for agents Executable code with real compiler loop
Dream cycles Prompt instructions Scheduled compilation with state tracking
Team No Team memory sync
Harvest Gmail, Calendar integrations ChatGPT, Claude, Gemini, universal
Workspace isolation No Multi-workspace with cross-search
Identity No Identity + Awareness layers
Infrastructure Requires Postgres (or PGLite WASM) Zero external deps (embedded SQLite)

vs. Mem0 / Zep / Hindsight

Dimension Memory Frameworks Hive Mind
Memory storage Yes (various backends) Yes (SQLite, embedded)
Search Yes (various strategies) Yes (hybrid BM25 + vector + RRF)
Knowledge graph Some (Mem0, Zep) Yes (typed, with confidence)
Wiki output No Yes
Human-readable No Yes
Compounding synthesis No Yes
Harvest pipeline No Yes (5+ AI systems)
Team sharing Mem0 Cloud only Built-in (team sync)

The key differentiator: no existing memory framework produces a human-readable compiled wiki. They all store memory for agents to use. None compile it into something humans can browse, learn from, and build on.

10. Open-Source Launch Strategy

9.1 Naming

Working title: hive-mind npm package: hive-mind-mcp GitHub: egzakta/hive-mind (or waggle-os/hive-mind)

Alternative names to consider:

  • mind-compiler
  • wiki-mind
  • memory-wiki
  • second-brain-engine

9.2 Positioning

Tagline options:

  • "Karpathy showed you the pattern. We built the engine."
  • "Your memory, compiled."
  • "The memory engine behind the wiki."
  • "From conversations to knowledge. Automatically."

README lead:

Hive Mind is an open-source memory engine + wiki compiler for AI agents. It extends Karpathy's LLM Wiki pattern with production-grade infrastructure: hybrid search (BM25 + vector + RRF), typed knowledge graph with confidence scoring, multi-source harvest (ChatGPT, Claude, Gemini), and automatic wiki compilation.

Works as an MCP server (Claude Code, Claude Desktop), CLI tool, or library. Your memory stays local. Your wiki stays current.

9.3 Launch Sequence

  1. Pre-launch (1 week before):

    • Ship working MCP server + CLI with at least compile, search, lint
    • Record 2-minute demo video: ingest → compile → browse in Obsidian
    • Write blog post: "Extending Karpathy's LLM Wiki with a Real Engine"

  2. Launch day:

    • GitHub repo public (MIT license)
    • npm publish
    • X thread (tag Karpathy, reference his gist, show the extension)
    • Hacker News post
    • r/LocalLLaMA, r/ClaudeAI, r/ObsidianMD posts

  3. Post-launch (week 1-2):

    • Product Hunt
    • Dev.to article
    • YouTube walkthrough
    • Community Discord

9.4 Funnel to Waggle / KVARK

Open-source hive-mind (free, MIT)
  → "Want a desktop app?" → Waggle Free tier
    → "Want unlimited agents?" → Waggle Pro ($19/mo)
      → "Want team knowledge?" → Waggle Teams ($49/seat)
        → "Want enterprise governance?" → KVARK (consultative sale)

The open-source product is the top of the funnel. Every hive-mind user who wants a GUI, team features, or enterprise governance flows toward Waggle/KVARK.

11. Risk Register

Risk Impact Mitigation
Compilation cost too high Users hit budget limits GEPA optimization, Haiku default, incremental-only
Hallucinated synthesis Wiki contains false claims Citation chains, confidence scores, lint passes
Wiki staleness Pages drift from reality Automated incremental compilation on frame count triggers
GBrain captures the market first Reduced differentiation window Ship faster, emphasize technical depth (real search, real KG, real code vs. markdown instructions)
Karpathy buzz fades Smaller launch impact Build product value independent of the reference — the wiki compiler is genuinely useful
LLM quality variance Inconsistent wiki quality Schema governance + GEPA + model routing
Token context limits Can't fit all frames for large entities Chunked compilation with summarization tiers

12. Success Metrics

Open-Source (hive-mind)

  • GitHub stars: 1,000 in first week (GBrain did 5,400 in 24h — aspirational)
  • npm downloads: 500/week within first month
  • MCP server installs: tracked via Claude Code marketplace
  • Community PRs: at least 5 within first month

Waggle Feature

  • Wiki compilation triggered by 30%+ of active workspace users
  • Average wiki size: 20+ pages per active workspace
  • User retention uplift: measurable increase in DAU after wiki feature ships

KVARK Pipeline

  • At least 3 enterprise inquiries attributable to wiki/hive-mind within first quarter
  • "Institutional knowledge wiki" becomes a slide in KVARK sales deck

13. Open Questions

  1. Should the wiki replace FilesApp or live alongside it? Current thinking: alongside — wiki is a compiled view, files are user-managed documents.

  2. Should compilation happen on the client or server? For Waggle desktop: client (sidecar). For Teams: server-side (team wiki needs central compilation).

  3. How to handle conflicting compilations in team wikis? If two team members' agents compile simultaneously, need merge strategy. Git-style (markdown is mergeable) or lock-based?

  4. Should the wiki be editable by humans? Karpathy says no — "the LLM owns the wiki." But users will want to correct things. Allow human edits with a "manually edited" flag that the compiler preserves?

  5. What's the minimum viable wiki? For launch: entity pages + index + lint. Concept pages, timelines, comparisons can follow.

  6. GEPA readiness? The GEPA prompt optimization system (backlog item) would dramatically reduce compilation cost. Should we prioritize GEPA before or after the wiki compiler?

References

Spec by Marko Markovic / Egzakta Group. April 2026. This document is a strategic planning artifact, not a commitment to ship all features described.

Reference in New Issue View Git Blame Copy Permalink