oleg.m/waggle-os

Fork 0

Files

Oleg Maslov 0c3e2ead3b

Installer Smoke / installer-smoke (push) Has been cancelled

Details

moving

2026-09-02 10:10:29 +02:00

26 KiB

Raw Permalink Blame History

Claude Code Source — Deep Architecture Analysis for Waggle OS

Date: 2026-04-01 Analyst: Cowork Session (CxO Advisory) Source: Full source tree at D:\Projects\Claude Code Source\src\src (~600+ files) Purpose: Extract production-grade architectural patterns for Waggle OS maturation

1. AGENT ORCHESTRATION — The Complete Picture

1.1 Agent Definition Schema

Every agent in Claude Code — built-in or custom — follows a single schema defined in loadAgentsDir.ts. This is the most immediately adoptable pattern for Waggle.

BaseAgentDefinition fields:

agentType        — unique identifier (e.g., "Explore", "Plan", "general-purpose")
whenToUse        — natural language description used in the tool prompt
tools?           — allowlist of tool names this agent can use
disallowedTools? — denylist of tool names (complementary to allowlist)
skills?          — skill names to preload when agent starts
mcpServers?      — MCP servers specific to this agent (by name reference or inline config)
hooks?           — session-scoped hook settings (pre/post tool execution)
color?           — display color for UI identification
model?           — model override ("sonnet", "opus", "haiku", or "inherit")
effort?          — reasoning effort level
permissionMode?  — "default", "plan", etc.
maxTurns?        — hard cap on agentic turns before forced stop
memory?          — persistent memory scope: "user" | "project" | "local"
background?      — always run as background task
isolation?       — "worktree" (git worktree) or "remote" (sandbox)
initialPrompt?   — prepended to first user turn
omitClaudeMd?    — skip CLAUDE.md injection for lightweight agents

Three agent sources with priority override:

Built-in (code-defined): generalPurposeAgent, exploreAgent, planAgent, verificationAgent, claudeCodeGuideAgent, statuslineSetup
Custom (markdown files with YAML frontmatter): loaded from .claude/agents/ directories across user, project, and policy settings
Plugin (external packages): loaded via plugin system with plugin metadata

Priority resolution: managed > flag > project > user > plugin > built-in. Later sources override earlier ones by agentType name.

Waggle implication: Your 13 agent personas should be formalized into this exact schema. The separation between built-in (your core agents), custom (user-defined in workspace), and plugin (marketplace agents) maps directly to your three-tier user model: simple users get built-in, power users define custom, admins manage plugins.

1.2 Agent Tool Pool Assembly

Each agent gets a custom-assembled tool set via assembleToolPool():

Start with ALL available tools
If agent has tools allowlist → filter to only those tools
If agent has disallowedTools denylist → remove those tools
If agent has memory enabled → inject FileWrite, FileEdit, FileRead tools automatically
If agent has mcpServers → connect those MCP servers and inject their tools
If agent has skills → preload those skill invocations
Apply permission mode restrictions on top

This is compositional security — an "Explore" agent literally cannot write files because FileWrite is not in its tool pool. Not enforced by prompt instructions, but by schema.

1.3 Coordinator Mode (Feature-Gated)

The coordinator is the most strategically valuable pattern for Waggle's Mission Control.

How it works:

Activated via CLAUDE_CODE_COORDINATOR_MODE=1
Coordinator gets ONLY: Agent, SendMessage, TaskStop, SyntheticOutput tools
Cannot directly read files, write code, or execute commands
ALL execution is delegated to "worker" subagents
Workers report back via <task-notification> XML messages

Coordinator workflow:

User request arrives
Coordinator spawns research workers in parallel (fan-out)
Workers report findings as task notifications
Coordinator SYNTHESIZES findings (this is the key insight — the coordinator must understand before delegating implementation)
Coordinator spawns implementation workers with precise specs
Implementation workers self-verify (first QA layer)
Coordinator spawns verification workers (second QA layer)
Coordinator reports results to user

Critical design principle: "Never delegate understanding." The coordinator's prompt explicitly forbids phrases like "based on your findings, fix the bug" — it must synthesize findings into specific instructions with file paths, line numbers, and exact changes.

Scratchpad pattern: Workers share durable cross-worker knowledge via a scratchpadDir — a designated directory where workers can read/write without permission prompts. This enables coordination through shared state on the filesystem.

Waggle implication: This is exactly what Mission Control should become. The Cockpit view shows agent status; Mission Control orchestrates. The scratchpad pattern maps to your existing workspace file system. The worker notification pattern (<task-notification> XML) should be adopted for your event stream.

1.4 Fork Subagent Pattern

Forks solve a specific problem: context pollution during research.

Without forks: Agent researches a question → all intermediate tool calls (file reads, greps, etc.) fill context → context is now cluttered with noise that isn't needed for the next task.

With forks: Agent spawns a fork → fork inherits the FULL parent conversation context → fork does research → fork returns a summary → parent's context stays clean.

Key rules:

Forks share the parent's prompt cache (cheap to spawn)
Don't set a different model on a fork (breaks cache sharing)
Don't "peek" at fork output files — wait for the completion notification
Fork prompts are directives, not full context briefs (context is inherited)
For fresh agents (subagent_type specified), write full context briefs

Continue vs. Spawn decision matrix:

Situation	Action	Why
Research found exactly the files to edit	Continue (SendMessage)	Worker has files in context
Research was broad, implementation is narrow	Spawn fresh	Avoid dragging exploration noise
Correcting a failure	Continue	Worker has error context
Verifying another worker's code	Spawn fresh	Fresh eyes, no implementation assumptions
Wrong approach entirely	Spawn fresh	Avoid anchoring on failed path

Waggle implication: This directly solves the "13 agents generating output" context management problem. Waggle Dance workflows should use the fork pattern for research phases and fresh spawns for implementation phases.

1.5 Agent Memory (Per-Agent Persistent State)

Each agent can have its own persistent memory, separate from the main conversation memory.

Three scopes:

user (~/.claude/agent-memory/<agentType>/): persists across all projects
project (.claude/agent-memory/<agentType>/): project-specific, version-controlled
local (.claude/agent-memory-local/<agentType>/): project-specific, NOT version-controlled

Memory snapshots: Teams can share agent memory via project-level snapshots. On first run, if a project snapshot exists but no local memory, the snapshot is copied to local. If a newer snapshot exists, the agent is notified to consider updating.

When memory is enabled for an agent, FileWrite/FileEdit/FileRead tools are automatically injected even if the agent has a restricted tool allowlist.

Waggle implication: Your graph database memory should support per-agent memory scopes. An "Analysis Agent" should remember different things than a "Writing Agent." The snapshot/sync pattern is essential for team workspaces.

2. MEMORY SYSTEM — Production-Grade Patterns

2.1 Four-Type Taxonomy (Enforced)

Types: user, feedback, project, reference

Each type has structured guidance:

user: role, goals, preferences. Always private scope. Saved when learning about who the user is.
feedback: behavioral corrections AND confirmations. "Record from failure AND success" — only saving corrections makes the system overly cautious. Structure: rule → Why → How to apply.
project: ongoing work context not derivable from code/git. Convert relative dates to absolute. Decay fast, so include "why" for judging staleness.
reference: pointers to external systems (Linear project, Grafana dashboard, Slack channel). Explicit exclusions (even on user request):
Code patterns, architecture, file paths — derivable from current project state
Git history — git log/git blame are authoritative
Debugging solutions — the fix is in the code
Anything in CLAUDE.md files
Ephemeral task details

If the user asks to save a PR list, the system asks "what was surprising or non-obvious about it?" — only that part gets saved.

2.2 Side-Query Relevance (The Key Innovation)

findRelevantMemories.ts uses a lightweight Sonnet call to pre-filter memories.

Flow:

scanMemoryFiles() reads YAML frontmatter from all .md files in the memory directory (max 200 files, first 30 lines per file — no full content loaded)
Headers are formatted as a manifest: - [type] filename (ISO timestamp): description
A Sonnet side-query gets: the user's current query + the manifest + list of recently-used tools
Sonnet returns up to 5 filenames that are clearly relevant
Only those 5 files are fully loaded into context

Anti-noise feature: If tools are currently in use (e.g., MCP spawn tool), their reference docs are excluded — the system is already exercising them, reference docs are noise. But warnings/gotchas about those tools ARE included.

Prefetch pattern: Memory relevance is prefetched at query start (startRelevantMemoryPrefetch), runs during model streaming, and is consumed after tools execute. Never blocks the main path.

2.3 Memory Freshness and Staleness

memoryAge.ts computes human-readable age ("47 days ago") and injects staleness warnings:

"This memory is 47 days old. Memories are point-in-time observations, not live state — claims about code behavior or file:line citations may be outdated. Verify against current code before asserting as fact." The "Before recommending from memory" section enforces verification:

Memory names a file path → check file exists
Memory names a function → grep for it
User about to act on recommendation → verify first
"The memory says X exists" ≠ "X exists now"

2.4 Team Memory

Shared memory across team members with enterprise-grade security:

Symlink traversal protection (realpath resolution)
Null byte injection prevention
URL-encoded traversal detection
Unicode normalization attack prevention
Backslash injection rejection
Path containment verification against real filesystem paths

Waggle implication: Your enterprise air-gapped deployment requires this level of path security for shared memory. The team memory pattern directly maps to your multi-workspace isolation — each workspace gets its own team memory directory.

3. QUERY ENGINE — The Brain

3.1 Architecture

QueryEngine class: one instance per conversation, persists across turns.

Per-turn lifecycle:

Context assembly: system prompt + userContext + systemContext (composable sections)
Memory prefetch: starts async Sonnet side-query for relevant memories
Skill prefetch: starts async skill discovery for potentially relevant skills
Snip compact: removes old context if history is too long
Microcompact: fine-grained compaction of tool results
Context collapse: coarser compaction via staged collapses
Autocompact: full conversation summarization when context is critically full8. Tool result budget: enforces per-message size limits on aggregate tool results
API call: streams model response
Streaming tool execution: tools begin executing as soon as their parameters stream in (not after full response)
Post-sampling hooks: execute after model completes
Token budget check: decide whether to continue or stop the agentic loop

3.2 Multi-Layer Compaction Strategy

This is critical for long sessions. Claude Code uses FOUR layers:

Snip compact: removes oldest messages, preserving a "protected tail" of recent context
Microcompact: replaces individual tool results with summaries when they exceed size limits. Cached variant edits the API cache directly.
Context collapse: groups related messages into collapsible summaries. Projection-based — the REPL keeps full history, collapse is a read-time view.
Autocompact: full conversation summary when nearing context limit. Triggered by token count threshold. Uses a side-query to generate the summary.

Each layer runs independently. Snip runs before microcompact. Microcompact before autocompact. Context collapse can prevent autocompact from firing.

3.3 Token Budget Management

tokenBudget.ts implements a continuation system:

Tracks continuationCount, lastDeltaTokens, lastGlobalTurnTokens
Continues if under 90% of budget AND not showing diminishing returns
Diminishing returns detected: 3+ continuations with <500 token delta
Agents don't get continuation — only the main loop
Blocking limit: hard stop when token count reaches critical threshold (reserves space for manual /compact)

3.4 Streaming Tool Execution

StreamingToolExecutor: tools begin executing while the model is still generating.

As soon as a tool_use block's parameters are complete in the stream, execution begins. Multiple tools can execute concurrently. Results are yielded as they complete, not in order. This dramatically reduces latency for multi-tool turns.

Waggle implication: This is a significant UX advantage. When an agent spawns three research sub-tasks, all three should begin immediately, not sequentially.

4. TOOL HARNESS — The Execution Layer

4.1 buildTool() Factory

Every tool in the system is created via buildTool<InputSchema, OutputSchema, Progress>():

type ToolDef = {
  name: string
  description: string | (() => string | Promise<string>)
  inputSchema: z.ZodType | (() => z.ZodType)  // Zod schema, can be lazy
  outputSchema?: z.ZodType | (() => z.ZodType)
  isEnabled: () => boolean                      // Dynamic enable/disable
  isReadOnly: () => boolean                     // Read-only tools skip permission checks
  maxResultSizeChars?: number                   // Tool result truncation limit
  userFacingName?: () => string                 // Display name for UI

  call: (input, context) => AsyncGenerator<Progress, Output>

  // React rendering
  renderToolUseMessage: (input) => ReactNode
  renderToolResultMessage: (output) => ReactNode
  renderToolUseRejectedMessage: (input) => ReactNode
  renderToolUseErrorMessage: (error) => ReactNode
  renderToolUseProgressMessage: (progress) => ReactNode

  // Validation
  validateInput?: (input) => ValidationResult
  backfillObservableInput?: (input) => void     // Enrich input for SDK/transcript
}

ToolUseContext provides rich execution context:

options.tools: full tool pool available
options.mcpClients: connected MCP servers
options.agentDefinitions: all agent definitions
abortController: cancellation signal
readFileState: file content cache
getAppState() / setAppState(): application state access
messages: conversation history
queryTracking: chain ID and depth for analytics

4.2 Permission System

Multi-layer permission model:

Permission mode: "default", "plan", "bypass"
Per-tool rules by source: alwaysAllowRules, alwaysDenyRules, alwaysAskRules
Sources: user settings, project settings, policy settings (enterprise)
Background agents: shouldAvoidPermissionPrompts — auto-deny when no UI available
Coordinator workers: awaitAutomatedChecksBeforeDialog — run classifier checks before showing prompt
Denial tracking: prevents repeated prompts for the same tool/input combination

4.3 Deferred Tool Loading (ToolSearch)

Tools are registered by name only. Full schema (description, inputSchema, outputSchema) is fetched on demand via ToolSearchTool. This keeps the system prompt compact when many MCP servers are connected.

Waggle implication: As your connector count grows past 32, this becomes essential. Each connected MCP server adds tool schemas to the prompt. Deferred loading keeps the base prompt stable and cache-friendly.

5. SKILLS FRAMEWORK

5.1 Skill Definition (Markdown with Frontmatter)

Skills are markdown files loaded from directories:

---
name: commit
description: Create a git commit with conventional commit message
tools: Agent, Bash, FileRead, FileEdit     # Tool allowlist when invoked
model: sonnet                              # Optional model override
effort: high                               # Reasoning effort
paths: src/**, tests/**                    # File path relevance patterns
hooks:                                     # Lifecycle hooks
  pre_tool_call: ...
  post_tool_call: ...
allowed_tools: Read, Edit                  # Tools the skill itself uses
args: message                              # Named arguments ($message substitution)
---

[Skill prompt content in markdown]

5.2 Skill Sources and Priority

Bundled (shipped with Claude Code)
MCP-generated (via mcpSkillBuilders.ts — MCP servers can register skill builders)
User (~/.claude/skills/)
Project (.claude/skills/)
Managed (enterprise policy)
Plugin (external packages)

Deduplication by canonical file path (resolves symlinks). Priority: managed > project > user > plugin > bundled.

5.3 Skill Discovery and Prefetch

Skills can be discovered at query time via a prefetch mechanism that runs alongside model streaming. The system identifies potentially relevant skills based on the user's query before the model requests them.

Waggle implication: Your "Skills & Apps" dock view should adopt this exact structure. Skills as markdown files is simple, composable, and human-editable. The frontmatter schema provides everything needed for display, filtering, and execution.

6. TASK SYSTEM

6.1 Seven Task Types

LocalAgentTask       — foreground subagent (sync or backgroundable)
RemoteAgentTask      — remote sandbox execution (always background)
InProcessTeammateTask — tmux-spawned teammate process
LocalShellTask       — background shell commands
LocalWorkflowTask    — multi-step workflow execution
MonitorMcpTask       — MCP server monitoring
DreamTask            — background "dreaming" (speculative processing)

6.2 Task Lifecycle

States: pending → running → completed|failed|killed

Background task indicator: shown when status === 'running' | 'pending' AND isBackgrounded === true.

Progress tracking: agents report via AgentToolProgress events with description, tokenCount, toolUseCount, lastToolName.

Auto-background: After 120 seconds, foreground agents can be automatically moved to background (feature-gated via CLAUDE_AUTO_BACKGROUND_TASKS).

6.3 Notification Pattern

Workers → coordinator communication uses structured XML:

<task-notification>
  <task-id>{agentId}</task-id>
  <status>completed|failed|killed</status>
  <summary>{human-readable status}</summary>
  <result>{agent's final text response}</result>
  <usage>
    <total_tokens>N</total_tokens>
    <tool_uses>N</tool_uses>
    <duration_ms>N</duration_ms>
  </usage>
</task-notification>

Waggle implication: This notification format should be adopted for your event stream. The structured XML enables parsing, display, and routing in your Cockpit and Events views.

7. ARCHITECTURE BLUEPRINT FOR WAGGLE

Based on the complete deep-dive, here is the prioritized implementation roadmap:

P0 — Foundation (Blocks Everything Else)

A. Agent Definition Schema Formalize the 13 agent personas into the BaseAgentDefinition format. Every agent gets: agentType, whenToUse, tools allowlist, disallowedTools, model, maxTurns, memory scope. Store these as markdown files with YAML frontmatter in the workspace.

B. Tool Pool Assembly Implement assembleToolPool() — per-agent tool filtering based on allowlist/denylist. This is the single most important security boundary in the system.

C. Token Budget Management Build a budget tracker for the chat view. Track continuation count, delta tokens, and implement the 90% threshold with diminishing-returns detection.

P1 — Orchestration (Enables Workflows)

D. Coordinator Mode for Mission Control Implement the coordinator pattern: restricted tool set (only Agent + SendMessage + TaskStop), worker spawn/track/notify lifecycle, and scratchpad for cross-worker state.

E. Task Notification Protocol Adopt the <task-notification> XML format for agent→orchestrator communication. Wire this into the Events view for real-time workflow visualization.

F. Fork Subagent for Waggle Dance Implement context-inheriting agent spawns for research phases. The fork's context stays clean, the parent gets only the summary.

P2 — Memory Maturation

G. Four-Type Memory Taxonomy Implement on top of your graph DB. Each memory node gets: type (user/feedback/project/reference), description, timestamp. Enforce the "what NOT to save" rules.

H. Side-Query Relevance Filter Use a lightweight model call to pre-filter which memory frames to inject per query. Scan frontmatter/descriptions only, select up to 5, load full content for only those.

I. Memory Freshness Warnings Compute age, inject staleness caveats for memories >1 day old, enforce "verify before recommending."

J. Per-Agent Memory Scopes Each agent type gets its own memory directory/subgraph. Analysis Agent remembers analysis patterns. Writing Agent remembers style preferences. Separate from conversation memory.

P3 — Context Management

K. Multi-Layer Compaction Implement at least two layers: microcompact (per-tool-result summarization) and autocompact (full conversation summary). This is non-negotiable for production quality with long agent sessions.

L. Streaming Tool Execution Begin tool execution as soon as parameters stream in. This alone can cut perceived latency by 30-50% for multi-tool turns.

M. Deferred Tool Loading Register connector tools by name only. Fetch full schemas on demand via a ToolSearch equivalent. Critical as connector count scales.

P4 — Enterprise

N. Team Memory with Security Shared memory across workspace users. Symlink protection, path traversal prevention, scope isolation between private and team.

O. Agent Memory Snapshots Allow teams to share agent memory configurations via project-level snapshots. New team members get bootstrapped agent memory automatically.

P. Permission Model Per-tool, per-agent, per-source permission rules. Enterprise policy settings override project settings override user settings. Background agents auto-deny permission prompts.

8. WHAT WAGGLE HAS THAT CLAUDE CODE DOESN'T

Do not lose sight of these advantages:

Graph-based memory — relationships between memory nodes enable queries that flat files cannot. "What does this user know about X and how does it relate to Y?" is a graph query, not a file scan.
Visual orchestration — the Cockpit, Waggle Dance, and Events views provide visibility into agent workflows that a terminal fundamentally cannot deliver. This is the enterprise sales differentiator.
Multi-workspace isolation — Claude Code is one project per session. Waggle can run multiple isolated workspaces simultaneously with independent agent pools and memory spaces.
Desktop-first distribution — one-click install via Tauri, not npm install -g. Dramatically lower barrier to entry for non-technical users.
Three-tier user model — Claude Code has one mode. Waggle can surface progressive complexity. Simple users see a chat interface. Power users see agent configuration. Admins see the full orchestration layer.
Encrypted vault — AES-256-GCM for credentials. Claude Code stores secrets in plaintext config files.

These are not incremental features — they are structural advantages that justify Waggle's existence as a distinct product.

CORRECTION: Section 2 Memory Assessment (Updated)

The original Section 2 incorrectly positioned Claude Code's flat-file memory as a benchmark that Waggle's graph DB should aspire to. The corrected assessment:

Waggle's SQLite + frames + graph architecture is the more mature design. It is enterprise-appropriate for a product with multi-workspace isolation, team memory, per-agent scopes, and importance-based pruning. Claude Code's flat markdown files are a pragmatic CLI choice, not a superior one.

What Waggle Has That Claude Code Cannot Match

Capability	Waggle (Graph DB)	Claude Code (Flat Files)
Relationship queries	Native graph traversal	Impossible
Concurrent access	Database-level locking	Filesystem conflicts
Importance scoring	accessCount + metadata	Not available
Per-agent memory scopes	Graph partitions	Directory convention
Cross-agent knowledge sharing	Shared nodes with edges	Shared files (no relationships)
Multimedia frame origins	Frames from any source type	Text-only markdown
Team memory security	DB-level + API auth	Filesystem symlink protection

Patterns to Adopt FROM Claude Code (as graph-layer enhancements)

Side-query relevance filter — Use a lightweight model call against frame metadata to select top-5 frames per query. Highest-value single improvement.
Staleness warnings — Surface frame age in context injection. "This frame is 47 days old — verify before acting on it."
Verify-before-recommend — When a frame references a file path or function name, check existence before surfacing.
Four-type taxonomy overlay — Classify frames as user/feedback/project/reference for consistent retrieval patterns.
Explicit exclusion rules — Prevent memory bloat by refusing to store code patterns, git history, or debugging solutions as frames.

Implementation Fixes Required (Bugs, Not Architecture)

P2: Unify memory counting (49 vs 37 discrepancy across UI views)
P3: Fix accessCount incrementing on frame access
P3: Render markdown content in frame display
New: Add side-query relevance pre-filtering before context injection

26 KiB Raw Permalink Blame History