oleg.m/waggle-os
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
0c3e2ead3beaffba9d5004ef7fd87d871d5e6886
waggle-os/docs/memory-architecture.md
Oleg Maslov 0c3e2ead3b
Some checks failed
Installer Smoke / installer-smoke (push) Has been cancelled
Details
moving
2026-09-02 10:10:29 +02:00

23 KiB
Raw Blame History

Memory Architecture — Waggle OS

Audience: Internal engineers and contributors touching packages/hive-mind-core/src/mind/ or the agent recall path. Scope: How memory is stored, searched, scored, and recalled — grounded in the source, not in CLAUDE.md. Status: Current as of April 15, 2026. Schema version 1.

⚠️ SUPERSEDED / partially stale (2026-06-26). This doc predates the April 2026 monorepo migration: the substrate moved from packages/hive-mind-core/src/mind/ to packages/hive-mind-core/src/mind/, and several line-of-code counts here are out of date. For the current, code-verified architecture see docs/transfer/MEMORY-SYSTEM-EXPLAINED.md. This file is retained for historical reference.

Path discrepancy fixed: An earlier revision of CLAUDE.md §2 listed memory-layer files (frames.ts, awareness.ts, etc.) at the top level of packages/core/src/. They actually live in packages/hive-mind-core/src/mind/. Corrected in the commit that promoted this doc.

1. Ten-second summary

A single SQLite database per mind (better-sqlite3 + sqlite-vec, WAL mode, FK on) holds every memory layer. One file per mind, one schema, loaded via MindDB. Five cooperating layers sit on top of that database:

  1. Identity — single-row table. Who this mind is.
  2. Awareness — ≤10 rolling items. What this mind is working on right now.
  3. Frames — append-only event log. Everything this mind has learned, as I/P/B frames grouped by GOP.
  4. Knowledge Graph — entities and relations with temporal validity. The structured distillation of the frame log.
  5. Hybrid Search — FTS5 + vec0 fused via Reciprocal Rank Fusion, re-ranked by a configurable scoring profile.

Additional tables in the same DB: sessions (GOP → project mapping), improvement signals, install audit, procedures (GEPA-optimized templates), AI interactions (EU AI Act Art. 12), execution traces, evolution runs, harvest sources. Those are subsystems; the five above are the memory substrate.

Agents consume this through one function: Orchestrator.recallMemory(query, limit) — invoked on every user turn, fused across personal and workspace minds.

2. Storage substrate: MindDB

File: packages/hive-mind-core/src/mind/db.ts

export class MindDB {
  constructor(dbPath: string) {
    this.db = new Database(dbPath);
    this.db.pragma('journal_mode = WAL');
    this.db.pragma('foreign_keys = ON');
    const vecPath = process.env.WAGGLE_SQLITE_VEC_PATH;
    if (vecPath) this.db.loadExtension(vecPath);
    else sqliteVec.load(this.db);
    this.initSchema();
  }
}

Three facts to remember:

  • WAL mode is on. Readers don't block writers. Don't turn it off.
  • sqlite-vec is loaded either from WAGGLE_SQLITE_VEC_PATH or the npm package. The env-var override is the desktop-build path — Tauri bundles the native .dll/.dylib and the sidecar points at it. If you see "no such module: vec0" at runtime, the extension didn't load — check that env var.
  • Schema init is idempotent and runs migrations. initSchema checks for the meta table. If absent: fresh DB, run SCHEMA_SQL + VEC_TABLE_SQL, write schema_version = '1'. If present: run runMigrations() (currently only adds the source column to memory_frames for pre-W2.1 databases).

Every layer class takes a MindDB and calls db.getDatabase() to get the raw better-sqlite3 handle. There is no ORM. Every query is hand-written prepared SQL. That's a feature — it's a deliberate ~500 LOC of surface area, not 50k.

3. Schema at a glance

File: packages/hive-mind-core/src/mind/schema.ts — one SCHEMA_SQL string for the whole mind, plus a separate VEC_TABLE_SQL for the virtual table.

Table Role Layer
meta schema_version infra
identity single-row CHECK (id = 1) 0 — Identity
awareness ≤10 active items, category-constrained 1 — Awareness
sessions gop_id ↔ project_id, status infra
memory_frames I/P/B, gop_id, t, importance, source, access_count 2 — Frames
memory_frames_fts FTS5 shadow, content_rowid='id', porter+unicode61 2 — Frames
memory_frames_vec vec0(embedding float[1024]) virtual 2 — Frames
knowledge_entities typed nodes with valid_from/valid_to 3 — KG
knowledge_relations typed edges, confidence, temporal 3 — KG
procedures GEPA prompt templates 4 — Procedures
improvement_signals recurring patterns, (category, pattern_key) unique 5
install_audit capability install trust trail 6
harvest_sources import sync state 8
ai_interactions EU AI Act Art. 12 event log 7
execution_traces agent run history, outcome, trace_json 9
evolution_runs self-evolution proposals/outcomes 10

Embedding dimensionality is pinned at 1024, baked into the vec0 DDL. If you change it, you must rebuild every vec table and re-embed every frame. Don't change it without a migration.

4. Layer 0 — Identity

File: packages/hive-mind-core/src/mind/identity.ts — 73 LOC.

CREATE TABLE identity (
  id INTEGER PRIMARY KEY CHECK (id = 1),
  name, role, department, personality, capabilities, system_prompt,
  created_at, updated_at
);

One row. Primary key forced to 1 via CHECK. IdentityLayer exposes create, get, exists, update, and toContext(). The toContext() output is pasted directly into the system prompt under # Identity (see §8).

Everything but name defaults to empty string. A bare name is enough to have an identity. Token budget target: <500 tokens (per schema comment).

5. Layer 1 — Awareness

File: packages/hive-mind-core/src/mind/awareness.ts — 170 LOC.

Working memory. Volatile, small, priority-ordered.

CREATE TABLE awareness (
  id, category CHECK IN ('task','action','pending','flag'),
  content, priority, metadata, created_at, expires_at
);

Hard cap: MAX_ITEMS = 10. getAll() and getByCategory() both apply LIMIT 10. expires_at is honored in every read query (expires_at IS NULL OR expires_at > datetime('now')).

Two things worth knowing:

  • ensureMetadataColumn() runs in the constructor — PRAGMA-inspects the table and runs ALTER TABLE ... ADD COLUMN metadata if it's missing. This is the graceful-upgrade path for pre-metadata databases. If you ever see the metadata column missing on a new DB, something went wrong in schema.ts.
  • getByStatus(status) is an in-memory filter over parsed metadata JSON. Each awareness item stores its own JSON blob — there's no index on status. Fine because MAX_ITEMS = 10.

toContext() groups active items by category (Active Tasks, Recent Actions, Pending Items, Context Flags) — this is what gets surfaced in the agent's self-awareness section of the system prompt.

6. Layer 2 — Frames

File: packages/hive-mind-core/src/mind/frames.ts — 377 LOC. This is the heart.

6.1 Data model

CREATE TABLE memory_frames (
  id, frame_type CHECK IN ('I','P','B'),
  gop_id, t, base_frame_id REFERENCES memory_frames(id),
  content, importance CHECK IN ('critical','important','normal','temporary','deprecated'),
  source CHECK IN ('user_stated','tool_verified','agent_inferred','import','system'),
  access_count, created_at, last_accessed
);

Three frame types:

  • I-frame (Index) — full-content snapshot. Standalone. No base.
  • P-frame (Patch) — incremental update referencing an I-frame via base_frame_id. Requires the base to reconstruct full state.
  • B-frame (Branch/reference) — alternative to a frame without replacing it. Points at another frame via base_frame_id.

gop_id is the Group-of-Pictures identifier — a session/conversation/topic bucket. t is the monotonically-increasing sequence number within a GOP. The (gop_id, t) index is the primary scan path for reconstructing a GOP's history.

6.2 Frame lifecycle

Create, read, update, delete, reconstruct, compact. The three creates:

  • createIFrame(gopId, content, opts) — INSERT with frame_type='I' and base_frame_id=null. Assigns t = MAX(t)+1 for the GOP.
  • createPFrame(gopId, baseFrameId, content, opts) — asserts the base is an I-frame in the same GOP before inserting.
  • createBFrame(gopId, referencedFrameId, content, opts) — branch/alternative on any base.

Every insert triggers FTS indexing (INSERT INTO memory_frames_fts(rowid, content)) and — if an embedder is configured — vector indexing via HybridSearch.indexFrame. Vector indexing is wrapped in try/catch because memory_frames_vec may not exist in older DBs.

6.3 Reconstruction

reconstructState(gopId) returns a ReconstructedState:

{ iFrame: MemoryFrame, patches: MemoryFrame[], currentContent: string, frameCount: number }

Logic:

  1. Find the most recent I-frame in the GOP.
  2. Pull every P-frame with base_frame_id = iFrame.id, ordered by t.
  3. Apply patches in order (current implementation concatenates — see §6.6 "Known rough edges").

6.4 Compaction

compact({ deleteTempOlderThan = 30, deleteDeprecatedOlderThan = 90, compactPFramesWhen = 10 }):

  • Delete temporary frames older than 30 days.
  • Delete deprecated frames older than 90 days.
  • Per GOP: if P-frames on the current I-frame ≥ 10, merge them into a new I-frame and delete all but the 5 most recent P-frames. (The recent tail is kept so branches still have something to diff against.)

This is designed to be run nightly or on-demand. It's not automatic.

6.5 Deduplication

findDuplicate(content) computes SHA-256(content.trim()) and checks against the last 500 frames. Returns the frame ID if found, null otherwise. Callers should use this before createIFrame on import paths (Memory Harvest does).

Trim-stable hash means trailing/leading whitespace won't spawn duplicates. Case-sensitive, punctuation-sensitive. If you want semantic dedup, that's what vector search and contradiction-detector.ts are for — don't touch this.

6.6 Known rough edges

  • Patch application is string concatenation, not a real diff. P-frames say "here's more" rather than "here's what changed at line N." Fine for the current usage (each P-frame is a new fact on top of the index), but it means you can't use P-frames to retract or correct — use update() on the I-frame or mark deprecated.
  • Vector indexing uses rowid-literal SQL (see §7.2) because sqlite-vec cannot parameterize rowid. If you add a new insert path, remember to mirror the FTS + vec writes.
  • Access count updates are best-effort — they happen on read paths, not inside a transaction. Under concurrent readers the count can drift; that's acceptable because it only feeds a log-scaled popularity score.

7. Layer 3 — Hybrid Search

File: packages/hive-mind-core/src/mind/search.ts — 255 LOC.

7.1 Top-level shape

class HybridSearch {
  async search(query, opts?: SearchOptions): Promise<SearchResult[]>
  async keywordSearch(query, opts?): Promise<...>
  async vectorSearch(query, opts?): Promise<...>
  async indexFrame(frame): Promise<void>
  async indexFramesBatch(frames): Promise<void>
}

SearchOptions: limit, gopId, profile (scoring profile — see §7.3), context (graph distances from the KG), since, until.

SearchResult: { frame, rrfScore, relevanceScore, finalScore }.

7.2 The fusion

keyword results (FTS5)  ─┐
                         ├─►  RRF (K=60)  ─►  re-rank via computeRelevance  ─►  finalScore = rrfScore * relevanceScore
vector results (vec0)   ─┘

const RRF_K = 60 at the top of the file — the canonical RRF constant, well-attested in the IR literature. Don't tune it without a reason.

Keyword path:

  • Strip a hardcoded ~60-word stop-word list ("the", "is", "of", …) from the query.
  • Build an OR-joined MATCH expression (term1 OR term2 OR …). We do not require all terms — memory queries are too short and ad-hoc for AND-matching to be useful.
  • SELECT rank FROM memory_frames_fts MATCH ? ORDER BY rank LIMIT ?.
  • Join back to memory_frames to hydrate the row.

Vector path:

  • Embed the query via the injected Embedder.
  • SELECT rowid, distance FROM memory_frames_vec WHERE embedding MATCH ? ORDER BY distance LIMIT ?.
  • Hydrate from memory_frames by rowid.

Both paths respect gopId, since, until filters.

RRF: for each frame appearing in either result list, rrfScore += 1 / (RRF_K + rank). This is the rank-fusion line — don't reimplement it elsewhere.

7.3 Scoring profiles

File: packages/hive-mind-core/src/mind/scoring.ts

SCORING_PROFILES = {
  balanced:  { temporal: 0.4, popularity: 0.2, contextual: 0.2, importance: 0.2 },
  recent:    { temporal: 0.6, popularity: 0.1, contextual: 0.2, importance: 0.1 },
  important: { temporal: 0.1, popularity: 0.1, contextual: 0.2, importance: 0.6 },
  connected: { temporal: 0.1, popularity: 0.1, contextual: 0.6, importance: 0.2 },
};

Four signals, weighted:

  • Temporal: full score for frames accessed in the last 7 days; exponential decay with 30-day half-life afterward.
  • Popularity: 1 + log10(1 + access_count) * 0.1. Log-scaled so a frame read 1000 times only beats a fresh one by ~0.3.
  • Contextual: if the caller passed graphDistances, frames at BFS distance 0/1/2/3 from the query-relevant entities get scores 1.0/0.7/0.4/0.2. Otherwise 0.
  • Importance: critical=2.0, important=1.5, normal=1.0, temporary=0.7, deprecated=0.3 (these are the IMPORTANCE_WEIGHTS multipliers you'll see referenced across the codebase).

finalScore = rrfScore * computeRelevance(frame, weights, context). Multiplicative on purpose — a frame with a rotten RRF score can't be saved by being important, and an important frame still needs to show up in one of the two index paths.

7.4 Indexing contract

Every memory_frames row should have a matching row in memory_frames_fts (rowid = frame id) and memory_frames_vec (rowid = frame id). If you write directly to memory_frames and skip HybridSearch.indexFrame, that frame is invisible to search. Don't do that.

indexFramesBatch exists for imports — uses a single transaction + embedBatch for throughput.

8. Layer 4 — Knowledge Graph

File: packages/hive-mind-core/src/mind/knowledge.ts — 265 LOC.

knowledge_entities(id, entity_type, name, properties, valid_from, valid_to, recorded_at)
knowledge_relations(id, source_id, target_id, relation_type, confidence, properties, valid_from, valid_to, recorded_at)

Two departures from a vanilla graph DB matter:

8.1 Bitemporal validity

valid_from and valid_to are the "when this was true" dimension. recorded_at is the "when we wrote it down" dimension. A fact that was true last month and is no longer true now sets valid_to to last month's date. Reads default to current ("valid_to IS NULL OR valid_to > now"). Historical queries can override.

retireEntity(id) and retireRelation(id) set valid_to = datetime('now'). We soft-delete. Hard-delete is only for schema violations.

8.2 Schema validation

ValidationSchema defines required properties and allowed relation types per entity type. Violations throw at insert time. This is the main reason entities are typed rather than free-form: catch bad data at write time, not at query time.

8.3 Traversal

traverse(startId, depth) is BFS, returns visited entity set. bfsDistances(frameAnchoredEntityIds) is the one that feeds HybridSearch's contextual score — returns Map<frameId, distance> for the closest entity.

9. How the agent actually uses it

File: packages/agent/src/orchestrator.ts

9.1 System prompt assembly

Orchestrator.buildSystemPrompt() stitches three sections:

  1. Identity (cached) — this.identity.toContext() under # Identity. Cache key is 'exists'|'empty'.
  2. Self-awareness (uncached) — buildSelfAwareness(caps) with tools, skills, memory stats, mode, version, and any actionable improvementSignals.
  3. Recent context (uncached) — loadRecentContext() output under # Context From Your Memory.

The identity section is cached because it changes rarely. The other two are rebuilt every turn.

9.2 Recall path

Orchestrator.recallMemory(query, limit = 10) runs on every user turn.

Two modes:

  • Catch-up mode — triggered by a regex match on the query ("catch me up", "where were we", "what did we decide", "brief me", "what's next", etc.). Bypasses semantic search. Goes straight to memory_frames and pulls:
    • up to limit frames with importance IN ('critical','important') OR content LIKE 'Decision%' OR content LIKE '%decided%', ordered by importance then recency,
    • plus the 3 most recent non-deprecated, non-temporary frames for recency context,
    • deduplicated by first 100 chars of content. This is the right move: when a user says "catch me up," they want the important things, not the semantically-closest things. Literal text similarity will return recent small talk over last week's critical decision every time.
  • Normal mode — this.search.search(query, { limit, profile: 'balanced' }) on both the personal mind and the workspace mind (if present), fused into one result list with source attribution.

The returned { text, count, recalled? } gets injected into the conversation as additional context before the model sees it.

9.3 Per-workspace minds

An agent instance can hold one personal mind + one workspace mind. Each is a full MindDB + layer stack. The workspace mind is scoped to a workspace; the personal mind is scoped to the user. recallMemory queries both. Frames written during a workspace session land in the workspace mind.

This is why the codebase has FrameStore and HybridSearch instantiated twice on the Orchestrator — once in the constructor for the personal mind, and again in setWorkspace for the workspace mind.

10. Embeddings

Files: packages/hive-mind-core/src/mind/embedding-provider.ts, embeddings.ts, api-embedder.ts, inprocess-embedder.ts, litellm-embedder.ts, ollama-embedder.ts.

10.1 The fallback chain

createEmbeddingProvider({ provider: 'auto' }) probes in order:

inprocess  →  ollama  →  voyage  →  openai  →  mock

Each probe embeds the literal string 'waggle embedding probe', checks the dim count matches the target (1024), and either returns an active embedder or logs and moves on. The first one that works wins. If all fail, you get createMockEmbedder(1024) — a deterministic hash-to-floats stub that keeps the app running with degraded semantic search quality.

Mock mode is not silent. getStatus().activeProvider === 'mock' with lastError populated. If you're debugging weird search results, check that first.

10.2 Tier enforcement

TIER_CAPABILITIES[tier].embeddingProviders lists what a tier can use. Free gets inprocess+ollama (local only). Pro gets voyage+openai. Enterprise gets litellm. An explicitly-requested provider outside the tier throws TierError. In auto mode, disallowed providers are skipped.

10.3 Quota

Monthly embed count tracked in the embedding_usage table (user_id, year_month, count). checkQuota(n) before each call; recordUsage(n) after. Exceeding throws EmbeddingQuotaExceededError with tier + quota + current. 80% of quota logs a warn. Quota is only enforced if quotaDb is wired and userTier is explicitly set — local dev doesn't hit it.

10.4 Dimensionality

Hardcoded 1024 in schema.ts (memory_frames_vec USING vec0(embedding float[1024])). Every embedder normalizes to 1024 via truncation or zero-padding. If an embedder can't hit 1024, the probe fails and we move on.

11. Additional tables in the same DB (not "memory" but worth naming)

These share the MindDB but are not part of the five-layer stack:

  • sessions — GOP registry. Every frame's gop_id must FK here. Sessions have project_id, status, summary.
  • procedures — GEPA-optimized prompt templates with version, success_rate, avg_cost.
  • improvement_signals — (category, pattern_key) unique; counts recurring corrections / capability gaps / workflow patterns. surfaced flag prevents re-nagging.
  • install_audit — every capability install (native/skill/plugin/mcp) gets a row. risk + trust_source + approval_class + action.
  • ai_interactions — EU AI Act Art. 12 event log. Every model call gets a row. Don't disable.
  • execution_traces — per-run trace_json + outcome. Foundation for self-evolution.
  • evolution_runs — the other half: proposed/accepted/rejected prompt/schema evolution runs with baseline vs. winner and gate verdict.
  • harvest_sources — Memory Harvest sync state per source.

12. Conventions to follow when you extend this

  1. Hand-write SQL, use prepared statements, put the prepared statement in a method. No ORM creep. Every prepare() call in frames.ts/search.ts can be traced back to one method.
  2. New tables go in schema.ts. Add the CREATE TABLE IF NOT EXISTS there. If the schema shape changes, bump SCHEMA_VERSION and write a migration branch in MindDB.runMigrations().
  3. New search sources feed RRF, not replace it. If you add a third index path (say, a graph-walk index), fuse its results into the same RRF loop — don't build a parallel search2().
  4. Importance + source are cheap. Use them. Every frame writer should set importance honestly — temporary gets compacted in 30 days, deprecated in 90. source is enforced at insert time and feeds the trust model.
  5. Don't bypass HybridSearch.indexFrame. Writing to memory_frames directly leaves frames invisible to search. There's no "rebuild index" script — it'd be expensive and slow, so don't rely on one.
  6. Every layer's toContext() goes into the system prompt. Keep that output terse. Token budget for identity is <500 tokens; awareness is 10 items max. Don't inflate these.
  7. Schema version is a single string in meta. If you break backwards compatibility, change both the version string and the migration path. Don't let the two drift.

13. Where to read next

  • packages/hive-mind-core/src/mind/reconcile.ts — how conflicting frames get reconciled (not covered here).
  • packages/hive-mind-core/src/mind/ontology.ts — the entity/relation type registry.
  • packages/hive-mind-core/src/mind/entity-normalizer.ts — how raw strings become KG entities.
  • packages/hive-mind-core/src/mind/sessions.ts — GOP lifecycle (open, close, summarize).
  • packages/agent/src/orchestrator.ts lines 273–400 — the recall path end-to-end.
  • packages/core/src/harvest/pipeline.ts — how Memory Harvest turns external sources into frames.

14. Changelog for this doc

Date Change Owner
2026-04-15 Initial grounded deep-dive. Notes CLAUDE.md §1.3 path discrepancy (memory files live in mind/, not at top level of packages/core/src/). Claude + Marko
Reference in New Issue View Git Blame Copy Permalink