Files
waggle-os/docs/oss/research/memory-inventory.md
shumeiko.it 10ad678547 docs(oss): record the memory contract decision (ADR-0007)
ADR-0007 maps today's memory usage onto the contract and mem0: profile
card and tasks move to waggle.db, learned facts stay in mem0, extraction
runs in the background in batches. Research notes cover what mem0 3.3.1
does and how 68 files use memory today. Two defects found on the way are
logged for the main project (UF-025, UF-026).
2026-10-18 14:28:13 +02:00

9.4 KiB

How the code uses memory today (input for P1.5)

Read-only inventory of the commercial memory substrate's callers, made on 2026-10-16 at commit 5e0c369 by reading the code. Scope: non-test sources of agent, server, weaver, cli, core, and HTTP calls in apps/web. Paths are relative to packages/ unless marked.

Size

68 files import memory (agent 19, server 38, weaver 1, cli 1, core 10), all through @waggle/core. About 60 sites build new FrameStore(db) / new SessionStore(db) per request. MindDB.getDatabase() is called 130 times in 26 files.

Classes and the methods callers use

Class Used (calls) Never used outside the substrate
FrameStore createIFrame 31, setMetadata 19, getById ~12, getLatestIFrame 10, createPFrame 9, getRecent 8, list 6, delete 6, update 4, deleteByContentPrefix 3, findDuplicate 2, getGopFrames 2, touch 2, compact 2, reconstructState 1, createBFrame 1 (dead path) hasSession, getPFramesSinceLastI, getImportanceMultiplier, getRecentFiltered, getBFrameReferences, runInTransaction
SessionStore ensure 16, getActive 14 (always [0].gop_id to pick where a write goes), create 11, ensureActive 3, getByGopId 1, getByProject 1 (dead) close, archive
HybridSearch search ~20 (options: limit, profile, reranker, since/until, excludeDeprecated once); indexFrame 2, indexFramesBatch 2
KnowledgeGraph createEntity 4, getEntitiesByType 4, getRelationsFrom 4, createRelation 4, searchEntities 3, getEntity 3, getEntityCount 2, linkEntityToFrame 1; wiki-compiler reads more traverse, bfsDistances, findEntityByName, getEntitiesValidAt, …
IdentityLayer exists 8, get 4, toContext 4, create 3, update 2
AwarenessLayer add 4, toContext 3, getAll 1, getByStatus/parseMetadata/updateMetadata 1 each (loop-executor) get, remove, update, getByCategory, clear, clearCategory
MultiMind .personal 92 references in 21 files; search (FTS) 3; getFrameStore 2; setWorkspace 1 getIdentity, getAwareness
MultiMindCache acquire 16 / release 17 (leases), getOrOpen 5, getIfOpen 2; agentState.getWorkspaceMindDb() 44 calls in 14 files

Search result fields read: frame.content (16), frame.importance (10), frame.id, frame.created_at, frame.frame_type, frame.source, finalScore.

Data model

  • memory_frames: id (autoincrement), frame_type I/P/B, gop_id (FK to sessions), t, base_frame_id, content, importance (critical/important/normal/temporary/deprecated), source (user_stated/tool_verified/agent_inferred/import/system), access_count, created_at, last_accessed, metadata JSON, content_hash; plus FTS5, vec0 and chunk side tables.
  • Fields callers read: content, importance, created_at, id, source heavily; frame_type, gop_id, access_count, last_accessed moderately; t, base_frame_id, content_hash effectively never.
  • Metadata keys in use: kind, title, scope, status, confidence, tags, evidence, sourceId, sourceUrl, archiveUids, relatedMemoryIds, relatedArtifactIds, updatedAt, trace_id.
  • Content prefixes act as types and are queried with LIKE: [mind-fact], [mind-event], [mind-profile Name], [mind-rawturn …]; User preference:, Correction from user:, Style note:, Workspace topic:, Decision:, Research findings:, Recommendation:, User asked:, Work completed:; [Session summary — key], [Loop: name], [Sub-agent: role], [Team:author], # Monthly Agent Assessment — period.
  • sessions are named buckets, not chat sessions: 'harvest', 'agent-runs', 'loop', 'team-sync', 'memory-lanes', 'agent-signals', 'session:file-index'. Chat transcripts are JSONL files under <dataDir>/workspaces/<id>/sessions/.
  • identity: one row (name, role, department, personality, capabilities, system_prompt).
  • awareness: category (task/action/pending/flag), content, priority, metadata, expires_at; at most 10 returned; loop-executor keeps status/result/lastTickAt/score in metadata.
  • knowledge_entities / knowledge_relations with validity columns.

Personal and workspace minds

  • Personal: <dataDir>/personal.mind as server.multiMind.personal.
  • Workspace: <dataDir>/workspaces/<id>/workspace.mind, opened through an LRU MultiMindCache (max 20). Some routes open new MindDB(path) directly; routes/workspaces.ts:211 opens a raw read-only better-sqlite3 handle.
  • Every session orchestrator is built on the personal DB (server/src/local/index.ts:1298) and gets the workspace mind added with setWorkspaceMind. getFrames(), getSessions() and getKnowledge() therefore return personal layers (see UF-025).
  • Ids collide across minds (separate autoincrements); Memory Center routes take ?mind=personal|workspace to resolve them.
  • The personal mind also holds non-memory tables: cron, audit, traces, evolution, signals, compliance, feedback, wiki_pages, harvest sources/runs, raw_archive, file_index.

Direct SQL outside the substrate

File Tables Purpose
agent/src/orchestrator.ts memory_frames, sessions, knowledge_entities counts, empty-mind check, catch-up and importance lanes, prefix lanes
agent/src/context-loader.ts memory_frames, knowledge_* recent frames, top entities, personal-preference LIKE
agent/src/tools.ts memory_frames save_memory dedup, contradiction scan, search fallback
agent/src/pattern-write-back.ts memory_frames style-note duplicate check
server/src/local/index.ts memory_frames, memory_frames_vec, install_audit health and vector coverage counts
local/vector-backfill.ts frames, vec, chunks, meta embedding backfill
local/memory-lane-cron.ts memory_frames, meta nightly lane extraction watermark
local/monthly-assessment.ts sessions, install_audit, optimization_log assessment
local/proactive-handlers.ts, local/workspace-state.ts awareness, memory_frames counts, decisions LIKE
routes/memory.ts, routes/knowledge.ts, routes/workspace-context.ts frames FTS, knowledge_* global FTS, graph dump, workspace context
routes/workspaces.ts memory_frames, sessions counts; raw INSERT of template starter frames
routes/feedback.ts, routes/wiki.ts feedback_entries, improvement_signals, wiki_pages non-memory data in the mind
weaver/src/consolidation.ts memory_frames, sessions, knowledge_entities decay/strengthen, delete, archive
cli/src/repl.ts memory_frames, sessions counts

HTTP surface used by the web app

/api/memory/search, /api/memory/frames (GET/POST/PUT/PATCH access/DELETE), /api/memory/stats, /api/quick-capture, Memory Center /api/memory and /api/memory/:id (GET/PATCH/DELETE, POST, archive, confirm, trace, source), /api/memory/erase, /api/memory/suppression, /api/memory/merge, /api/memory/graph, /api/mind/identity|awareness|skills, /api/identity, /api/wiki/*, /api/harvest/*, /api/import/*, /api/weaver/status, /api/workspaces/:id/context|state|activity|export, /api/home/briefing, /api/onboarding/status, /api/embedding/*, /api/data/erase. Direct fetches outside the adapter: /api/export, /api/weaver/trigger, /api/dreams, /api/backup, /api/restore.

Agent tools

get_identity, get_awareness, search_memory (workspace first, then personal), search_all_workspaces, save_memory (injection scan, 50 saves per session, routing guard, three-level dedup, contradiction check, cognify), query_knowledge, add_task, correct_knowledge, read_other_workspace and the cross-workspace file tools. Three dependencies are never wired (UF-026).

Recall per chat turn (agent/src/orchestrator.ts recallMemory)

Empty-mind check; "catch me up" path; otherwise date window, lazy reranker, personal and workspace hybrid search (limit 10, retried without the window), importance lane (top 5 critical/important), filter temporary/deprecated, prefix lanes (all profiles, last 60 facts, last 40 events), raw-detail lane, render as Profiles / Facts / Events / Workspace Memory / Personal Memory, injection scan, provenance preamble and temporal guidance. buildSystemPrompt adds identity, memory stats, top-5 frames, awareness, top-10 entities and 5 personal preferences.

Automatic writes

After each chat turn (pattern write-back, entity extraction), context compaction summaries, agent collaboration results, sub-agent completion, waggle-dance signals, loop cron, weaver (hourly consolidation, daily decay/strengthen, transcript distillation), harvest auto-sync and cron, connector fetch, nightly memory-lane extraction, compaction and reindex crons, weekly workspace health flags, monthly assessment, team sync, vector enrichment, file upload indexing, workspace template starter frames, harvest commit.

Call patterns the contract must cover first

  1. Write into a chosen mind with importance and source; duplicates return the existing memory.
  2. Ranked search per mind with limit and date window; result needs id, text, importance, source, created time, kind.
  3. Prefix and importance lanes, which become kinds and minImportance filters.
  4. Recent listing for the context loader, Memory Center and home.
  5. Metadata, get, update and delete by id, with no id collisions across minds.
  6. Personal plus many workspace scopes, and cross-workspace search.
  7. Named buckets (harvest, agent-runs, loop, memory-lanes…), which become origin.
  8. Identity singleton and awareness items: not memory, they move to waggle.db (ADR-0007).
  9. Knowledge graph: hidden in v1.
  10. Maintenance (re-index, decay, compaction, replace-on-update, erasure, counts).