Files
waggle-os/docs/oss/research/memory-inventory.md
shumeiko.it 10ad678547 docs(oss): record the memory contract decision (ADR-0007)
ADR-0007 maps today's memory usage onto the contract and mem0: profile
card and tasks move to waggle.db, learned facts stay in mem0, extraction
runs in the background in batches. Research notes cover what mem0 3.3.1
does and how 68 files use memory today. Two defects found on the way are
logged for the main project (UF-025, UF-026).
2026-10-18 14:28:13 +02:00

135 lines
9.4 KiB
Markdown

# How the code uses memory today (input for P1.5)
Read-only inventory of the commercial memory substrate's callers, made on 2026-10-16 at commit
`5e0c369` by reading the code. Scope: non-test sources of `agent`, `server`, `weaver`, `cli`,
`core`, and HTTP calls in `apps/web`. Paths are relative to `packages/` unless marked.
## Size
68 files import memory (agent 19, server 38, weaver 1, cli 1, core 10), all through
`@waggle/core`. About 60 sites build `new FrameStore(db)` / `new SessionStore(db)` per request.
`MindDB.getDatabase()` is called 130 times in 26 files.
## Classes and the methods callers use
| Class | Used (calls) | Never used outside the substrate |
|---|---|---|
| FrameStore | createIFrame 31, setMetadata 19, getById ~12, getLatestIFrame 10, createPFrame 9, getRecent 8, list 6, delete 6, update 4, deleteByContentPrefix 3, findDuplicate 2, getGopFrames 2, touch 2, compact 2, reconstructState 1, createBFrame 1 (dead path) | hasSession, getPFramesSinceLastI, getImportanceMultiplier, getRecentFiltered, getBFrameReferences, runInTransaction |
| SessionStore | ensure 16, getActive 14 (always `[0].gop_id` to pick where a write goes), create 11, ensureActive 3, getByGopId 1, getByProject 1 (dead) | close, archive |
| HybridSearch | search ~20 (options: limit, profile, reranker, since/until, excludeDeprecated once); indexFrame 2, indexFramesBatch 2 | |
| KnowledgeGraph | createEntity 4, getEntitiesByType 4, getRelationsFrom 4, createRelation 4, searchEntities 3, getEntity 3, getEntityCount 2, linkEntityToFrame 1; wiki-compiler reads more | traverse, bfsDistances, findEntityByName, getEntitiesValidAt, … |
| IdentityLayer | exists 8, get 4, toContext 4, create 3, update 2 | |
| AwarenessLayer | add 4, toContext 3, getAll 1, getByStatus/parseMetadata/updateMetadata 1 each (loop-executor) | get, remove, update, getByCategory, clear, clearCategory |
| MultiMind | `.personal` 92 references in 21 files; search (FTS) 3; getFrameStore 2; setWorkspace 1 | getIdentity, getAwareness |
| MultiMindCache | acquire 16 / release 17 (leases), getOrOpen 5, getIfOpen 2; `agentState.getWorkspaceMindDb()` 44 calls in 14 files | |
Search result fields read: `frame.content` (16), `frame.importance` (10), `frame.id`,
`frame.created_at`, `frame.frame_type`, `frame.source`, `finalScore`.
## Data model
- `memory_frames`: id (autoincrement), frame_type I/P/B, gop_id (FK to sessions), t,
base_frame_id, content, importance (critical/important/normal/temporary/deprecated), source
(user_stated/tool_verified/agent_inferred/import/system), access_count, created_at,
last_accessed, metadata JSON, content_hash; plus FTS5, vec0 and chunk side tables.
- Fields callers read: content, importance, created_at, id, source heavily; frame_type, gop_id,
access_count, last_accessed moderately; t, base_frame_id, content_hash effectively never.
- Metadata keys in use: kind, title, scope, status, confidence, tags, evidence, sourceId,
sourceUrl, archiveUids, relatedMemoryIds, relatedArtifactIds, updatedAt, trace_id.
- **Content prefixes act as types** and are queried with `LIKE`: `[mind-fact]`, `[mind-event]`,
`[mind-profile Name]`, `[mind-rawturn …]`; `User preference:`, `Correction from user:`,
`Style note:`, `Workspace topic:`, `Decision:`, `Research findings:`, `Recommendation:`,
`User asked:`, `Work completed:`; `[Session summary — key]`, `[Loop: name]`,
`[Sub-agent: role]`, `[Team:author]`, `# Monthly Agent Assessment — period`.
- `sessions` are named buckets, not chat sessions: 'harvest', 'agent-runs', 'loop',
'team-sync', 'memory-lanes', 'agent-signals', 'session:file-index'. Chat transcripts are JSONL
files under `<dataDir>/workspaces/<id>/sessions/`.
- `identity`: one row (name, role, department, personality, capabilities, system_prompt).
- `awareness`: category (task/action/pending/flag), content, priority, metadata, expires_at; at
most 10 returned; loop-executor keeps status/result/lastTickAt/score in metadata.
- `knowledge_entities` / `knowledge_relations` with validity columns.
## Personal and workspace minds
- Personal: `<dataDir>/personal.mind` as `server.multiMind.personal`.
- Workspace: `<dataDir>/workspaces/<id>/workspace.mind`, opened through an LRU `MultiMindCache`
(max 20). Some routes open `new MindDB(path)` directly; `routes/workspaces.ts:211` opens a raw
read-only better-sqlite3 handle.
- Every session orchestrator is built on the personal DB (`server/src/local/index.ts:1298`) and
gets the workspace mind added with `setWorkspaceMind`. `getFrames()`, `getSessions()` and
`getKnowledge()` therefore return personal layers (see UF-025).
- Ids collide across minds (separate autoincrements); Memory Center routes take
`?mind=personal|workspace` to resolve them.
- The personal mind also holds non-memory tables: cron, audit, traces, evolution, signals,
compliance, feedback, wiki_pages, harvest sources/runs, raw_archive, file_index.
## Direct SQL outside the substrate
| File | Tables | Purpose |
|---|---|---|
| agent/src/orchestrator.ts | memory_frames, sessions, knowledge_entities | counts, empty-mind check, catch-up and importance lanes, prefix lanes |
| agent/src/context-loader.ts | memory_frames, knowledge_* | recent frames, top entities, personal-preference `LIKE` |
| agent/src/tools.ts | memory_frames | save_memory dedup, contradiction scan, search fallback |
| agent/src/pattern-write-back.ts | memory_frames | style-note duplicate check |
| server/src/local/index.ts | memory_frames, memory_frames_vec, install_audit | health and vector coverage counts |
| local/vector-backfill.ts | frames, vec, chunks, meta | embedding backfill |
| local/memory-lane-cron.ts | memory_frames, meta | nightly lane extraction watermark |
| local/monthly-assessment.ts | sessions, install_audit, optimization_log | assessment |
| local/proactive-handlers.ts, local/workspace-state.ts | awareness, memory_frames | counts, decisions `LIKE` |
| routes/memory.ts, routes/knowledge.ts, routes/workspace-context.ts | frames FTS, knowledge_* | global FTS, graph dump, workspace context |
| routes/workspaces.ts | memory_frames, sessions | counts; raw INSERT of template starter frames |
| routes/feedback.ts, routes/wiki.ts | feedback_entries, improvement_signals, wiki_pages | non-memory data in the mind |
| weaver/src/consolidation.ts | memory_frames, sessions, knowledge_entities | decay/strengthen, delete, archive |
| cli/src/repl.ts | memory_frames, sessions | counts |
## HTTP surface used by the web app
`/api/memory/search`, `/api/memory/frames` (GET/POST/PUT/PATCH access/DELETE), `/api/memory/stats`,
`/api/quick-capture`, Memory Center `/api/memory` and `/api/memory/:id` (GET/PATCH/DELETE, POST,
archive, confirm, trace, source), `/api/memory/erase`, `/api/memory/suppression`,
`/api/memory/merge`, `/api/memory/graph`, `/api/mind/identity|awareness|skills`,
`/api/identity`, `/api/wiki/*`, `/api/harvest/*`, `/api/import/*`, `/api/weaver/status`,
`/api/workspaces/:id/context|state|activity|export`, `/api/home/briefing`,
`/api/onboarding/status`, `/api/embedding/*`, `/api/data/erase`. Direct fetches outside the
adapter: `/api/export`, `/api/weaver/trigger`, `/api/dreams`, `/api/backup`, `/api/restore`.
## Agent tools
get_identity, get_awareness, search_memory (workspace first, then personal), search_all_workspaces,
save_memory (injection scan, 50 saves per session, routing guard, three-level dedup, contradiction
check, cognify), query_knowledge, add_task, correct_knowledge, read_other_workspace and the
cross-workspace file tools. Three dependencies are never wired (UF-026).
## Recall per chat turn (`agent/src/orchestrator.ts` recallMemory)
Empty-mind check; "catch me up" path; otherwise date window, lazy reranker, personal and
workspace hybrid search (limit 10, retried without the window), importance lane (top 5
critical/important), filter temporary/deprecated, prefix lanes (all profiles, last 60 facts,
last 40 events), raw-detail lane, render as Profiles / Facts / Events / Workspace Memory /
Personal Memory, injection scan, provenance preamble and temporal guidance.
`buildSystemPrompt` adds identity, memory stats, top-5 frames, awareness, top-10 entities and
5 personal preferences.
## Automatic writes
After each chat turn (pattern write-back, entity extraction), context compaction summaries,
agent collaboration results, sub-agent completion, waggle-dance signals, loop cron, weaver
(hourly consolidation, daily decay/strengthen, transcript distillation), harvest auto-sync and
cron, connector fetch, nightly memory-lane extraction, compaction and reindex crons, weekly
workspace health flags, monthly assessment, team sync, vector enrichment, file upload indexing,
workspace template starter frames, harvest commit.
## Call patterns the contract must cover first
1. Write into a chosen mind with importance and source; duplicates return the existing memory.
2. Ranked search per mind with limit and date window; result needs id, text, importance,
source, created time, kind.
3. Prefix and importance lanes, which become `kinds` and `minImportance` filters.
4. Recent listing for the context loader, Memory Center and home.
5. Metadata, get, update and delete by id, with no id collisions across minds.
6. Personal plus many workspace scopes, and cross-workspace search.
7. Named buckets (harvest, agent-runs, loop, memory-lanes…), which become `origin`.
8. Identity singleton and awareness items: not memory, they move to `waggle.db` (ADR-0007).
9. Knowledge graph: hidden in v1.
10. Maintenance (re-index, decay, compaction, replace-on-update, erasure, counts).