docs(oss): record the memory contract decision (ADR-0007)

ADR-0007 maps today's memory usage onto the contract and mem0: profile
card and tasks move to waggle.db, learned facts stay in mem0, extraction
runs in the background in batches. Research notes cover what mem0 3.3.1
does and how 68 files use memory today. Two defects found on the way are
logged for the main project (UF-025, UF-026).
This commit is contained in:
shumeiko.it
2026-10-18 14:28:13 +02:00
parent cd6553061b
commit 10ad678547
7 changed files with 299 additions and 6 deletions

View File

@@ -0,0 +1,61 @@
# mem0ai 3.3.1 (OSS build): what it does and what we must configure
Source: the published npm package `mem0ai@3.3.1`, entry `mem0ai/oss` (`dist/oss/index.d.ts`
and `dist/oss/index.js`), read on 2026-10-16. Line numbers refer to `dist/oss/index.js`.
## Package
- License Apache-2.0. Runtime dependencies: `zod`, `uuid`, `axios`, `openai`.
- 41 peer dependencies, 38 of them optional. The three required peers are `better-sqlite3`
(already in Waggle), `compromise` (MIT, NLP) and `@cloudflare/workers-types` (types only,
MIT OR Apache-2.0). npm installs required peers automatically.
## API (`class Memory`)
| Method | Signature | Notes |
|---|---|---|
| `add` | `add(messages: string \| Message[], { userId?, agentId?, runId?, metadata?, infer?, timestamp?, expirationDate? })` | `infer: true` (default) asks the LLM to extract facts and decide add/update/delete against existing memories in the same `userId`/`agentId`/`runId` scope |
| `search` | `search(query, { topK?, filters?, threshold?, rerank?, showExpired? })` | `filters` holds `user_id`/`agent_id`/`run_id` and any metadata key, with operators `eq`, `ne`, `gt`, `gte`, `lt`, `lte`, `in`, `nin`, `contains`, `icontains` |
| `get` | `get(memoryId)` | `MemoryItem \| null` |
| `getAll` | `getAll({ topK?, filters?, showExpired? })` | listing per scope |
| `update` | `update(memoryId, { text?, metadata?, expirationDate? })` | |
| `delete` / `deleteAll` | `delete(memoryId)`, `deleteAll({ userId?, agentId?, runId? })` | |
| `history` | `history(memoryId)` | change log from the history store |
| `reset` | `reset()` | drops everything |
`MemoryItem` is `{ id, memory, hash?, createdAt?, updatedAt?, score?, rerankScore?, metadata? }`.
There is no graph memory in the OSS build.
## Storage
- Vector store `memory` (`MemoryVectorStore`, line 896): a SQLite file via `better-sqlite3` at
`vectorStore.config.dbPath`, table `vectors(id, vector BLOB, payload TEXT)`. Search loads the
rows matching the filters and ranks them by cosine similarity in JS. Fine for personal scale.
- History store `sqlite` at `historyDbPath`.
- Default paths fall back to the working directory or `~/.mem0`; always set them explicitly.
## Model providers
- Embedders: `openai`, `ollama`, `lmstudio`, `huggingface` (an HTTP TEI endpoint, not
in-process), `fastembed`, `google`, `azure_openai`, `vertexai`, `aws_bedrock`, `together`,
`langchain`. The `langchain` provider accepts any object with `embedQuery(text)` and
`embedDocuments(texts)` (duck-typed, no LangChain import), so Waggle's in-process embedder can
be passed in directly.
- LLMs: `openai` (accepts `baseURL` and `apiKey`, sends `response_format`), `anthropic`, `ollama`,
`litellm`, `groq`, `mistral`, `google`, `deepseek`, `xai`, `vllm`, `lmstudio`, and others. The
`langchain` LLM provider imports `@langchain/core` at runtime, so we do not use it. Pointing the
`openai` provider at the sidecar's built-in proxy (`http://127.0.0.1:<port>/v1`, session token
as key) reuses whatever model the user configured.
## Must-configure settings
- **Telemetry is on by default.** Every `add`, `search`, `update`, `delete`, `get_all` and `reset`
sends an event to mem0's PostHog project (`https://us.i.posthog.com`, line 14821). It is read
once at module load from `MEM0_TELEMETRY` (line 14817). Set `MEM0_TELEMETRY=false` before the
module is imported, which means importing `mem0ai/oss` dynamically after setting the variable.
With telemetry on, mem0 also downloads a notice configuration from
`raw.githubusercontent.com` (observed on 2026-10-18 with a stubbed `fetch`); both stop when
telemetry is off.
- **Home directory.** mem0 reads and writes `~/.mem0/config.json` unless `MEM0_DIR` is set
(lines 15056, 16962). Point `MEM0_DIR` into the Waggle data directory.
- Console notices (first run, decay, scale thresholds) are printed by default.

View File

@@ -0,0 +1,134 @@
# How the code uses memory today (input for P1.5)
Read-only inventory of the commercial memory substrate's callers, made on 2026-10-16 at commit
`5e0c369` by reading the code. Scope: non-test sources of `agent`, `server`, `weaver`, `cli`,
`core`, and HTTP calls in `apps/web`. Paths are relative to `packages/` unless marked.
## Size
68 files import memory (agent 19, server 38, weaver 1, cli 1, core 10), all through
`@waggle/core`. About 60 sites build `new FrameStore(db)` / `new SessionStore(db)` per request.
`MindDB.getDatabase()` is called 130 times in 26 files.
## Classes and the methods callers use
| Class | Used (calls) | Never used outside the substrate |
|---|---|---|
| FrameStore | createIFrame 31, setMetadata 19, getById ~12, getLatestIFrame 10, createPFrame 9, getRecent 8, list 6, delete 6, update 4, deleteByContentPrefix 3, findDuplicate 2, getGopFrames 2, touch 2, compact 2, reconstructState 1, createBFrame 1 (dead path) | hasSession, getPFramesSinceLastI, getImportanceMultiplier, getRecentFiltered, getBFrameReferences, runInTransaction |
| SessionStore | ensure 16, getActive 14 (always `[0].gop_id` to pick where a write goes), create 11, ensureActive 3, getByGopId 1, getByProject 1 (dead) | close, archive |
| HybridSearch | search ~20 (options: limit, profile, reranker, since/until, excludeDeprecated once); indexFrame 2, indexFramesBatch 2 | |
| KnowledgeGraph | createEntity 4, getEntitiesByType 4, getRelationsFrom 4, createRelation 4, searchEntities 3, getEntity 3, getEntityCount 2, linkEntityToFrame 1; wiki-compiler reads more | traverse, bfsDistances, findEntityByName, getEntitiesValidAt, … |
| IdentityLayer | exists 8, get 4, toContext 4, create 3, update 2 | |
| AwarenessLayer | add 4, toContext 3, getAll 1, getByStatus/parseMetadata/updateMetadata 1 each (loop-executor) | get, remove, update, getByCategory, clear, clearCategory |
| MultiMind | `.personal` 92 references in 21 files; search (FTS) 3; getFrameStore 2; setWorkspace 1 | getIdentity, getAwareness |
| MultiMindCache | acquire 16 / release 17 (leases), getOrOpen 5, getIfOpen 2; `agentState.getWorkspaceMindDb()` 44 calls in 14 files | |
Search result fields read: `frame.content` (16), `frame.importance` (10), `frame.id`,
`frame.created_at`, `frame.frame_type`, `frame.source`, `finalScore`.
## Data model
- `memory_frames`: id (autoincrement), frame_type I/P/B, gop_id (FK to sessions), t,
base_frame_id, content, importance (critical/important/normal/temporary/deprecated), source
(user_stated/tool_verified/agent_inferred/import/system), access_count, created_at,
last_accessed, metadata JSON, content_hash; plus FTS5, vec0 and chunk side tables.
- Fields callers read: content, importance, created_at, id, source heavily; frame_type, gop_id,
access_count, last_accessed moderately; t, base_frame_id, content_hash effectively never.
- Metadata keys in use: kind, title, scope, status, confidence, tags, evidence, sourceId,
sourceUrl, archiveUids, relatedMemoryIds, relatedArtifactIds, updatedAt, trace_id.
- **Content prefixes act as types** and are queried with `LIKE`: `[mind-fact]`, `[mind-event]`,
`[mind-profile Name]`, `[mind-rawturn …]`; `User preference:`, `Correction from user:`,
`Style note:`, `Workspace topic:`, `Decision:`, `Research findings:`, `Recommendation:`,
`User asked:`, `Work completed:`; `[Session summary — key]`, `[Loop: name]`,
`[Sub-agent: role]`, `[Team:author]`, `# Monthly Agent Assessment — period`.
- `sessions` are named buckets, not chat sessions: 'harvest', 'agent-runs', 'loop',
'team-sync', 'memory-lanes', 'agent-signals', 'session:file-index'. Chat transcripts are JSONL
files under `<dataDir>/workspaces/<id>/sessions/`.
- `identity`: one row (name, role, department, personality, capabilities, system_prompt).
- `awareness`: category (task/action/pending/flag), content, priority, metadata, expires_at; at
most 10 returned; loop-executor keeps status/result/lastTickAt/score in metadata.
- `knowledge_entities` / `knowledge_relations` with validity columns.
## Personal and workspace minds
- Personal: `<dataDir>/personal.mind` as `server.multiMind.personal`.
- Workspace: `<dataDir>/workspaces/<id>/workspace.mind`, opened through an LRU `MultiMindCache`
(max 20). Some routes open `new MindDB(path)` directly; `routes/workspaces.ts:211` opens a raw
read-only better-sqlite3 handle.
- Every session orchestrator is built on the personal DB (`server/src/local/index.ts:1298`) and
gets the workspace mind added with `setWorkspaceMind`. `getFrames()`, `getSessions()` and
`getKnowledge()` therefore return personal layers (see UF-025).
- Ids collide across minds (separate autoincrements); Memory Center routes take
`?mind=personal|workspace` to resolve them.
- The personal mind also holds non-memory tables: cron, audit, traces, evolution, signals,
compliance, feedback, wiki_pages, harvest sources/runs, raw_archive, file_index.
## Direct SQL outside the substrate
| File | Tables | Purpose |
|---|---|---|
| agent/src/orchestrator.ts | memory_frames, sessions, knowledge_entities | counts, empty-mind check, catch-up and importance lanes, prefix lanes |
| agent/src/context-loader.ts | memory_frames, knowledge_* | recent frames, top entities, personal-preference `LIKE` |
| agent/src/tools.ts | memory_frames | save_memory dedup, contradiction scan, search fallback |
| agent/src/pattern-write-back.ts | memory_frames | style-note duplicate check |
| server/src/local/index.ts | memory_frames, memory_frames_vec, install_audit | health and vector coverage counts |
| local/vector-backfill.ts | frames, vec, chunks, meta | embedding backfill |
| local/memory-lane-cron.ts | memory_frames, meta | nightly lane extraction watermark |
| local/monthly-assessment.ts | sessions, install_audit, optimization_log | assessment |
| local/proactive-handlers.ts, local/workspace-state.ts | awareness, memory_frames | counts, decisions `LIKE` |
| routes/memory.ts, routes/knowledge.ts, routes/workspace-context.ts | frames FTS, knowledge_* | global FTS, graph dump, workspace context |
| routes/workspaces.ts | memory_frames, sessions | counts; raw INSERT of template starter frames |
| routes/feedback.ts, routes/wiki.ts | feedback_entries, improvement_signals, wiki_pages | non-memory data in the mind |
| weaver/src/consolidation.ts | memory_frames, sessions, knowledge_entities | decay/strengthen, delete, archive |
| cli/src/repl.ts | memory_frames, sessions | counts |
## HTTP surface used by the web app
`/api/memory/search`, `/api/memory/frames` (GET/POST/PUT/PATCH access/DELETE), `/api/memory/stats`,
`/api/quick-capture`, Memory Center `/api/memory` and `/api/memory/:id` (GET/PATCH/DELETE, POST,
archive, confirm, trace, source), `/api/memory/erase`, `/api/memory/suppression`,
`/api/memory/merge`, `/api/memory/graph`, `/api/mind/identity|awareness|skills`,
`/api/identity`, `/api/wiki/*`, `/api/harvest/*`, `/api/import/*`, `/api/weaver/status`,
`/api/workspaces/:id/context|state|activity|export`, `/api/home/briefing`,
`/api/onboarding/status`, `/api/embedding/*`, `/api/data/erase`. Direct fetches outside the
adapter: `/api/export`, `/api/weaver/trigger`, `/api/dreams`, `/api/backup`, `/api/restore`.
## Agent tools
get_identity, get_awareness, search_memory (workspace first, then personal), search_all_workspaces,
save_memory (injection scan, 50 saves per session, routing guard, three-level dedup, contradiction
check, cognify), query_knowledge, add_task, correct_knowledge, read_other_workspace and the
cross-workspace file tools. Three dependencies are never wired (UF-026).
## Recall per chat turn (`agent/src/orchestrator.ts` recallMemory)
Empty-mind check; "catch me up" path; otherwise date window, lazy reranker, personal and
workspace hybrid search (limit 10, retried without the window), importance lane (top 5
critical/important), filter temporary/deprecated, prefix lanes (all profiles, last 60 facts,
last 40 events), raw-detail lane, render as Profiles / Facts / Events / Workspace Memory /
Personal Memory, injection scan, provenance preamble and temporal guidance.
`buildSystemPrompt` adds identity, memory stats, top-5 frames, awareness, top-10 entities and
5 personal preferences.
## Automatic writes
After each chat turn (pattern write-back, entity extraction), context compaction summaries,
agent collaboration results, sub-agent completion, waggle-dance signals, loop cron, weaver
(hourly consolidation, daily decay/strengthen, transcript distillation), harvest auto-sync and
cron, connector fetch, nightly memory-lane extraction, compaction and reindex crons, weekly
workspace health flags, monthly assessment, team sync, vector enrichment, file upload indexing,
workspace template starter frames, harvest commit.
## Call patterns the contract must cover first
1. Write into a chosen mind with importance and source; duplicates return the existing memory.
2. Ranked search per mind with limit and date window; result needs id, text, importance,
source, created time, kind.
3. Prefix and importance lanes, which become `kinds` and `minImportance` filters.
4. Recent listing for the context loader, Memory Center and home.
5. Metadata, get, update and delete by id, with no id collisions across minds.
6. Personal plus many workspace scopes, and cross-workspace search.
7. Named buckets (harvest, agent-runs, loop, memory-lanes…), which become `origin`.
8. Identity singleton and awareness items: not memory, they move to `waggle.db` (ADR-0007).
9. Knowledge graph: hidden in v1.
10. Maintenance (re-index, decay, compaction, replace-on-update, erasure, counts).