Files
waggle-os/docs/oss/adr/0007-memory-contract.md

148 lines
9.7 KiB
Markdown

# ADR-0007: The memory contract and how it maps onto mem0
- Status: Accepted
- Date: 2026-10-18
## Context
ADR-0002 decided that Waggle reaches memory through a contract backed by local mem0. The
inventory of today's callers ([research/memory-inventory.md](../research/memory-inventory.md))
shows what the contract must cover: 68 files use the frame store; content prefixes such as
`[mind-fact]` or `Decision:` act as types and are queried with `LIKE`; named "sessions" group
writes by producer; personal and workspace minds are separate databases whose ids collide; the
same databases also hold the user's profile card, a task list and a dozen non-memory tables.
What mem0 3.3.1 offers is recorded in [research/mem0-3.3.1.md](../research/mem0-3.3.1.md).
The owner decided on 2026-10-16 that the profile card keeps exact fields while facts learned
about the user live in mem0, and that facts are extracted in the background, in batches.
## Decision
1. **Two packages.** `@waggle/memory-contract` holds the `MemoryStore` interface, the record
model, the capability map, an in-memory store for development and tests, and a
compatibility suite every implementation must pass. `@waggle/memory-mem0` implements the
contract on mem0. Callers depend only on the contract.
2. **Record model.** A memory has text, a scope (personal or one workspace), a kind (fact,
event, preference, profile, decision, summary, note) that replaces the content prefixes, an
importance (critical, important, normal, temporary), a source, an origin (the producer:
'chat', 'agent-run', 'loop'…) that replaces the named sessions, an optional chat session id,
timestamps and free metadata. Ids are unique across scopes.
3. **Operations.** `remember` stores text (identical text in the same scope is not written
twice) or extracts facts from a conversation; `search` ranks; `list` returns newest first;
`count`, `get`, `update`, `forget`, `forgetMatching` and `history` complete the set. Filters
cover scopes, kinds, minimum importance, origin, session and a creation-time window.
4. **No direct access.** Code outside a `MemoryStore` implementation does not query a memory
database with SQL and does not import `@waggle/hive-mind-*`, `memory-mcp` or
`wiki-compiler`. New code follows this rule now; P1.5 removes the existing violations.
5. **Not memory.** The user's profile card (name, role, department, style and the other
identity fields), the task list (awareness) and the other Waggle tables move to `waggle.db`
in P1.5. What the model learns about the user is memory: kind 'profile', personal scope.
Done on 2026-10-23, see "waggle.db" below.
6. **Extraction policy.** Facts are extracted in the background, in batches (after several
turns or a pause), with the model the user chose, through the sidecar's built-in
OpenAI-compatible proxy. A setting turns it off. Wiring this is part of P1.5.
7. **mem0 configuration.**
- One store per data directory, in `<dataDir>/memory/`: `vectors.db`, `vectors_entities.db`,
`history.db` and mem0's `config.json`. Every memory belongs to user 'local'; the scope is
mem0's `agent_id` ('personal' or 'workspace:<id>'), so mem0 never compares facts across
scopes. Contract fields live in mem0 metadata.
- Embeddings come from Waggle's in-process embedder, passed through mem0's `langchain`
provider, so memory works offline. Verified on 2026-10-18 with network access blocked:
no request was made and paraphrased queries found the right memory.
- Fact extraction uses mem0's `openai` provider pointed at an OpenAI-compatible endpoint.
Without one, `capabilities.extraction` is false.
- `MEM0_TELEMETRY=false` and `MEM0_DIR` are set before mem0 is imported. With telemetry on,
mem0 sends every call to its PostHog project and downloads a notice configuration from
GitHub; a test asserts that no request is made.
- `mem0ai` is pinned to exactly 3.3.1. An upgrade re-runs the compatibility suite and
re-checks the telemetry behaviour.
8. **Knowledge graph.** `capabilities.knowledgeGraph` is false; graph features stay hidden
(ADR-0002).
## Consequences
- P1.5 maps the callers onto the contract: prefix lanes become `kinds`, the importance lane
becomes `minImportance`, named sessions become `origin`, `getLatestIFrame`/`createPFrame`
chains become plain `remember` calls.
- Compared with the current engine: a write cannot set its own creation time (mem0's OSS build
rejects timestamps; history import is hidden in v1 anyway); importance filters but does not
boost ranking; P/B-frame chains, access counts and full-text search are gone, replaced by
mem0's hybrid vector and BM25 search. P1.5 compares search quality on a reference set.
- `list`, `count` and `forgetMatching` read every memory of the scopes involved. That is fine
at personal scale and should be revisited if a store grows past tens of thousands.
- mem0 keeps its SQLite handles open for the life of the process; `close` only marks the store
closed.
## waggle.db (P1.5, day 6)
`WaggleDB` in `@waggle/core` opens `<dataDir>/waggle.db`. It holds:
- automations: `cron_schedules`, `cron_execution_history`, `cron_run_leases`;
- `notifications` and held actions (`pending_actions`);
- `skill_hashes` and thumbs feedback (`feedback_entries`);
- cost tracking: `execution_traces`, `execution_trace_spend`,
`execution_trace_spend_reservations`;
- the profile card (`profiles`) and awareness items (`awareness_items`), one set per scope:
`personal` or `workspace:<id>`.
The stores of these tables take a `WaggleDB`, not a generic SQLite handle, so handing them a
memory database is a type error. Telemetry (`telemetry.db`) and the audit log (`audit.db`)
already had their own files and stay there.
On the first start the server copies these tables from the legacy `personal.mind`, and the
profile and awareness items of every `workspace.mind`, into `waggle.db`
(`importLegacyMindData`). The legacy files are only read, because the data directory may be
shared with the commercial edition; each scope is copied once (recorded in `waggle_meta`) and a
table only while its target is empty.
The logger, injection scanner, external-memory ingress guard, `WorkspaceManager` and the
embedders were copied from `@waggle/hive-mind-core` into `@waggle/core`, which now re-exports
its own copies. The substrate keeps its copies until it is removed. Those files came from an
Apache-2.0 package into the MIT `core` package; both are Egzakta's code, and the licence of the
copies is part of the licensing review.
## The server on mem0 (P1.5, days 7 and 8)
- **One store.** The sidecar opens one mem0 store in `<dataDir>/memory/` at startup and
decorates it as `server.memory`. Agent sessions select a workspace with
`Orchestrator.setWorkspace(id)`; nothing opens a database per workspace any more.
Embeddings come from the in-process model (`all-MiniLM-L6-v2`), loaded on first use and
stored at its native 384 dimensions (`createMemoryEmbedder` in core). The CLI REPL opens
the same store. Tests use a deterministic word embedder (`WAGGLE_MEMORY_EMBEDDER=words`).
- **Background extraction** (decision 6). Finished chat turns are queued per session and
handed to `remember({ conversation })` after six messages or a two-minute pause. The
facts land in personal memory as kind `profile`, origin `extraction`. mem0 calls the
model through the built-in OpenAI-compatible proxy (or LiteLLM when the user runs it)
and reads the user's current model on every call. `memoryExtraction` in `config.json`
turns it off (default on). Failures are logged and never block a turn.
- **Legacy memories** are copied once into the store in the background
(`legacy-memory-import.ts`). The `.mind` files are opened read-only. Content prefixes
become kinds, `[mind-rawturn …]` frames and deprecated frames are skipped, the original
creation time is kept in `metadata.legacy_created_at` because mem0 sets its own. Progress
is kept per scope in `waggle_meta`, so an interrupted copy resumes.
- **Writers.** Chat write-back, compaction summaries, sub-agent and agent-run results
(`run-memory.ts`), loops, connector fetch, indexed workspace files and the failed-turn
capture all call `remember`. Contract field names are reserved in metadata; agent-run
metadata renames them with a `run_` prefix.
- **Recall** keeps the importance lane, the date window and the catch-up branch, renders
`profile` and `event` memories as their own sections, and drops the raw-dialogue and
extracted-fact lanes of the old engine. The cross-encoder reranker is opt-in
(`WAGGLE_RERANKER=1`).
- **Search quality** was compared on a reference set of 100 memories and 67 queries
([research/memory-search-quality.md](../research/memory-search-quality.md)). mem0 matches
the old engine with its reranker (recall@1 0.72, recall@5 0.89 against 0.72 and 0.90) at
a seventh of the latency. The reranker did not improve mem0's ranking and costs about
130 ms per search, so it stays off. The benchmark keeps running against mem0 alone after
the old engine is removed.
- **Waggle data moved to waggle.db** as well: the install audit trail (`install_audit`,
copied from `personal.mind` with the other personal tables) and the file index
(`file_index`, keyed by scope and path, pointing at memory ids; files are re-indexed on
their next write rather than migrated).
- **Removed with the old engine:** the weaver package (consolidation, decay, transcript
distillation), harvest and its auto-sync, wiki, chat-export import, memory lanes, vector
backfill and index reconciliation, knowledge-graph tools and extraction, and the
per-workspace mind cache in sessions. Scheduled jobs left over from legacy data
(`index_reconcile`, `harvest_sync`, `memory_compact`, `memory_lane_extract`,
`workspace_health`) run as no-ops.