AGENTS.md no longer describes memory as half-migrated or asks for build:packages before running the sidecar, lists the tsc -b build and the test typecheck among the checks, and drops the removed weaver package. ADR-0007 and ADR-0008 record that core no longer re-exports the legacy substrate and that sqlite-vec is gone.
149 lines
9.8 KiB
Markdown
149 lines
9.8 KiB
Markdown
# ADR-0007: The memory contract and how it maps onto mem0
|
|
|
|
- Status: Accepted
|
|
- Date: 2026-10-18
|
|
|
|
## Context
|
|
|
|
ADR-0002 decided that Waggle reaches memory through a contract backed by local mem0. The
|
|
inventory of today's callers ([research/memory-inventory.md](../research/memory-inventory.md))
|
|
shows what the contract must cover: 68 files use the frame store; content prefixes such as
|
|
`[mind-fact]` or `Decision:` act as types and are queried with `LIKE`; named "sessions" group
|
|
writes by producer; personal and workspace minds are separate databases whose ids collide; the
|
|
same databases also hold the user's profile card, a task list and a dozen non-memory tables.
|
|
What mem0 3.3.1 offers is recorded in [research/mem0-3.3.1.md](../research/mem0-3.3.1.md).
|
|
|
|
The owner decided on 2026-10-16 that the profile card keeps exact fields while facts learned
|
|
about the user live in mem0, and that facts are extracted in the background, in batches.
|
|
|
|
## Decision
|
|
|
|
1. **Two packages.** `@waggle/memory-contract` holds the `MemoryStore` interface, the record
|
|
model, the capability map, an in-memory store for development and tests, and a
|
|
compatibility suite every implementation must pass. `@waggle/memory-mem0` implements the
|
|
contract on mem0. Callers depend only on the contract.
|
|
2. **Record model.** A memory has text, a scope (personal or one workspace), a kind (fact,
|
|
event, preference, profile, decision, summary, note) that replaces the content prefixes, an
|
|
importance (critical, important, normal, temporary), a source, an origin (the producer:
|
|
'chat', 'agent-run', 'loop'…) that replaces the named sessions, an optional chat session id,
|
|
timestamps and free metadata. Ids are unique across scopes.
|
|
3. **Operations.** `remember` stores text (identical text in the same scope is not written
|
|
twice) or extracts facts from a conversation; `search` ranks; `list` returns newest first;
|
|
`count`, `get`, `update`, `forget`, `forgetMatching` and `history` complete the set. Filters
|
|
cover scopes, kinds, minimum importance, origin, session and a creation-time window.
|
|
4. **No direct access.** Code outside a `MemoryStore` implementation does not query a memory
|
|
database with SQL and does not import `@waggle/hive-mind-*`, `memory-mcp` or
|
|
`wiki-compiler`. New code follows this rule now; P1.5 removes the existing violations.
|
|
5. **Not memory.** The user's profile card (name, role, department, style and the other
|
|
identity fields), the task list (awareness) and the other Waggle tables move to `waggle.db`
|
|
in P1.5. What the model learns about the user is memory: kind 'profile', personal scope.
|
|
Done on 2026-10-23, see "waggle.db" below.
|
|
6. **Extraction policy.** Facts are extracted in the background, in batches (after several
|
|
turns or a pause), with the model the user chose, through the sidecar's built-in
|
|
OpenAI-compatible proxy. A setting turns it off. Wiring this is part of P1.5.
|
|
7. **mem0 configuration.**
|
|
- One store per data directory, in `<dataDir>/memory/`: `vectors.db`, `vectors_entities.db`,
|
|
`history.db` and mem0's `config.json`. Every memory belongs to user 'local'; the scope is
|
|
mem0's `agent_id` ('personal' or 'workspace:<id>'), so mem0 never compares facts across
|
|
scopes. Contract fields live in mem0 metadata.
|
|
- Embeddings come from Waggle's in-process embedder, passed through mem0's `langchain`
|
|
provider, so memory works offline. Verified on 2026-10-18 with network access blocked:
|
|
no request was made and paraphrased queries found the right memory.
|
|
- Fact extraction uses mem0's `openai` provider pointed at an OpenAI-compatible endpoint.
|
|
Without one, `capabilities.extraction` is false.
|
|
- `MEM0_TELEMETRY=false` and `MEM0_DIR` are set before mem0 is imported. With telemetry on,
|
|
mem0 sends every call to its PostHog project and downloads a notice configuration from
|
|
GitHub; a test asserts that no request is made.
|
|
- `mem0ai` is pinned to exactly 3.3.1. An upgrade re-runs the compatibility suite and
|
|
re-checks the telemetry behaviour.
|
|
8. **Knowledge graph.** `capabilities.knowledgeGraph` is false; graph features stay hidden
|
|
(ADR-0002).
|
|
|
|
## Consequences
|
|
|
|
- P1.5 maps the callers onto the contract: prefix lanes become `kinds`, the importance lane
|
|
becomes `minImportance`, named sessions become `origin`, `getLatestIFrame`/`createPFrame`
|
|
chains become plain `remember` calls.
|
|
- Compared with the current engine: a write cannot set its own creation time (mem0's OSS build
|
|
rejects timestamps; history import is hidden in v1 anyway); importance filters but does not
|
|
boost ranking; P/B-frame chains, access counts and full-text search are gone, replaced by
|
|
mem0's hybrid vector and BM25 search. P1.5 compares search quality on a reference set.
|
|
- `list`, `count` and `forgetMatching` read every memory of the scopes involved. That is fine
|
|
at personal scale and should be revisited if a store grows past tens of thousands.
|
|
- mem0 keeps its SQLite handles open for the life of the process; `close` only marks the store
|
|
closed.
|
|
|
|
## waggle.db (P1.5, day 6)
|
|
|
|
`WaggleDB` in `@waggle/core` opens `<dataDir>/waggle.db`. It holds:
|
|
|
|
- automations: `cron_schedules`, `cron_execution_history`, `cron_run_leases`;
|
|
- `notifications` and held actions (`pending_actions`);
|
|
- `skill_hashes` and thumbs feedback (`feedback_entries`);
|
|
- cost tracking: `execution_traces`, `execution_trace_spend`,
|
|
`execution_trace_spend_reservations`;
|
|
- the profile card (`profiles`) and awareness items (`awareness_items`), one set per scope:
|
|
`personal` or `workspace:<id>`.
|
|
|
|
The stores of these tables take a `WaggleDB`, not a generic SQLite handle, so handing them a
|
|
memory database is a type error. Telemetry (`telemetry.db`) and the audit log (`audit.db`)
|
|
already had their own files and stay there.
|
|
|
|
On the first start the server copies these tables from the legacy `personal.mind`, and the
|
|
profile and awareness items of every `workspace.mind`, into `waggle.db`
|
|
(`importLegacyMindData`). The legacy files are only read, because the data directory may be
|
|
shared with the commercial edition; each scope is copied once (recorded in `waggle_meta`) and a
|
|
table only while its target is empty.
|
|
|
|
The logger, injection scanner, external-memory ingress guard, `WorkspaceManager` and the
|
|
embedders were copied from `@waggle/hive-mind-core` into `@waggle/core`, which exports its own
|
|
copies. Since P1.5 `@waggle/core` no longer depends on or re-exports the substrate; the
|
|
substrate keeps its copies until it is removed in P2.2. Those files came from an
|
|
Apache-2.0 package into the MIT `core` package; both are Egzakta's code, and the licence of the
|
|
copies is part of the licensing review.
|
|
|
|
|
|
## The server on mem0 (P1.5, days 7 and 8)
|
|
|
|
- **One store.** The sidecar opens one mem0 store in `<dataDir>/memory/` at startup and
|
|
decorates it as `server.memory`. Agent sessions select a workspace with
|
|
`Orchestrator.setWorkspace(id)`; nothing opens a database per workspace any more.
|
|
Embeddings come from the in-process model (`all-MiniLM-L6-v2`), loaded on first use and
|
|
stored at its native 384 dimensions (`createMemoryEmbedder` in core). The CLI REPL opens
|
|
the same store. Tests use a deterministic word embedder (`WAGGLE_MEMORY_EMBEDDER=words`).
|
|
- **Background extraction** (decision 6). Finished chat turns are queued per session and
|
|
handed to `remember({ conversation })` after six messages or a two-minute pause. The
|
|
facts land in personal memory as kind `profile`, origin `extraction`. mem0 calls the
|
|
model through the built-in OpenAI-compatible proxy (or LiteLLM when the user runs it)
|
|
and reads the user's current model on every call. `memoryExtraction` in `config.json`
|
|
turns it off (default on). Failures are logged and never block a turn.
|
|
- **Legacy memories** are copied once into the store in the background
|
|
(`legacy-memory-import.ts`). The `.mind` files are opened read-only. Content prefixes
|
|
become kinds, `[mind-rawturn …]` frames and deprecated frames are skipped, the original
|
|
creation time is kept in `metadata.legacy_created_at` because mem0 sets its own. Progress
|
|
is kept per scope in `waggle_meta`, so an interrupted copy resumes.
|
|
- **Writers.** Chat write-back, compaction summaries, sub-agent and agent-run results
|
|
(`run-memory.ts`), loops, connector fetch, indexed workspace files and the failed-turn
|
|
capture all call `remember`. Contract field names are reserved in metadata; agent-run
|
|
metadata renames them with a `run_` prefix.
|
|
- **Recall** keeps the importance lane, the date window and the catch-up branch, renders
|
|
`profile` and `event` memories as their own sections, and drops the raw-dialogue and
|
|
extracted-fact lanes of the old engine. The cross-encoder reranker is opt-in
|
|
(`WAGGLE_RERANKER=1`).
|
|
- **Search quality** was compared on a reference set of 100 memories and 67 queries
|
|
([research/memory-search-quality.md](../research/memory-search-quality.md)). mem0 matches
|
|
the old engine with its reranker (recall@1 0.72, recall@5 0.89 against 0.72 and 0.90) at
|
|
a seventh of the latency. The reranker did not improve mem0's ranking and costs about
|
|
130 ms per search, so it stays off. The benchmark keeps running against mem0 alone after
|
|
the old engine is removed.
|
|
- **Waggle data moved to waggle.db** as well: the install audit trail (`install_audit`,
|
|
copied from `personal.mind` with the other personal tables) and the file index
|
|
(`file_index`, keyed by scope and path, pointing at memory ids; files are re-indexed on
|
|
their next write rather than migrated).
|
|
- **Removed with the old engine:** the weaver package (consolidation, decay, transcript
|
|
distillation), harvest and its auto-sync, wiki, chat-export import, memory lanes, vector
|
|
backfill and index reconciliation, knowledge-graph tools and extraction, and the
|
|
per-workspace mind cache in sessions. Scheduled jobs left over from legacy data
|
|
(`index_reconcile`, `harvest_sync`, `memory_compact`, `memory_lane_extract`,
|
|
`workspace_health`) run as no-ops.
|