Files
waggle-os/docs/oss/adr/0007-memory-contract.md
shumeiko.it ad2ca875cb feat(core): give Waggle its own database, apart from memory (P1.5)
Waggle kept its own tables in the memory database. Before the memory
engine can be replaced by mem0 they need a home of their own.

- add WaggleDB (<dataDir>/waggle.db). Automations, notifications, held
  actions, skill hashes, thumbs feedback, cost tracking, the profile card
  and awareness items live there. Their stores take a WaggleDB, so passing
  a memory database is a type error.
- the profile card and awareness items are now per scope ('personal' or
  'workspace:<id>') in one table each, instead of one table per .mind file
- on the first start, copy these tables from personal.mind and every
  workspace.mind (importLegacyMindData). The legacy files are only read,
  each scope is copied once and a table only while its target is empty.
- copy the logger, injection scanner, external-memory ingress guard,
  WorkspaceManager and the embedders from @waggle/hive-mind-core into
  @waggle/core, which now re-exports its own copies
- point the server, agent orchestrator and CLI at waggle.db; update the
  tests that seeded these tables in a memory database

ADR-0007 records the layout and the licensing question about the copied
files.
2026-10-23 14:31:28 +02:00

104 lines
6.5 KiB
Markdown

# ADR-0007: The memory contract and how it maps onto mem0
- Status: Accepted
- Date: 2026-10-18
## Context
ADR-0002 decided that Waggle reaches memory through a contract backed by local mem0. The
inventory of today's callers ([research/memory-inventory.md](../research/memory-inventory.md))
shows what the contract must cover: 68 files use the frame store; content prefixes such as
`[mind-fact]` or `Decision:` act as types and are queried with `LIKE`; named "sessions" group
writes by producer; personal and workspace minds are separate databases whose ids collide; the
same databases also hold the user's profile card, a task list and a dozen non-memory tables.
What mem0 3.3.1 offers is recorded in [research/mem0-3.3.1.md](../research/mem0-3.3.1.md).
The owner decided on 2026-10-16 that the profile card keeps exact fields while facts learned
about the user live in mem0, and that facts are extracted in the background, in batches.
## Decision
1. **Two packages.** `@waggle/memory-contract` holds the `MemoryStore` interface, the record
model, the capability map, an in-memory store for development and tests, and a
compatibility suite every implementation must pass. `@waggle/memory-mem0` implements the
contract on mem0. Callers depend only on the contract.
2. **Record model.** A memory has text, a scope (personal or one workspace), a kind (fact,
event, preference, profile, decision, summary, note) that replaces the content prefixes, an
importance (critical, important, normal, temporary), a source, an origin (the producer:
'chat', 'agent-run', 'loop'…) that replaces the named sessions, an optional chat session id,
timestamps and free metadata. Ids are unique across scopes.
3. **Operations.** `remember` stores text (identical text in the same scope is not written
twice) or extracts facts from a conversation; `search` ranks; `list` returns newest first;
`count`, `get`, `update`, `forget`, `forgetMatching` and `history` complete the set. Filters
cover scopes, kinds, minimum importance, origin, session and a creation-time window.
4. **No direct access.** Code outside a `MemoryStore` implementation does not query a memory
database with SQL and does not import `@waggle/hive-mind-*`, `memory-mcp` or
`wiki-compiler`. New code follows this rule now; P1.5 removes the existing violations.
5. **Not memory.** The user's profile card (name, role, department, style and the other
identity fields), the task list (awareness) and the other Waggle tables move to `waggle.db`
in P1.5. What the model learns about the user is memory: kind 'profile', personal scope.
Done on 2026-10-23, see "waggle.db" below.
6. **Extraction policy.** Facts are extracted in the background, in batches (after several
turns or a pause), with the model the user chose, through the sidecar's built-in
OpenAI-compatible proxy. A setting turns it off. Wiring this is part of P1.5.
7. **mem0 configuration.**
- One store per data directory, in `<dataDir>/memory/`: `vectors.db`, `vectors_entities.db`,
`history.db` and mem0's `config.json`. Every memory belongs to user 'local'; the scope is
mem0's `agent_id` ('personal' or 'workspace:<id>'), so mem0 never compares facts across
scopes. Contract fields live in mem0 metadata.
- Embeddings come from Waggle's in-process embedder, passed through mem0's `langchain`
provider, so memory works offline. Verified on 2026-10-18 with network access blocked:
no request was made and paraphrased queries found the right memory.
- Fact extraction uses mem0's `openai` provider pointed at an OpenAI-compatible endpoint.
Without one, `capabilities.extraction` is false.
- `MEM0_TELEMETRY=false` and `MEM0_DIR` are set before mem0 is imported. With telemetry on,
mem0 sends every call to its PostHog project and downloads a notice configuration from
GitHub; a test asserts that no request is made.
- `mem0ai` is pinned to exactly 3.3.1. An upgrade re-runs the compatibility suite and
re-checks the telemetry behaviour.
8. **Knowledge graph.** `capabilities.knowledgeGraph` is false; graph features stay hidden
(ADR-0002).
## Consequences
- P1.5 maps the callers onto the contract: prefix lanes become `kinds`, the importance lane
becomes `minImportance`, named sessions become `origin`, `getLatestIFrame`/`createPFrame`
chains become plain `remember` calls.
- Compared with the current engine: a write cannot set its own creation time (mem0's OSS build
rejects timestamps; history import is hidden in v1 anyway); importance filters but does not
boost ranking; P/B-frame chains, access counts and full-text search are gone, replaced by
mem0's hybrid vector and BM25 search. P1.5 compares search quality on a reference set.
- `list`, `count` and `forgetMatching` read every memory of the scopes involved. That is fine
at personal scale and should be revisited if a store grows past tens of thousands.
- mem0 keeps its SQLite handles open for the life of the process; `close` only marks the store
closed.
## waggle.db (P1.5, day 6)
`WaggleDB` in `@waggle/core` opens `<dataDir>/waggle.db`. It holds:
- automations: `cron_schedules`, `cron_execution_history`, `cron_run_leases`;
- `notifications` and held actions (`pending_actions`);
- `skill_hashes` and thumbs feedback (`feedback_entries`);
- cost tracking: `execution_traces`, `execution_trace_spend`,
`execution_trace_spend_reservations`;
- the profile card (`profiles`) and awareness items (`awareness_items`), one set per scope:
`personal` or `workspace:<id>`.
The stores of these tables take a `WaggleDB`, not a generic SQLite handle, so handing them a
memory database is a type error. Telemetry (`telemetry.db`) and the audit log (`audit.db`)
already had their own files and stay there.
On the first start the server copies these tables from the legacy `personal.mind`, and the
profile and awareness items of every `workspace.mind`, into `waggle.db`
(`importLegacyMindData`). The legacy files are only read, because the data directory may be
shared with the commercial edition; each scope is copied once (recorded in `waggle_meta`) and a
table only while its target is empty.
The logger, injection scanner, external-memory ingress guard, `WorkspaceManager` and the
embedders were copied from `@waggle/hive-mind-core` into `@waggle/core`, which now re-exports
its own copies. The substrate keeps its copies until it is removed. Those files came from an
Apache-2.0 package into the MIT `core` package; both are Egzakta's code, and the licence of the
copies is part of the licensing review.