Files
waggle-os/docs/oss/adr/0007-memory-contract.md
shumeiko.it 0fef37ac53 chore(core): mark the files copied from hive-mind-core as Apache-2.0
The logger, injection scanner, ingress guard, WorkspaceManager, embedders,
recall helpers and execution traces came from @waggle/hive-mind-core
(Apache-2.0); awareness, identity and parts of WaggleDB were adapted from
it. They keep Apache-2.0: an SPDX header with the original copyright line
and the source file, packages/core/LICENSE.md listing them,
LICENSE-APACHE with the full text, and "MIT AND Apache-2.0" in
package.json. A test keeps the list and the headers in step. No code
changes.

Logs UF-069: the hive-mind-core LICENSE is an abridged Apache text.
2026-11-04 10:19:05 +01:00

9.7 KiB

ADR-0007: The memory contract and how it maps onto mem0

  • Status: Accepted
  • Date: 2026-10-18, amended 2026-10-28 (no copy of legacy data)

Context

ADR-0002 decided that Waggle reaches memory through a contract backed by local mem0. The inventory of today's callers (research/memory-inventory.md) shows what the contract must cover: 68 files use the frame store; content prefixes such as [mind-fact] or Decision: act as types and are queried with LIKE; named "sessions" group writes by producer; personal and workspace minds are separate databases whose ids collide; the same databases also hold the user's profile card, a task list and a dozen non-memory tables. What mem0 3.3.1 offers is recorded in research/mem0-3.3.1.md.

The owner decided on 2026-10-16 that the profile card keeps exact fields while facts learned about the user live in mem0, and that facts are extracted in the background, in batches.

Decision

  1. Two packages. @waggle/memory-contract holds the MemoryStore interface, the record model, the capability map, an in-memory store for development and tests, and a compatibility suite every implementation must pass. @waggle/memory-mem0 implements the contract on mem0. Callers depend only on the contract.
  2. Record model. A memory has text, a scope (personal or one workspace), a kind (fact, event, preference, profile, decision, summary, note) that replaces the content prefixes, an importance (critical, important, normal, temporary), a source, an origin (the producer: 'chat', 'agent-run', 'loop'…) that replaces the named sessions, an optional chat session id, timestamps and free metadata. Ids are unique across scopes.
  3. Operations. remember stores text (identical text in the same scope is not written twice) or extracts facts from a conversation; search ranks; list returns newest first; count, get, update, forget, forgetMatching and history complete the set. Filters cover scopes, kinds, minimum importance, origin, session and a creation-time window.
  4. No direct access. Code outside a MemoryStore implementation does not query a memory database with SQL and does not import @waggle/hive-mind-*, memory-mcp or wiki-compiler. New code follows this rule now; P1.5 removes the existing violations.
  5. Not memory. The user's profile card (name, role, department, style and the other identity fields), the task list (awareness) and the other Waggle tables move to waggle.db in P1.5. What the model learns about the user is memory: kind 'profile', personal scope. Done on 2026-10-23, see "waggle.db" below.
  6. Extraction policy. Facts are extracted in the background, in batches (after several turns or a pause), with the model the user chose, through the sidecar's built-in OpenAI-compatible proxy. A setting turns it off. Wiring this is part of P1.5.
  7. mem0 configuration.
    • One store per data directory, in <dataDir>/memory/: vectors.db, vectors_entities.db, history.db and mem0's config.json. Every memory belongs to user 'local'; the scope is mem0's agent_id ('personal' or 'workspace:'), so mem0 never compares facts across scopes. Contract fields live in mem0 metadata.
    • Embeddings come from Waggle's in-process embedder, passed through mem0's langchain provider, so memory works offline. Verified on 2026-10-18 with network access blocked: no request was made and paraphrased queries found the right memory.
    • Fact extraction uses mem0's openai provider pointed at an OpenAI-compatible endpoint. Without one, capabilities.extraction is false.
    • MEM0_TELEMETRY=false and MEM0_DIR are set before mem0 is imported. With telemetry on, mem0 sends every call to its PostHog project and downloads a notice configuration from GitHub; a test asserts that no request is made.
    • mem0ai is pinned to exactly 3.3.1. An upgrade re-runs the compatibility suite and re-checks the telemetry behaviour.
  8. Knowledge graph. capabilities.knowledgeGraph is false; graph features stay hidden (ADR-0002).

Consequences

  • P1.5 maps the callers onto the contract: prefix lanes become kinds, the importance lane becomes minImportance, named sessions become origin, getLatestIFrame/createPFrame chains become plain remember calls.
  • Compared with the current engine: a write cannot set its own creation time (mem0's OSS build rejects timestamps; history import is hidden in v1 anyway); importance filters but does not boost ranking; P/B-frame chains, access counts and full-text search are gone, replaced by mem0's hybrid vector and BM25 search. P1.5 compares search quality on a reference set.
  • list, count and forgetMatching read every memory of the scopes involved. That is fine at personal scale and should be revisited if a store grows past tens of thousands.
  • mem0 keeps its SQLite handles open for the life of the process; close only marks the store closed.

waggle.db (P1.5, day 6)

WaggleDB in @waggle/core opens <dataDir>/waggle.db. It holds:

  • automations: cron_schedules, cron_execution_history, cron_run_leases;
  • notifications and held actions (pending_actions);
  • skill_hashes and thumbs feedback (feedback_entries);
  • cost tracking: execution_traces, execution_trace_spend, execution_trace_spend_reservations;
  • the profile card (profiles) and awareness items (awareness_items), one set per scope: personal or workspace:<id>.

The stores of these tables take a WaggleDB, not a generic SQLite handle, so handing them a memory database is a type error. Telemetry (telemetry.db) and the audit log (audit.db) already had their own files and stay there.

Nothing is copied from the .mind files of earlier versions. Until 2026-10-28 the first start copied these tables and the legacy memories (importLegacyMindData, legacy-memory-import.ts); the owner removed both because the product is still in development and nobody has data to bring over. A data directory left by the commercial edition is ignored.

The logger, injection scanner, external-memory ingress guard, WorkspaceManager, the embedders, the recall helpers and execution traces were copied from @waggle/hive-mind-core into @waggle/core, which exports its own copies; the awareness and identity layers and parts of WaggleDB were adapted from it. Since P1.5 @waggle/core no longer depends on or re-exports the substrate, which left the repository in P2.2. Those files came from an Apache-2.0 package into the MIT core package and keep Apache-2.0 (settled on 2026-11-04, no owner decision needed): each carries an SPDX-License-Identifier: Apache-2.0 header with the original copyright line, packages/core/LICENSE.md lists them, packages/core/package.json declares MIT AND Apache-2.0, and THIRD_PARTY_NOTICES.md names them with the Apache-2.0 text.

The server on mem0 (P1.5, days 7 and 8)

  • One store. The sidecar opens one mem0 store in <dataDir>/memory/ at startup and decorates it as server.memory. Agent sessions select a workspace with Orchestrator.setWorkspace(id); nothing opens a database per workspace any more. Embeddings come from the in-process model (all-MiniLM-L6-v2), loaded on first use and stored at its native 384 dimensions (createMemoryEmbedder in core). The CLI REPL opens the same store. Tests use a deterministic word embedder (WAGGLE_MEMORY_EMBEDDER=words).
  • Background extraction (decision 6). Finished chat turns are queued per session and handed to remember({ conversation }) after six messages or a two-minute pause. The facts land in personal memory as kind profile, origin extraction. mem0 calls the model through the built-in OpenAI-compatible proxy (or LiteLLM when the user runs it) and reads the user's current model on every call. memoryExtraction in config.json turns it off (default on). Failures are logged and never block a turn.
  • Writers. Chat write-back, compaction summaries, sub-agent and agent-run results (run-memory.ts), loops, connector fetch, indexed workspace files and the failed-turn capture all call remember. Contract field names are reserved in metadata; agent-run metadata renames them with a run_ prefix.
  • Recall keeps the importance lane, the date window and the catch-up branch, renders profile and event memories as their own sections, and drops the raw-dialogue and extracted-fact lanes of the old engine. The cross-encoder reranker is opt-in (WAGGLE_RERANKER=1).
  • Search quality was compared on a reference set of 100 memories and 67 queries (research/memory-search-quality.md). mem0 matches the old engine with its reranker (recall@1 0.72, recall@5 0.89 against 0.72 and 0.90) at a seventh of the latency. The reranker did not improve mem0's ranking and costs about 130 ms per search, so it stays off. The benchmark keeps running against mem0 alone after the old engine is removed.
  • Waggle data moved to waggle.db as well: the install audit trail (install_audit) and the file index (file_index, keyed by scope and path, pointing at memory ids; files are re-indexed on their next write rather than migrated).
  • Removed with the old engine: the weaver package (consolidation, decay, transcript distillation), harvest and its auto-sync, wiki, chat-export import, memory lanes, vector backfill and index reconciliation, knowledge-graph tools and extraction, and the per-workspace mind cache in sessions. Scheduled jobs left over from legacy data (index_reconcile, harvest_sync, memory_compact, memory_lane_extract, workspace_health) run as no-ops.