# 05b · Subsystem: Persistent Memory Engine (the MOAT)

**Purpose.** This is Waggle's durable, per-workspace memory substrate — the thing that makes agents remember across sessions. It is a single SQLite database (one file per "mind") layered into Identity → Awareness → Frames → Knowledge Graph, with **hybrid search** (vector + keyword fused), relevance scoring, index reconciliation, and a tiered embedding provider chain. Memory is **WRITTEN** by the `CognifyPipeline` (one frame per turn + entity/relation extraction) and **RECALLED** by `CombinedRetrieval` (workspace + personal + optional KVARK merge). For a frontend rebuild, treat this as a **contract**: you never touch SQLite directly — you call the sidecar HTTP routes that wrap these classes, and you render the typed shapes documented below.

> Grounding: every type, field, table, and constant below is quoted from
> `packages/hive-mind-core/src/mind/*` and `packages/agent/src/{cognify,combined-retrieval,memory-linker}.ts`.
> Where a fact is NOT in those files (e.g. exact HTTP route paths), it is flagged explicitly.

---

## 1. Mental model — five layers in one SQLite file

A "mind" is one `better-sqlite3` database (with the `sqlite-vec` extension loaded for the vector table). The schema (`schema.ts`, `SCHEMA_VERSION = '1'`) defines these layers. Each layer is a TypeScript class wrapping prepared SQL statements — there is **no ORM**.

| Layer | Class (file) | Table(s) | Cardinality | Role |
|---|---|---|---|---|
| 0 — Identity | `IdentityLayer` (`identity.ts`) | `identity` | exactly 1 row (`CHECK (id = 1)`) | Who the agent is: name/role/department/personality/capabilities/system_prompt. `<500 tokens` budget. |
| 1 — Awareness | `AwarenessLayer` (`awareness.ts`) | `awareness` | `MAX_ITEMS = 10` active | Short-term working state: active tasks, recent actions, pending items, context flags. Items can expire. |
| 2 — Frames | `FrameStore` (`frames.ts`) | `memory_frames` (+ `memory_frames_fts`, `memory_frames_vec`) | unbounded | The long-term memory store. I/P/B frame types grouped by session (`gop_id`). |
| 3 — Knowledge Graph | `KnowledgeGraph` (`knowledge.ts`) | `knowledge_entities`, `knowledge_relations` | unbounded | Entities + typed relations with temporal validity (bitemporal). |
| — Sessions | `SessionStore` (`sessions.ts`) | `sessions` | unbounded | Maps `gop_id` (Group-Of-Pictures id) → project. Parent of frames. |

The "GOP" naming (I-frame / P-frame / B-frame, `gop_id`, `t`) is borrowed from video compression: an **I-frame** is a self-contained keyframe (full state), a **P-frame** is a delta/update against its base I-frame, a **B-frame** is a bidirectional cross-reference frame. `t` is a per-session monotonic sequence number (`nextT` = `MAX(t)+1` for that `gop_id`).

```mermaid
graph TD
  subgraph MindDB["One SQLite file per workspace (a 'mind')"]
    ID[identity · 1 row]
    AW["awareness · ≤10 items"]
    SESS[sessions]
    MF[memory_frames]
    FTS["memory_frames_fts (FTS5)"]
    VEC["memory_frames_vec (vec0 · float[1024])"]
    KE[knowledge_entities]
    KR[knowledge_relations]
  end
  SESS -->|gop_id FK| MF
  MF -->|rowid = id| FTS
  MF -->|rowid = id| VEC
  KE -->|source_id / target_id| KR
```

---

## 2. Layer 2 — Frames (`FrameStore`)

The heart of the store. A `MemoryFrame` is:

| Field | Type | Notes |
|---|---|---|
| `id` | `number` | PK / rowid; same id used in FTS + vec tables |
| `frame_type` | `'I' \| 'P' \| 'B'` | `FrameType` |
| `gop_id` | `string` | session id this frame belongs to |
| `t` | `number` | per-`gop_id` monotonic sequence (`nextT`) |
| `base_frame_id` | `number \| null` | P/B frames point at their base I-frame |
| `content` | `string` | the actual text |
| `importance` | `'critical' \| 'important' \| 'normal' \| 'temporary' \| 'deprecated'` | `Importance` |
| `source` | `'user_stated' \| 'tool_verified' \| 'agent_inferred' \| 'import' \| 'system' \| 'personal' \| 'workspace' \| 'team_sync'` | `FrameSource` (TS union is wider than the DB `CHECK`, which only allows the first five) |
| `access_count` | `number` | incremented by `touch()` on every recall/dup-hit |
| `created_at` | `string` | ISO; harvest path can override to preserve source timestamp |
| `last_accessed` | `string` | ISO; drives temporal scoring |

### Write methods
- `createIFrame(gopId, content, importance='normal', source='user_stated', createdAt?)` — **dedup-guarded**: calls `findDuplicate(content)` first; if an identical frame exists it `touch()`es it and returns it instead of inserting. `createdAt` is honored only if it passes `isValidIsoTimestamp` (strict ISO-8601 with `T` + timezone) — used by the harvest path so imported frames keep their original timestamp.
- `createPFrame(gopId, content, baseFrameId, …)` — a delta against an I-frame.
- `createBFrame(gopId, content, baseFrameId, referencedFrameIds[])` — stores `{description, references}` as JSON in `content`.

Every create also runs `indexFts(frame)` to mirror content into the FTS5 table. **Vector indexing is NOT done here** — it happens in `HybridSearch.indexFrame()`, called by `CognifyPipeline` (see §6).

### Dedup (`findDuplicate`) — important quirks for the frontend
- Hash = `SHA-256( stripHmPrefix(content).trim() )`.
- `stripHmPrefix` removes a leading `[hm session:… src:… event:…] ` provenance prefix so two captures of the same turn from different sources collapse into one frame.
- **Only the last 500 frames** are scanned (cost bound). Dedup is best-effort beyond the recency window.

### Read / list methods
| Method | Returns |
|---|---|
| `getById(id)` | one frame |
| `getLatestIFrame(gopId)` | newest I-frame in a session |
| `getPFramesSinceLastI(gopId)` | P-frames after the latest I-frame |
| `getGopFrames(gopId)` | all frames in a session, `t ASC` |
| `reconstructState(gopId)` | `{ iframe, pframes }` — current state = latest I + its P-deltas |
| `getRecent(limit=50)` / `list({limit})` | newest frames, `id DESC` |
| `getRecentFiltered(limit, since?, until?)` | F20: date-bounded recent frames |
| `getStats()` | `{ total, byType, byImportance }` |
| `update(id, content, importance?)` | updates main + FTS + clears vec entry |
| `delete(id)` | removes from main + FTS + vec + `kg_entity_frames` and nulls referring `base_frame_id` |

### Compaction (`compact(maxTempAgeDays=30, maxDeprecatedAgeDays=90)`)
Maintenance op: deletes old `temporary` and `deprecated` frames, and for any `gop_id` with >10 P-frames merges all-but-the-5-most-recent P-frames into their I-frame (joined with `\n---\n`). Returns `{ temporaryPruned, deprecatedPruned, pframesMerged }`.

---

## 3. Hybrid Search (`HybridSearch`) — vector + keyword fusion

`search(query, options)` runs **keyword and vector searches in parallel**, fuses with **Reciprocal Rank Fusion (RRF)**, then multiplies by a relevance score.

`SearchOptions`:
| Field | Type | Default |
|---|---|---|
| `limit` | `number` | `20` |
| `gopId` | `string?` | (scope to one session) |
| `profile` | `'balanced' \| 'recent' \| 'important' \| 'connected'` | `'balanced'` |
| `context` | `ScoringContext` | `{}` |
| `since` / `until` | `string?` (ISO) | temporal filter on `created_at` |

`SearchResult`:
| Field | Type | Meaning |
|---|---|---|
| `frame` | `MemoryFrame` | the hit |
| `rrfScore` | `number` | fused rank score |
| `relevanceScore` | `number` | from `computeRelevance` |
| `finalScore` | `number` | `rrfScore * relevanceScore` — the sort key |

### Fusion algorithm (the actual constants)
1. Run `keywordSearch(query, limit*2, gopId)` and `vectorSearch(query, limit*2, gopId)` in parallel; each returns an ordered `number[]` of frame ids.
2. RRF with `RRF_K = 60`: each id accrues `1 / (RRF_K + rank)` from each list.
3. Fetch the union of frame ids (applying `since`/`until` filters here), compute `relevanceScore`, set `finalScore = rrfScore * relevanceScore`, sort desc, slice to `limit`.

### Keyword path (FTS5) — recall tuning
- Query is tokenized, punctuation stripped, a built-in **stop-word list** removed, tokens shorter than 3 chars dropped, then OR-joined (`"foo" OR "bar"`) for recall. FTS5 `ORDER BY rank`.
- On FTS5 parse error it falls back to `likeFallbackSearch` — OR-ed `LIKE … ESCAPE '\'` over `content` (parameterized, metachars escaped). This guarantees a query never returns a false "no memory found" because the user typed an FTS5 operator.

### Vector path (`sqlite-vec` vec0)
- `vectorSearch` embeds the query (`embedder.embed`), converts the `Float32Array` to a blob, and does a `MATCH ? AND k = ?` KNN query on `memory_frames_vec` (`ORDER BY distance`).
- When `gopId` is set it over-fetches (`k = limit*3`) then filters by session.
- The vec table is `float[1024]` — embeddings MUST be 1024-dim (see §7).
- All vec operations are wrapped in `try/catch` returning `[]` — if the `sqlite-vec` extension or table is absent, search silently degrades to keyword-only.

### Indexing (called by cognify, not by FrameStore)
- `indexFrame(frameId, content)` — embed + `INSERT INTO memory_frames_vec`. (rowid is inlined as a SQL literal because vec0 doesn't accept a parameterized rowid.)
- `indexFramesBatch(frames[])` — batch embed + transactional insert.

---

## 4. Relevance scoring (`scoring.ts`)

`computeRelevance(frame, weights, context)` = weighted sum of four sub-scores. Profiles pick the weights:

| Profile | temporal | popularity | contextual | importance |
|---|---|---|---|---|
| `balanced` | 0.4 | 0.2 | 0.2 | 0.2 |
| `recent` | 0.6 | 0.1 | 0.2 | 0.1 |
| `important` | 0.1 | 0.1 | 0.2 | 0.6 |
| `connected` | 0.1 | 0.1 | 0.6 | 0.2 |

Sub-scores:
- **temporal** — `1.0` if `last_accessed` within `RECENCY_BOOST_DAYS = 7`, else exponential decay with `HALF_LIFE_DAYS = 30`.
- **popularity** — `1 + log10(1 + access_count) * 0.1`.
- **contextual** — graph proximity: distance 0→1.0, 1→0.7, 2→0.4, 3→0.2, else 0 (needs `context.graphDistances`, a `Map<frameId, BFS-distance>`).
- **importance** — `critical 2.0 / important 1.5 / normal 1.0 / temporary 0.7 / deprecated 0.3`.

`ScoringContext = { recentEntityIds?: number[]; graphDistances?: Map<number, number> }`.

---

## 5. Knowledge Graph (`KnowledgeGraph`)

Bitemporal entity-relation store. `Entity` and `Relation` both carry `valid_from` / `valid_to` (null = currently valid) plus `recorded_at`.

`Entity`: `{ id, entity_type, name, properties(JSON string), valid_from, valid_to, recorded_at }`
`Relation`: `{ id, source_id, target_id, relation_type, confidence(REAL), properties(JSON), valid_from, valid_to, recorded_at }`

Key methods:
| Method | Purpose |
|---|---|
| `createEntity(type, name, props, temporal?)` | validates against optional schema, inserts |
| `getEntitiesByType(type, limit=500)` / `getEntities(limit, offset)` | active entities (`valid_to IS NULL`) |
| `searchEntities(query, limit=100)` | `name LIKE` (escaped) |
| `getEntityTypeCounts()` / `getEntityCount()` | dashboard counts without full fetch |
| `getEntitiesValidAt(isoTime)` | time-travel: entities valid at a past instant |
| `createRelation(src, tgt, type, confidence=1.0, props)` | validated insert |
| `getRelationsFrom(id, type?)` / `getRelationsTo(id, type?)` | adjacency |
| `retireEntity(id)` / `retireRelation(id)` | sets `valid_to = now` (soft-delete, never hard delete) |
| `traverse(startId, relationType, maxDepth)` | BFS returning reached entities |
| `bfsDistances(startId, maxDepth)` | `Map<entityId, distance>` — feeds the `contextual` score |

Optional `setValidationSchema(ValidationSchema)` enforces required properties per entity-type and an allowed-relations list (throws on violation). Without a schema, all writes pass.

---

## 6. WRITE path — `CognifyPipeline` (`cognify.ts`)

`cognify(content, importance='normal', gopId?, turnId?)` is the canonical "remember this turn" call. Steps:

1. **Ensure session** — `sessions.ensureActive()` (transaction-wrapped to avoid the twin-session race).
2. **Save frame** — if a latest I-frame exists for the session → `createPFrame` (delta), else `createIFrame` (keyframe). Dedup applies inside the FrameStore.
3. **Extract entities** — `extractEntities(content.slice(0, 10_000))` (from `entity-extractor.ts`).
4. **Upsert entities** into the KG (skip if same type+name exists; cached per-type to avoid N queries).
5. **Co-occurrence relations** — `co_occurs_with` (confidence 0.8) between every entity pair found in the same text.
6. **Semantic relations** — `extractRelations` → typed relations (`led_by`, `reports_to`, `depends_on`, …) matched back to KG entities.
7. **Vector index** — `search.indexFrame(frame.id, content)`.
8. **Optional linking** — if `enableLinking`, `MemoryLinker.findRelated(content)` returns related frames (self excluded).

`CognifyResult = { frameId, entitiesExtracted, relationsCreated, relatedFrames? }`.

Other entry points: `cognifyFrame(frameId)` (re-process one imported frame — used post-harvest) and `cognifyBatch(frameIds[])` (sequential, so each frame's new entities can link to the next).

### `MemoryLinker` (`memory-linker.ts`)
Thin wrapper over `HybridSearch.search`. `findRelated(content, limit=5)` returns `MemoryLink[] = { frameId, content, score }`, filtered by a `threshold` (default `0.1` on `finalScore`).

```mermaid
flowchart TD
  A["agent turn / harvest / MCP save_memory"] --> B["CognifyPipeline.cognify(content)"]
  B --> C["SessionStore.ensureActive() → gop_id"]
  B --> D{latest I-frame exists?}
  D -- no --> E["FrameStore.createIFrame (dedup-guarded)"]
  D -- yes --> F["FrameStore.createPFrame"]
  E --> G["indexFts (FTS5)"]
  F --> G
  B --> H["extractEntities → KnowledgeGraph.upsert"]
  H --> I["co_occurs_with + semantic relations"]
  B --> J["HybridSearch.indexFrame → memory_frames_vec (1024-dim)"]
  B --> K["MemoryLinker.findRelated (optional)"]
```

---

## 7. Embeddings — provider chain (`embedding-provider.ts`)

`createEmbeddingProvider(config?)` returns an `EmbeddingProviderInstance` (implements the `Embedder` interface: `embed`, `embedBatch`, `dimensions`). Default `targetDimensions = 1024` — matches the `vec0 float[1024]` table.

**Auto fallback chain** (`provider: 'auto'`): `inprocess → ollama → voyage → openai → mock`. Each is probed with a 1024-dim test embedding; the first that succeeds becomes active.

| Provider (`EmbeddingProviderType`) | Default model | Needs |
|---|---|---|
| `inprocess` | `Xenova/all-MiniLM-L6-v2` (Transformers.js) | nothing — fully local |
| `ollama` | `nomic-embed-text` | local Ollama server |
| `voyage` | `voyage-3-lite` | `voyage.apiKey` (from Vault) |
| `openai` | `text-embedding-3-small` | `openai.apiKey` |
| `litellm` | `text-embedding` | `litellm.url` |
| `mock` | `deterministic-mock` | always available; **semantically meaningless** (last resort) |

**Tier gating**: provider availability is gated by `TIER_CAPABILITIES[tier].embeddingProviders`; `embeddingQuotaPerMonth` is enforced per `user_id` per month in the `embedding_usage` table (`-1` = unlimited). Quota throws `EmbeddingQuotaExceededError` (carries `tier/quota/current/upgradeUrl`); over-tier provider request throws `TierError`. `WAGGLE_EVAL_MODE=1` disables all gating (eval harness only). `getStatus()` / `getQuotaStatus()` / `reprobe()` expose state for a settings UI.

> Frontend note: if the active provider is `mock`, surface a "semantic search degraded" warning — `getStatus().activeProvider === 'mock'` and `lastError` tell you. Mismatched embedder dimensions would break the vec table, so the provider hard-asserts 1024 on probe.

---

## 8. RECALL path — `CombinedRetrieval` (`combined-retrieval.ts`)

The merge engine the agent calls to answer "what do I know about X". Merges **workspace** + **personal** memory and optionally **KVARK** enterprise search. Pure data in / out (no formatting).

`search(query, opts)` → `CombinedRetrievalResult`:
| Field | Type |
|---|---|
| `query` | `string` |
| `workspaceResults` / `personalResults` / `kvarkResults` | `CombinedResult[]` |
| `kvarkAvailable` | `boolean` |
| `kvarkSkipped` | `boolean` (available but coverage was sufficient) |
| `kvarkError?` | `string` |
| `hasConflict` | `boolean` |
| `conflictNote?` | `string` |

`CombinedResult = { content, source: 'workspace'|'personal'|'kvark', attribution, score, metadata }` where `attribution` is a human tag like `[workspace memory]` / `[personal memory]`, and `metadata` carries `frameId/frameType/importance` (memory) or `documentId/documentType` (KVARK).

`CombinedSearchOptions = { limit=10, profile='balanced', scope: 'all'|'personal'|'workspace', turnId? }`.

**KVARK gating logic** (`shouldQueryKvark`): KVARK is queried only when a client exists, `scope==='all'`, AND local coverage is insufficient — `hasSufficientLocalCoverage` = fewer than `LOCAL_COVERAGE_MIN_COUNT = 3` results with `score ≥ LOCAL_COVERAGE_SCORE_THRESHOLD = 0.7`. KVARK failures degrade gracefully (local results preserved, `kvarkError` set).

**Conflict detection** (`detectConflict`): if both workspace and KVARK have strong results (`score ≥ CONFLICT_SCORE_THRESHOLD = 0.6`) and their top-3 texts disagree on status polarity (`POSITIVE_STATUS` words like *approved/selected* vs `NEGATIVE_STATUS` like *rejected/cancelled*), it returns a human-readable `conflictNote` for the UI to surface ("these sources may be out of sync").

```mermaid
flowchart TD
  Q["CombinedRetrieval.search(query, scope)"] --> WS["searchWorkspace → HybridSearch"]
  Q --> PS["searchPersonal → HybridSearch"]
  WS --> LC{"sufficient local coverage?\n≥3 results @ score ≥0.7"}
  PS --> LC
  LC -- yes / scope≠all / no client --> OUT["return local results, kvarkSkipped"]
  LC -- no --> KV["searchKvark (tier-gated, graceful fail)"]
  KV --> CF["detectConflict(workspace, kvark)"]
  CF --> OUT2["return merged + hasConflict/conflictNote"]
```

---

## 9. Reconciliation & integrity (`reconcile.ts`)

A crash between frame insert and FTS/vec indexing leaves frames that exist but aren't searchable. The reconcile functions repair this (idempotent, cron-friendly):
- `reconcileFtsIndex(db)` — re-index frames missing from FTS5 (no embedder needed).
- `reconcileVecIndex(db, embedder)` — embed + re-index frames missing from the vec table (batches of 50).
- `cleanOrphanFts(db)` / `cleanOrphanVectors(db)` — drop FTS/vec rows whose frame was deleted.
- `reconcileIndexes(db, embedder?)` — runs all of the above; FTS-only if no embedder. Returns `{ ftsFixed, vecFixed }`.

---

## 10. Sessions (`SessionStore`)

`Session = { id, gop_id, project_id, status: 'active'|'closed'|'archived', started_at, ended_at, summary }`. `gop_id` format: `session:<ISO>:<rand6>`.

| Method | Purpose |
|---|---|
| `create(projectId?)` | new timestamped session |
| `ensureActive(projectId?)` | **transaction-wrapped** — returns existing active session or creates one (prevents twin-session race; used by cognify) |
| `ensure(gopId, …)` | idempotent named session (e.g. a stable `harvest` parent) |
| `close(gopId, summary?)` / `archive(gopId)` | lifecycle |
| `getByProject` / `getActive` / `getByGopId` | queries |

---

## 11. Schema reference (DDL, verbatim from `schema.ts`)

`SCHEMA_VERSION = '1'`. Tables relevant to this subsystem:

| Table | Key columns / constraints |
|---|---|
| `identity` | `id CHECK (id = 1)` (single row), name/role/department/personality/capabilities/system_prompt, created_at, updated_at |
| `awareness` | category `CHECK IN ('task','action','pending','flag')`, content, priority, `metadata` (JSON), created_at, expires_at |
| `sessions` | `gop_id UNIQUE`, project_id, status `CHECK IN ('active','closed','archived')`, started_at, ended_at, summary; index `(project_id, started_at)` |
| `memory_frames` | frame_type `CHECK IN ('I','P','B')`, gop_id (FK→sessions), t, base_frame_id (self-FK), content, importance `CHECK IN (critical/important/normal/temporary/deprecated)`, source `CHECK IN (user_stated/tool_verified/agent_inferred/import/system)`, access_count, created_at, last_accessed; indexes on (gop_id,t),(frame_type,gop_id),(base_frame_id) |
| `memory_frames_fts` | `CREATE VIRTUAL TABLE … USING fts5(content, content_rowid='id', tokenize='porter unicode61')` |
| `memory_frames_vec` | `CREATE VIRTUAL TABLE … USING vec0(embedding float[1024])` (separate `VEC_TABLE_SQL`, requires `sqlite-vec`) |
| `knowledge_entities` | entity_type, name, properties(JSON), valid_from, valid_to, recorded_at; indexes on type and name |
| `knowledge_relations` | source_id/target_id (FK→entities), relation_type, confidence(REAL), properties(JSON), valid_from, valid_to, recorded_at; indexes on (source_id,relation_type),(target_id,relation_type) |
| `embedding_usage` | (in `embedding-provider.ts`) user_id, year_month, count, updated_at; `UNIQUE(user_id, year_month)` — monthly quota counter |
| `harvest_sources` | source UNIQUE, display_name, source_path, last_synced_at, items_imported, frames_created, auto_sync, sync_interval_hours, last_content_hash, created_at |

> Note: the TS `FrameSource` union (`frames.ts`) includes `'personal' | 'workspace' | 'team_sync'` which the DB `CHECK` does **not** list — those extra sources are application-level and not written through the constrained column path.

---

## 12. HTTP surface (how the frontend reaches this)

These engine classes are **server-side only**; the frontend talks to the Fastify sidecar and the `hive-mind` MCP server, not to SQLite. The exact route paths are defined in `packages/server/src` (outside this section's read scope) — **do not invent them**. What this subsystem guarantees, and the MCP tool names that wrap it (from the `hive-mind` MCP server, visible in this environment), are:

| MCP tool | Wraps |
|---|---|
| `recall_memory` | `CombinedRetrieval.search` / `HybridSearch.search` |
| `save_memory` | `CognifyPipeline.cognify` |
| `save_entity` / `create_relation` / `search_entities` | `KnowledgeGraph` |
| `get_identity` / `set_identity` | `IdentityLayer` |
| `get_awareness` / `set_awareness` / `clear_awareness` | `AwarenessLayer` |
| `cleanup_frames` / `cleanup_entities` | `FrameStore.compact` / KG retire |
| `create_workspace` / `list_workspaces` | per-mind DB lifecycle |
| `harvest_import` / `harvest_sources` / `ingest_source` | harvest → `cognifyFrame`/`cognifyBatch` |

For the precise sidecar REST routes (method + path), consult the server-routes section of this backend map — they are the authoritative contract the Lovable frontend will call.