waggle-os/docs/backend-map/sections/05b-subsystem-memory.md

05b · Subsystem: Persistent Memory Engine (the MOAT)

Purpose. This is Waggle's durable, per-workspace memory substrate — the thing that makes agents remember across sessions. It is a single SQLite database (one file per "mind") layered into Identity → Awareness → Frames → Knowledge Graph, with hybrid search (vector + keyword fused), relevance scoring, index reconciliation, and a tiered embedding provider chain. Memory is WRITTEN by the CognifyPipeline (one frame per turn + entity/relation extraction) and RECALLED by CombinedRetrieval (workspace + personal + optional KVARK merge). For a frontend rebuild, treat this as a contract: you never touch SQLite directly — you call the sidecar HTTP routes that wrap these classes, and you render the typed shapes documented below.

Grounding: every type, field, table, and constant below is quoted from packages/hive-mind-core/src/mind/* and packages/agent/src/{cognify,combined-retrieval,memory-linker}.ts. Where a fact is NOT in those files (e.g. exact HTTP route paths), it is flagged explicitly.

---

1. Mental model — five layers in one SQLite file

A "mind" is one better-sqlite3 database (with the sqlite-vec extension loaded for the vector table). The schema (schema.ts, SCHEMA_VERSION = '1') defines these layers. Each layer is a TypeScript class wrapping prepared SQL statements — there is no ORM.

Layer	Class (file)	Table(s)	Cardinality	Role
0 — Identity	IdentityLayer (identity.ts)	identity	exactly 1 row (CHECK (id = 1))	Who the agent is: name/role/department/personality/capabilities/system_prompt. <500 tokens budget.
1 — Awareness	AwarenessLayer (awareness.ts)	awareness	MAX_ITEMS = 10 active	Short-term working state: active tasks, recent actions, pending items, context flags. Items can expire.
2 — Frames	FrameStore (frames.ts)	memory_frames (+ memory_frames_fts, memory_frames_vec)	unbounded	The long-term memory store. I/P/B frame types grouped by session (gop_id).
3 — Knowledge Graph	KnowledgeGraph (knowledge.ts)	knowledge_entities, knowledge_relations	unbounded	Entities + typed relations with temporal validity (bitemporal).
— Sessions	SessionStore (sessions.ts)	sessions	unbounded	Maps gop_id (Group-Of-Pictures id) → project. Parent of frames.

The "GOP" naming (I-frame / P-frame / B-frame, gop_id, t) is borrowed from video compression: an I-frame is a self-contained keyframe (full state), a P-frame is a delta/update against its base I-frame, a B-frame is a bidirectional cross-reference frame. t is a per-session monotonic sequence number (nextT = MAX(t)+1 for that gop_id).

graph TD
  subgraph MindDB["One SQLite file per workspace (a 'mind')"]
    ID[identity · 1 row]
    AW["awareness · ≤10 items"]
    SESS[sessions]
    MF[memory_frames]
    FTS["memory_frames_fts (FTS5)"]
    VEC["memory_frames_vec (vec0 · float[1024])"]
    KE[knowledge_entities]
    KR[knowledge_relations]
  end
  SESS -->|gop_id FK| MF
  MF -->|rowid = id| FTS
  MF -->|rowid = id| VEC
  KE -->|source_id / target_id| KR

---

2. Layer 2 — Frames (FrameStore)

The heart of the store. A MemoryFrame is:

Field	Type	Notes
id	number	PK / rowid; same id used in FTS + vec tables
frame_type	'I' | 'P' | 'B'	FrameType
gop_id	string	session id this frame belongs to
t	number	per-gop_id monotonic sequence (nextT)
base_frame_id	number | null	P/B frames point at their base I-frame
content	string	the actual text
importance	'critical' | 'important' | 'normal' | 'temporary' | 'deprecated'	Importance
source	'user_stated' | 'tool_verified' | 'agent_inferred' | 'import' | 'system' | 'personal' | 'workspace' | 'team_sync'	FrameSource (TS union is wider than the DB CHECK, which only allows the first five)
access_count	number	incremented by touch() on every recall/dup-hit
created_at	string	ISO; harvest path can override to preserve source timestamp
last_accessed	string	ISO; drives temporal scoring

Write methods

• createIFrame(gopId, content, importance='normal', source='user_stated', createdAt?) — dedup-guarded: calls findDuplicate(content) first; if an identical frame exists it touch()es it and returns it instead of inserting. createdAt is honored only if it passes isValidIsoTimestamp (strict ISO-8601 with T + timezone) — used by the harvest path so imported frames keep their original timestamp.
• createPFrame(gopId, content, baseFrameId, …) — a delta against an I-frame.
• createBFrame(gopId, content, baseFrameId, referencedFrameIds[]) — stores {description, references} as JSON in content.

Every create also runs indexFts(frame) to mirror content into the FTS5 table. Vector indexing is NOT done here — it happens in HybridSearch.indexFrame(), called by CognifyPipeline (see §6).

Dedup (findDuplicate) — important quirks for the frontend

• Hash = SHA-256( stripHmPrefix(content).trim() ).
• stripHmPrefix removes a leading [hm session:… src:… event:…] provenance prefix so two captures of the same turn from different sources collapse into one frame.
• Only the last 500 frames are scanned (cost bound). Dedup is best-effort beyond the recency window.

Read / list methods

Method	Returns
getById(id)	one frame
getLatestIFrame(gopId)	newest I-frame in a session
getPFramesSinceLastI(gopId)	P-frames after the latest I-frame
getGopFrames(gopId)	all frames in a session, t ASC
reconstructState(gopId)	{ iframe, pframes } — current state = latest I + its P-deltas
getRecent(limit=50) / list({limit})	newest frames, id DESC
getRecentFiltered(limit, since?, until?)	F20: date-bounded recent frames
getStats()	{ total, byType, byImportance }
update(id, content, importance?)	updates main + FTS + clears vec entry
delete(id)	removes from main + FTS + vec + kg_entity_frames and nulls referring base_frame_id

Compaction (compact(maxTempAgeDays=30, maxDeprecatedAgeDays=90))

Maintenance op: deletes old temporary and deprecated frames, and for any gop_id with >10 P-frames merges all-but-the-5-most-recent P-frames into their I-frame (joined with \n---\n). Returns { temporaryPruned, deprecatedPruned, pframesMerged }.

---

3. Hybrid Search (HybridSearch) — vector + keyword fusion

search(query, options) runs keyword and vector searches in parallel, fuses with Reciprocal Rank Fusion (RRF), then multiplies by a relevance score.

SearchOptions:

Field	Type	Default
limit	number	20
gopId	string?	(scope to one session)
profile	'balanced' | 'recent' | 'important' | 'connected'	'balanced'
context	ScoringContext	{}
since / until	string? (ISO)	temporal filter on created_at

SearchResult:

Field	Type	Meaning
frame	MemoryFrame	the hit
rrfScore	number	fused rank score
relevanceScore	number	from computeRelevance
finalScore	number	rrfScore * relevanceScore — the sort key

Fusion algorithm (the actual constants)

1. Run keywordSearch(query, limit*2, gopId) and vectorSearch(query, limit*2, gopId) in parallel; each returns an ordered number[] of frame ids.
2. RRF with RRF_K = 60: each id accrues 1 / (RRF_K + rank) from each list.
3. Fetch the union of frame ids (applying since/until filters here), compute relevanceScore, set finalScore = rrfScore * relevanceScore, sort desc, slice to limit.

Keyword path (FTS5) — recall tuning

• Query is tokenized, punctuation stripped, a built-in stop-word list removed, tokens shorter than 3 chars dropped, then OR-joined ("foo" OR "bar") for recall. FTS5 ORDER BY rank.
• On FTS5 parse error it falls back to likeFallbackSearch — OR-ed LIKE … ESCAPE '\' over content (parameterized, metachars escaped). This guarantees a query never returns a false "no memory found" because the user typed an FTS5 operator.

Vector path (sqlite-vec vec0)

• vectorSearch embeds the query (embedder.embed), converts the Float32Array to a blob, and does a MATCH ? AND k = ? KNN query on memory_frames_vec (ORDER BY distance).
• When gopId is set it over-fetches (k = limit*3) then filters by session.
• The vec table is float[1024] — embeddings MUST be 1024-dim (see §7).
• All vec operations are wrapped in try/catch returning [] — if the sqlite-vec extension or table is absent, search silently degrades to keyword-only.

Indexing (called by cognify, not by FrameStore)

• indexFrame(frameId, content) — embed + INSERT INTO memory_frames_vec. (rowid is inlined as a SQL literal because vec0 doesn't accept a parameterized rowid.)
• indexFramesBatch(frames[]) — batch embed + transactional insert.

---

4. Relevance scoring (scoring.ts)

computeRelevance(frame, weights, context) = weighted sum of four sub-scores. Profiles pick the weights:

Profile	temporal	popularity	contextual	importance
balanced	0.4	0.2	0.2	0.2
recent	0.6	0.1	0.2	0.1
important	0.1	0.1	0.2	0.6
connected	0.1	0.1	0.6	0.2

Sub-scores:

• temporal — 1.0 if last_accessed within RECENCY_BOOST_DAYS = 7, else exponential decay with HALF_LIFE_DAYS = 30.
• popularity — 1 + log10(1 + access_count) * 0.1.
• contextual — graph proximity: distance 0→1.0, 1→0.7, 2→0.4, 3→0.2, else 0 (needs context.graphDistances, a Map<frameId, BFS-distance>).
• importance — critical 2.0 / important 1.5 / normal 1.0 / temporary 0.7 / deprecated 0.3.

ScoringContext = { recentEntityIds?: number[]; graphDistances?: Map<number, number> }.

---

5. Knowledge Graph (KnowledgeGraph)

Bitemporal entity-relation store. Entity and Relation both carry valid_from / valid_to (null = currently valid) plus recorded_at.

Entity: { id, entity_type, name, properties(JSON string), valid_from, valid_to, recorded_at } Relation: { id, source_id, target_id, relation_type, confidence(REAL), properties(JSON), valid_from, valid_to, recorded_at }

Key methods:

Method	Purpose
createEntity(type, name, props, temporal?)	validates against optional schema, inserts
getEntitiesByType(type, limit=500) / getEntities(limit, offset)	active entities (valid_to IS NULL)
searchEntities(query, limit=100)	name LIKE (escaped)
getEntityTypeCounts() / getEntityCount()	dashboard counts without full fetch
getEntitiesValidAt(isoTime)	time-travel: entities valid at a past instant
createRelation(src, tgt, type, confidence=1.0, props)	validated insert
getRelationsFrom(id, type?) / getRelationsTo(id, type?)	adjacency
retireEntity(id) / retireRelation(id)	sets valid_to = now (soft-delete, never hard delete)
traverse(startId, relationType, maxDepth)	BFS returning reached entities
bfsDistances(startId, maxDepth)	Map<entityId, distance> — feeds the contextual score

Optional setValidationSchema(ValidationSchema) enforces required properties per entity-type and an allowed-relations list (throws on violation). Without a schema, all writes pass.

---

6. WRITE path — CognifyPipeline (cognify.ts)

cognify(content, importance='normal', gopId?, turnId?) is the canonical "remember this turn" call. Steps:

1. Ensure session — sessions.ensureActive() (transaction-wrapped to avoid the twin-session race).
2. Save frame — if a latest I-frame exists for the session → createPFrame (delta), else createIFrame (keyframe). Dedup applies inside the FrameStore.
3. Extract entities — extractEntities(content.slice(0, 10_000)) (from entity-extractor.ts).
4. Upsert entities into the KG (skip if same type+name exists; cached per-type to avoid N queries).
5. Co-occurrence relations — co_occurs_with (confidence 0.8) between every entity pair found in the same text.
6. Semantic relations — extractRelations → typed relations (led_by, reports_to, depends_on, …) matched back to KG entities.
7. Vector index — search.indexFrame(frame.id, content).
8. Optional linking — if enableLinking, MemoryLinker.findRelated(content) returns related frames (self excluded).

CognifyResult = { frameId, entitiesExtracted, relationsCreated, relatedFrames? }.

Other entry points: cognifyFrame(frameId) (re-process one imported frame — used post-harvest) and cognifyBatch(frameIds[]) (sequential, so each frame's new entities can link to the next).

MemoryLinker (memory-linker.ts)

Thin wrapper over HybridSearch.search. findRelated(content, limit=5) returns MemoryLink[] = { frameId, content, score }, filtered by a threshold (default 0.1 on finalScore).

flowchart TD
  A["agent turn / harvest / MCP save_memory"] --> B["CognifyPipeline.cognify(content)"]
  B --> C["SessionStore.ensureActive() → gop_id"]
  B --> D{latest I-frame exists?}
  D -- no --> E["FrameStore.createIFrame (dedup-guarded)"]
  D -- yes --> F["FrameStore.createPFrame"]
  E --> G["indexFts (FTS5)"]
  F --> G
  B --> H["extractEntities → KnowledgeGraph.upsert"]
  H --> I["co_occurs_with + semantic relations"]
  B --> J["HybridSearch.indexFrame → memory_frames_vec (1024-dim)"]
  B --> K["MemoryLinker.findRelated (optional)"]

---

7. Embeddings — provider chain (embedding-provider.ts)

createEmbeddingProvider(config?) returns an EmbeddingProviderInstance (implements the Embedder interface: embed, embedBatch, dimensions). Default targetDimensions = 1024 — matches the vec0 float[1024] table.

Auto fallback chain (provider: 'auto'): inprocess → ollama → voyage → openai → mock. Each is probed with a 1024-dim test embedding; the first that succeeds becomes active.

Provider (EmbeddingProviderType)	Default model	Needs
inprocess	Xenova/all-MiniLM-L6-v2 (Transformers.js)	nothing — fully local
ollama	nomic-embed-text	local Ollama server
voyage	voyage-3-lite	voyage.apiKey (from Vault)
openai	text-embedding-3-small	openai.apiKey
litellm	text-embedding	litellm.url
mock	deterministic-mock	always available; semantically meaningless (last resort)

Tier gating: provider availability is gated by TIER_CAPABILITIES[tier].embeddingProviders; embeddingQuotaPerMonth is enforced per user_id per month in the embedding_usage table (-1 = unlimited). Quota throws EmbeddingQuotaExceededError (carries tier/quota/current/upgradeUrl); over-tier provider request throws TierError. WAGGLE_EVAL_MODE=1 disables all gating (eval harness only). getStatus() / getQuotaStatus() / reprobe() expose state for a settings UI.

Frontend note: if the active provider is mock, surface a "semantic search degraded" warning — getStatus().activeProvider === 'mock' and lastError tell you. Mismatched embedder dimensions would break the vec table, so the provider hard-asserts 1024 on probe.

---

8. RECALL path — CombinedRetrieval (combined-retrieval.ts)

The merge engine the agent calls to answer "what do I know about X". Merges workspace + personal memory and optionally KVARK enterprise search. Pure data in / out (no formatting).

search(query, opts) → CombinedRetrievalResult:

Field	Type
query	string
workspaceResults / personalResults / kvarkResults	CombinedResult[]
kvarkAvailable	boolean
kvarkSkipped	boolean (available but coverage was sufficient)
kvarkError?	string
hasConflict	boolean
conflictNote?	string

CombinedResult = { content, source: 'workspace'|'personal'|'kvark', attribution, score, metadata } where attribution is a human tag like [workspace memory] / [personal memory], and metadata carries frameId/frameType/importance (memory) or documentId/documentType (KVARK).

CombinedSearchOptions = { limit=10, profile='balanced', scope: 'all'|'personal'|'workspace', turnId? }.

KVARK gating logic (shouldQueryKvark): KVARK is queried only when a client exists, scope==='all', AND local coverage is insufficient — hasSufficientLocalCoverage = fewer than LOCAL_COVERAGE_MIN_COUNT = 3 results with score ≥ LOCAL_COVERAGE_SCORE_THRESHOLD = 0.7. KVARK failures degrade gracefully (local results preserved, kvarkError set).

Conflict detection (detectConflict): if both workspace and KVARK have strong results (score ≥ CONFLICT_SCORE_THRESHOLD = 0.6) and their top-3 texts disagree on status polarity (POSITIVE_STATUS words like approved/selected vs NEGATIVE_STATUS like rejected/cancelled), it returns a human-readable conflictNote for the UI to surface ("these sources may be out of sync").

flowchart TD
  Q["CombinedRetrieval.search(query, scope)"] --> WS["searchWorkspace → HybridSearch"]
  Q --> PS["searchPersonal → HybridSearch"]
  WS --> LC{"sufficient local coverage?\n≥3 results @ score ≥0.7"}
  PS --> LC
  LC -- yes / scope≠all / no client --> OUT["return local results, kvarkSkipped"]
  LC -- no --> KV["searchKvark (tier-gated, graceful fail)"]
  KV --> CF["detectConflict(workspace, kvark)"]
  CF --> OUT2["return merged + hasConflict/conflictNote"]

---

9. Reconciliation & integrity (reconcile.ts)

A crash between frame insert and FTS/vec indexing leaves frames that exist but aren't searchable. The reconcile functions repair this (idempotent, cron-friendly):

• reconcileFtsIndex(db) — re-index frames missing from FTS5 (no embedder needed).
• reconcileVecIndex(db, embedder) — embed + re-index frames missing from the vec table (batches of 50).
• cleanOrphanFts(db) / cleanOrphanVectors(db) — drop FTS/vec rows whose frame was deleted.
• reconcileIndexes(db, embedder?) — runs all of the above; FTS-only if no embedder. Returns { ftsFixed, vecFixed }.

---

10. Sessions (SessionStore)

Session = { id, gop_id, project_id, status: 'active'|'closed'|'archived', started_at, ended_at, summary }. gop_id format: session:<ISO>:<rand6>.

Method	Purpose
create(projectId?)	new timestamped session
ensureActive(projectId?)	transaction-wrapped — returns existing active session or creates one (prevents twin-session race; used by cognify)
ensure(gopId, …)	idempotent named session (e.g. a stable harvest parent)
close(gopId, summary?) / archive(gopId)	lifecycle
getByProject / getActive / getByGopId	queries

---

11. Schema reference (DDL, verbatim from schema.ts)

SCHEMA_VERSION = '1'. Tables relevant to this subsystem:

Table	Key columns / constraints
identity	id CHECK (id = 1) (single row), name/role/department/personality/capabilities/system_prompt, created_at, updated_at
awareness	category CHECK IN ('task','action','pending','flag'), content, priority, metadata (JSON), created_at, expires_at
sessions	gop_id UNIQUE, project_id, status CHECK IN ('active','closed','archived'), started_at, ended_at, summary; index (project_id, started_at)
memory_frames	frame_type CHECK IN ('I','P','B'), gop_id (FK→sessions), t, base_frame_id (self-FK), content, importance CHECK IN (critical/important/normal/temporary/deprecated), source CHECK IN (user_stated/tool_verified/agent_inferred/import/system), access_count, created_at, last_accessed; indexes on (gop_id,t),(frame_type,gop_id),(base_frame_id)
memory_frames_fts	CREATE VIRTUAL TABLE … USING fts5(content, content_rowid='id', tokenize='porter unicode61')
memory_frames_vec	CREATE VIRTUAL TABLE … USING vec0(embedding float[1024]) (separate VEC_TABLE_SQL, requires sqlite-vec)
knowledge_entities	entity_type, name, properties(JSON), valid_from, valid_to, recorded_at; indexes on type and name
knowledge_relations	source_id/target_id (FK→entities), relation_type, confidence(REAL), properties(JSON), valid_from, valid_to, recorded_at; indexes on (source_id,relation_type),(target_id,relation_type)
embedding_usage	(in embedding-provider.ts) user_id, year_month, count, updated_at; UNIQUE(user_id, year_month) — monthly quota counter
harvest_sources	source UNIQUE, display_name, source_path, last_synced_at, items_imported, frames_created, auto_sync, sync_interval_hours, last_content_hash, created_at

Note: the TS FrameSource union (frames.ts) includes 'personal' | 'workspace' | 'team_sync' which the DB CHECK does not list — those extra sources are application-level and not written through the constrained column path.

---

12. HTTP surface (how the frontend reaches this)

These engine classes are server-side only; the frontend talks to the Fastify sidecar and the hive-mind MCP server, not to SQLite. The exact route paths are defined in packages/server/src (outside this section's read scope) — do not invent them. What this subsystem guarantees, and the MCP tool names that wrap it (from the hive-mind MCP server, visible in this environment), are:

MCP tool	Wraps
recall_memory	CombinedRetrieval.search / HybridSearch.search
save_memory	CognifyPipeline.cognify
save_entity / create_relation / search_entities	KnowledgeGraph
get_identity / set_identity	IdentityLayer
get_awareness / set_awareness / clear_awareness	AwarenessLayer
cleanup_frames / cleanup_entities	FrameStore.compact / KG retire
create_workspace / list_workspaces	per-mind DB lifecycle
harvest_import / harvest_sources / ingest_source	harvest → cognifyFrame/cognifyBatch

For the precise sidecar REST routes (method + path), consult the server-routes section of this backend map — they are the authoritative contract the Lovable frontend will call.