This commit is contained in:
Oleg Maslov
534
docs/transfer/MEMORY-SYSTEM-EXPLAINED.md
Normal file
534
docs/transfer/MEMORY-SYSTEM-EXPLAINED.md
Normal file
@@ -0,0 +1,534 @@
|
||||
# Waggle / hive-mind — The Memory System, Explained
|
||||
|
||||
**Audience:** A developer taking over Waggle OS who needs a working mental model of how the
|
||||
agent remembers things — fast enough to be productive, deep enough to not break invariants.
|
||||
|
||||
**Status:** Written 2026-06-26 against `packages/hive-mind-core/src/{mind,harvest}` (the
|
||||
substrate), `packages/agent/src` (cognify/recall wiring), `packages/server/src/local/routes`
|
||||
(REST), and `packages/memory-mcp/src` (MCP surface).
|
||||
|
||||
> ⚠️ **Path warning up front.** The older `docs/memory-architecture.md` (April 2026) and one
|
||||
> of the source-reader passes describe the substrate as living under `packages/core/src/mind/`.
|
||||
> **That path no longer exists.** The memory substrate was moved to
|
||||
> `packages/hive-mind-core/src/{mind,harvest}/` in the 2026-04-30 monorepo migration. Every
|
||||
> file reference in this document points at the real, current location. See §8 for the full
|
||||
> list of stale claims in the old doc.
|
||||
|
||||
---
|
||||
|
||||
## 1. Mental model
|
||||
|
||||
Waggle's memory is **one SQLite database per "mind"** (`better-sqlite3` + the `sqlite-vec`
|
||||
extension, WAL mode, foreign keys on). A *mind* is either the user's **personal** mind
|
||||
(always loaded) or a **workspace** mind (lazy-loaded when a workspace is active). Every memory
|
||||
layer — identity, awareness, the frame event-log, the knowledge graph, embeddings, full-text
|
||||
index, audit logs — lives as tables in that same file.
|
||||
|
||||
The unit of memory is a **frame**: an append-only, immutable record of something the agent
|
||||
learned or did. Frames are deduplicated by content hash, embedded into a 1024-dim vector index,
|
||||
mirrored into an FTS5 full-text index, and mined for typed **knowledge-graph entities**. At
|
||||
recall time, a 10-lane retrieval pipeline pulls the most relevant frames (vector + keyword +
|
||||
importance + date-window + raw-turn lanes), fuses and re-ranks them, and **appends the result
|
||||
to the system prompt** before the LLM ever sees the user's message. The whole thing runs twice
|
||||
per turn: **read before** (recall), **write after** (cognify).
|
||||
|
||||
```
|
||||
WRITE PATH
|
||||
┌──────────────┐ ┌──────────────┐ ┌────────────────────────────────────┐
|
||||
│ live turn │ │ pattern │ │ cognify(): │
|
||||
│ user+asst │──▶│ write-back │──▶│ • createIFrame/PFrame (dedup'd) │
|
||||
│ exchange │ │ (30+ regex) │ │ • extractEntities → KG upsert │
|
||||
└──────────────┘ └──────────────┘ │ • co-occurrence + semantic rels │
|
||||
│ • search.indexFrame() → vectors │
|
||||
┌──────────────┐ ┌──────────────┐ └──────────────────┬─────────────────┘
|
||||
│ bulk import │ │ 4-pass │ │
|
||||
│ (ChatGPT, │──▶│ harvest │──────────────────────┤
|
||||
│ Claude,PDF…) │ │ pipeline │ writes I/P/B frames │
|
||||
└──────────────┘ └──────────────┘ ▼
|
||||
┌────────────────────────────────────┐
|
||||
│ ONE SQLite DB (per mind) │
|
||||
│ memory_frames ── content_hash idx │
|
||||
│ memory_frames_fts (FTS5) │
|
||||
│ memory_frames_vec (sqlite-vec) │
|
||||
│ memory_frame_chunks(_vec) │
|
||||
│ knowledge_entities / _relations │
|
||||
│ identity / awareness / sessions │
|
||||
└────────────────────────────────────┘
|
||||
▲
|
||||
READ PATH │
|
||||
┌──────────────┐ ┌────────────────────────────────────┴───────────────┐
|
||||
│ user query │ │ recallMemory(): 10-lane pipeline │
|
||||
│ (turn start) │──▶│ importance · semantic(BM25+vec via RRF) · date- │
|
||||
└──────────────┘ │ window · profiles · facts · events · raw-detail · │
|
||||
│ catch-up · workspace/personal split │
|
||||
└──────────────┬─────────────────────────────────────┘
|
||||
▼
|
||||
cross-encoder rerank → scoring (recency/importance/popularity/context)
|
||||
▼
|
||||
dedup by frame-id → injection scan (all-or-nothing) → formatted text
|
||||
▼
|
||||
APPENDED to system prompt: [recalled memories] + [system prompt] + [user query]
|
||||
▼
|
||||
LLM
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. The core data model
|
||||
|
||||
### What a "frame" is
|
||||
|
||||
A `MemoryFrame` is one immutable row in `memory_frames`. It is the atom of memory. Three kinds:
|
||||
|
||||
| Frame type | Meaning | `base_frame_id` |
|
||||
|---|---|---|
|
||||
| **I** (Index) | Full snapshot / initial fact. One per GOP to start. | `null` |
|
||||
| **P** (Patch) | Incremental append referencing an I-frame; applied in `t` order. **String concatenation, not a real diff — P-frames can only add, never retract.** | the I-frame's id |
|
||||
| **B** (Branch) | Alternative to a frame without replacing it; allows divergent histories. Content serialized as JSON `{description, references[]}`. | the branched frame's id |
|
||||
|
||||
A **GOP** ("Group of Pictures", borrowed from video codecs) is a session/conversation; `gop_id`
|
||||
is a FK to `sessions.gop_id`, and `t` is a monotonic logical timestamp **within** that GOP.
|
||||
|
||||
Key columns: `id, frame_type, gop_id, t, base_frame_id, content, importance, source,
|
||||
access_count, created_at, last_accessed, content_hash, metadata`.
|
||||
|
||||
- **importance** ∈ `critical | important | normal | temporary | deprecated`
|
||||
- **source** ∈ `user_stated | tool_verified | agent_inferred | import | system` — these are the **only 5** the SQL `CHECK` permits (`schema.ts:57`). ⚠️ The TypeScript `FrameSource` type (`frames.ts:25`) over-declares 3 more (`personal | workspace | team_sync`) that the DB would **reject at insert**; `server/src/local/index.ts:1333` actually passes `'team_sync'` — a latent constraint-violation bug. Fix by widening the CHECK or correcting the call site.
|
||||
- **content_hash** = `SHA256(stripHmPrefix(content).trim())` — the provenance-insensitive dedup key
|
||||
- **metadata** = JSON `TEXT` (`NOT NULL DEFAULT '{}'`); never full-text indexed; carries the
|
||||
Phase-2B "Memory Center" contract (kind/status/scope/tags/confidence/trace_id/…)
|
||||
|
||||
### How the layers relate
|
||||
|
||||
| Layer | Table(s) | Shape | Role |
|
||||
|---|---|---|---|
|
||||
| **Frames** | `memory_frames` | append-only I/P/B log | The substrate of truth — everything learned |
|
||||
| **Vectors** | `memory_frames_vec`, `memory_frame_chunks_vec` | `vec0` virtual tables, **1024-dim** | Semantic search; `rowid` = frame id (or chunk id) |
|
||||
| **Full-text** | `memory_frames_fts` | FTS5 virtual table | Keyword (BM25) search; must stay in sync with frames |
|
||||
| **Chunks** | `memory_frame_chunks` (+ `_vec`) | per-frame paragraph chunks, `ON DELETE CASCADE` | Chunk-level retrieval for long frames |
|
||||
| **Knowledge graph** | `knowledge_entities`, `knowledge_relations` | typed nodes + directional typed edges, **bitemporal** | Structured distillation of frames (person/project/file/…) |
|
||||
| **Identity** | `identity` | single row, `CHECK(id=1)` | Who the agent is — injected into prompt, never a frame |
|
||||
| **Awareness** | `awareness` | ≤10 rolling items, expiry-aware | What it's working on *right now* — injected, never a frame |
|
||||
| **Sessions** | `sessions` | GOP → project container | Groups frames; `memory_frames.gop_id` → `sessions.gop_id` |
|
||||
| **Audit (append-only)** | `install_audit`, `ai_interactions` | DDL-triggered immutable | EU AI Act compliance; DB *refuses* DELETE/UPDATE |
|
||||
| **Concept mastery** | `concept_mastery` | spaced-repetition rows | Orthogonal learning tracker — **not linked** to frames or KG |
|
||||
|
||||
Identity and awareness are **out-of-band layers**: they format themselves into markdown via
|
||||
`toContext()` for prompt inclusion, but are never serialized as `MemoryFrame` records. Only
|
||||
memory *content* (facts, events, profiles, decisions) becomes frames.
|
||||
|
||||
---
|
||||
|
||||
## 3. The WRITE path
|
||||
|
||||
There are **two entry points** that produce frames: live cognify (during a turn) and bulk
|
||||
harvest (importing external corpora). Both converge on `FrameStore.createIFrame/createPFrame`
|
||||
and the same content-hash dedup.
|
||||
|
||||
### (a) Live cognify — during an agent turn
|
||||
|
||||
Fires **after** the agent loop succeeds (never on failure, so error traces don't become
|
||||
"ground truth"). Entry point in the chat route:
|
||||
|
||||
```
|
||||
packages/server/src/local/routes/chat.ts:1456
|
||||
const saved = await sessionOrch.autoSaveFromExchange(message, result.content, { traceId });
|
||||
```
|
||||
|
||||
1. **Pattern write-back** — `autoSaveFromExchange()` (`orchestrator.ts:839`) delegates to
|
||||
`runPatternWriteBack()` (`pattern-write-back.ts`), which runs **30+ calibrated regexes** over
|
||||
the (user, assistant) exchange:
|
||||
- Preferences (`"I prefer"`, `"call me"`, `"from now on"`), corrections (`"actually no"`,
|
||||
`"that's wrong"`), decisions (`"let's go with"`, `"decided to"`), findings
|
||||
(`"discovered that"`, `"turns out"`), style signals (`"keep it brief"`, `"bullets"`).
|
||||
- Casual chatter (`"lunch"`, `"weather"`, `"thanks"`) is **intentionally skipped**.
|
||||
- **Routing:** preferences/corrections/style → **personal** mind; decisions/findings/
|
||||
work-output → **workspace** mind (or personal if no workspace active).
|
||||
2. **Cognify pipeline** — each surviving snippet runs `cognify()` (`cognify.ts:52`):
|
||||
1. Frame creation: I-frame if none exists for the GOP, else P-frame.
|
||||
2. `extractEntities(content)` → persons/orgs/products/concepts.
|
||||
3. `upsertEntities()` → reuse existing entity id on case-insensitive **type+name** match, else create.
|
||||
4. Co-occurrence relations: all-pairs `co_occurs_with` (strength 0.8) among entities in the same text.
|
||||
5. Semantic relations: `extractRelations()` finds `led_by`/`reports_to`/`depends_on`/… and upserts edges.
|
||||
6. `search.indexFrame(frameId, content)` → embeds + writes the vector row.
|
||||
7. Optional `MemoryLinker.findRelated()` → related-frame links.
|
||||
3. **Signal commit** — `commitSurfacedSignals()` (`chat.ts:1447`) marks M8 awareness signals
|
||||
"surfaced" so they don't re-show next turn.
|
||||
|
||||
### (b) Bulk harvest — importing external corpora
|
||||
|
||||
`HarvestPipeline.run()` (`harvest/pipeline.ts:100`) converts ChatGPT/Claude/Gemini exports,
|
||||
Markdown, PDFs, and URLs into frames + KG entities via a 4-pass distillation:
|
||||
|
||||
```
|
||||
External source ─▶ Adapter ─▶ UniversalImportItem ─▶ Pass0..4 ─▶ DistilledKnowledge ─▶ Frames+KG
|
||||
```
|
||||
|
||||
- **Stage 1 — Adapter parsing.** Each source has an adapter (`claude-adapter.ts`,
|
||||
`markdown-adapter.ts`, `pdf-adapter.ts`, …) normalizing wildly different formats into a
|
||||
`UniversalImportItem` (`id, source, type, title, content, messages[]`). Adapters use
|
||||
`raw-types.ts` helpers (`asRecord`, `getString`) to safely narrow untrusted JSON.
|
||||
- **Stage 2 — Injection scan (Pass 0).** Every item is scanned (`pipeline.ts:111`) with
|
||||
`scanForInjection(probe, 'tool_output')` over title + first 4KB **before any LLM touches it.**
|
||||
Hostile items are dropped entirely.
|
||||
- **Stage 3 — 4-pass distillation:**
|
||||
- **Pass 1 Classify** (Haiku, cheap, batches of 20): assigns `value` (skip/low/medium/high)
|
||||
+ domain. `value='skip'` items (greetings, loops) never reach Pass 2.
|
||||
- **Pass 2 Extract** (Sonnet): decisions/preferences/facts/knowledge/entities/relations.
|
||||
- **Pass 3 Synthesize** (Sonnet): maps to `DistilledKnowledge` with `targetLayer`
|
||||
(identity/frame/kg_entity/kg_relation) + importance + confidence + provenance.
|
||||
- **Pass 4 Dedup** (local, no LLM — `dedup.ts:70`): see below.
|
||||
- **Stage 4 — Frame writing:**
|
||||
- `writeRawTurnFrames()` (`raw-turns.ts:112`): each user/assistant message stored as
|
||||
`[mind-rawturn conv:KEY turn:N speaker:S]` frame for adjacency lookups (system messages skipped).
|
||||
- `writeMemoryLaneFrames()` (`extract-memory-lanes.ts:288`): three parallel lanes —
|
||||
`[mind-fact]` (importance normal), `[mind-event] [YYYY-MM-DD]` (date resolved from a relative
|
||||
cue against the session date), `[mind-profile NAME]` (**replace-on-update** — old profile for
|
||||
that speaker deleted first, unlike facts which accumulate).
|
||||
- `writeKgEntities()` (`extract-kg-entities.ts:230`): exact-name dedup, bump `seen_count` on hit.
|
||||
- Every extraction output is **injection-scanned again** before write (verbatim dialogue is the
|
||||
most injection-prone surface).
|
||||
|
||||
### Deduplication (shared by both paths)
|
||||
|
||||
- **Frame-level (`FrameStore.findDuplicate`, `frames.ts:264`):** O(1) indexed lookup —
|
||||
`SELECT … WHERE content_hash = SHA256(stripHmPrefix(content).trim())`. If found, `touch()` the
|
||||
existing frame (bump access count) and skip insertion. `stripHmPrefix` removes the
|
||||
`[hm session:… src:…]` metadata prefix so the *same turn captured from two different sources
|
||||
collapses into one frame*. This hash is shared by insert, dedup lookup, update, compaction, and
|
||||
backfill so the semantics never drift.
|
||||
- **Harvest-level (`dedup.ts:70`):** two-tier — (1) exact SHA256 on normalized content, then
|
||||
(2) fuzzy trigram similarity (threshold 0.75) against existing content. Contradictions
|
||||
(similarity 0.4–0.75 + importance `important`) are **logged but still written** — the caller
|
||||
decides policy.
|
||||
|
||||
---
|
||||
|
||||
## 4. The READ path
|
||||
|
||||
`Orchestrator.recallMemory(query, limit, opts)` (`orchestrator.ts:453–831`) runs at the start
|
||||
of every turn (`chat.ts:767`) and returns `{ text, count, recalled, recalledFrames }`. It is
|
||||
**not** simple keyword search — it's a 10-lane pipeline:
|
||||
|
||||
1. **Catch-up detection.** Regex on the query (`"catch me up"`, `"where were we"`,
|
||||
`"what did we decide"`, `"brief me"`, …) → switches to an **importance-based fetch** instead
|
||||
of semantic search (catch-up should return *important* things, not semantically-close small
|
||||
talk). **This lane skips the reranker.**
|
||||
2. **Importance lane (K=5).** Critical/Important frames surface on *every* query.
|
||||
3. **Semantic search.** `HybridSearch.search()` runs FTS5 (BM25, stop-words stripped, OR-MATCH)
|
||||
and `vec0` vector distance **in parallel**, fused via **Reciprocal Rank Fusion (RRF, K=60)**.
|
||||
4. **Date-window lane.** If the query names a period (`"in May 2026"`), `parseDateWindow(query)`
|
||||
(`parse-date-window.ts`) produces `since`/`until` SQL filters.
|
||||
5. **Profile lane** — `[mind-profile]` frames.
|
||||
6. **Facts lane** — `[mind-fact]` frames, capped 60, oldest-first.
|
||||
7. **Events lane** — `[mind-event]` frames, chronological, capped 40; a dedicated "Events during
|
||||
X" section for windowed queries.
|
||||
8. **Raw-detail lane** — verbatim `[mind-rawturn]` excerpts, reranked via the cross-encoder
|
||||
(`raw-detail-lane.ts`, `inprocess-reranker.ts`).
|
||||
9. **Workspace / personal split** — active mind renders **first** (visual precedence;
|
||||
`orchestrator.ts:692` workspace, `:700` personal).
|
||||
10. **Injection scan (all-or-nothing)** — `scanForInjection(joinedLines, 'tool_output')`
|
||||
(`orchestrator.ts:777`). On a poisoned hit the **entire** recall returns empty. No partial recall.
|
||||
|
||||
**Fusion → rerank → scoring.** After RRF, results are re-ranked. The optional **cross-encoder
|
||||
reranker** (`inprocess-reranker.ts`) soft-fails back to RRF ordering if unavailable. Final
|
||||
ranking applies `computeRelevance()` (`scoring.ts`), four signals weighted by a **profile**
|
||||
(`balanced | recent | important | connected`):
|
||||
|
||||
| Signal | How it scores |
|
||||
|---|---|
|
||||
| **temporal** | 7-day full strength, ~30-day half-life decay |
|
||||
| **popularity** | `log10` of `access_count` (1000× difference ≈ 0.3 score delta) |
|
||||
| **contextual** | KG BFS distance 0/1/2/3 → 1.0/0.7/0.4/0.2 — **currently 0 in production** (see gotchas) |
|
||||
| **importance** | critical 2.0 / important 1.5 / normal 1.0 / temporary 0.7 / deprecated 0.3 |
|
||||
|
||||
`finalScore = rrfScore × relevanceScore`.
|
||||
|
||||
**Recall-context assembly & prompt injection.** The lanes are merged, deduped **by frame id**,
|
||||
the `TEMPORAL_GUIDANCE` constant is bundled in (`orchestrator.ts:808`), and the block is returned
|
||||
as text. The chat route then does:
|
||||
|
||||
```
|
||||
chat.ts:776 recalledContext = '\n\n' + recall.text;
|
||||
chat.ts:905 systemPrompt = … orch.buildSystemPrompt() … (assembled ? '' : recalledContext);
|
||||
```
|
||||
|
||||
producing the sandwich **`[recalled memories] + [system prompt] + [user query]`**. Crucially,
|
||||
**recall is NOT part of `buildSystemPrompt()`** — it's appended separately. The exception:
|
||||
if **PromptAssembler** is enabled, the assembler embeds recall internally (`chat.ts:887`) and the
|
||||
route skips the manual append.
|
||||
|
||||
`buildSystemPrompt()` itself (`orchestrator.ts:301`) assembles three *other* sections: identity
|
||||
(cached by content hash), self-awareness (improvement signals, uncached), and preloaded recent
|
||||
context (uncached).
|
||||
|
||||
---
|
||||
|
||||
## 5. Layers reference
|
||||
|
||||
### 5.1 Storage & schema
|
||||
- **Purpose:** the SQLite substrate — frames, embeddings, FTS, KG, audit, dedup, migrations.
|
||||
- **Files:** `mind/db.ts`, `mind/schema.ts`, `mind/frames.ts`, `mind/content-hash.ts`, `mind/chunker.ts`
|
||||
- **Key functions:** `MindDB` ctor (loads `sqlite-vec`, WAL, FK, `initSchema`), `MindDB.initSchema`
|
||||
(idempotent: SCHEMA_SQL + VEC_TABLE_SQL on fresh DB, else `runMigrations`), `MindDB.runMigrations`
|
||||
(crash-recovery + guarded `ADD COLUMN` + content-hash backfill + append-only triggers),
|
||||
`FrameStore.createIFrame/createPFrame/createBFrame`, `FrameStore.findDuplicate`,
|
||||
`FrameStore.update` / `delete` / `touch`, `hashFrameContent` / `stripHmPrefix`, `chunkText`.
|
||||
- **Chunking:** `chunkText()` (`chunker.ts:58`) splits on blank lines, greedily aggregates
|
||||
paragraphs to ≤2000 chars, sub-splits oversized paragraphs on sentence boundaries, applies
|
||||
200-char overlap, and preserves **absolute** `charStart`/`charEnd` relative to the parent frame.
|
||||
|
||||
### 5.2 Retrieval (hybrid search + scoring)
|
||||
- **Purpose:** fuse keyword + vector, re-rank by profile.
|
||||
- **Files:** `mind/search.ts`, `mind/scoring.ts`, `mind/inprocess-reranker.ts`,
|
||||
`mind/parse-date-window.ts`, `mind/raw-detail-lane.ts`, `mind/recall-context.ts`,
|
||||
`mind/resolve-relative-date.ts`
|
||||
- **Key functions:** `HybridSearch.search` (FTS5 + vec0 → RRF K=60 → `computeRelevance`),
|
||||
`HybridSearch.indexFrame` (**must** be called on every frame insert), `computeRelevance`,
|
||||
`parseDateWindow`.
|
||||
|
||||
### 5.3 Knowledge graph & semantic layers
|
||||
- **Purpose:** typed entities + temporal relations distilled from frames; canonicalization.
|
||||
- **Files:** `mind/knowledge.ts`, `mind/ontology.ts`, `mind/entity-normalizer.ts`,
|
||||
`mind/concept-tracker.ts`, `harvest/extract-kg-entities.ts`
|
||||
- **Key functions:** `KnowledgeGraph.findEntityByName` (**exact, case-sensitive** — the dedup
|
||||
primitive), `KnowledgeGraph.searchEntities` (fuzzy LIKE — *not* for dedup), `dedupeByName`
|
||||
(merge by `normalizeEntityName(name)::type`, survivor = most-relations / lowest-id, repoint
|
||||
edges, retire dups, sum `seen_count`), `traverse` (BFS one edge type), `bfsDistances`,
|
||||
`retireEntity` (soft-delete via `valid_to=now()`), `normalizeEntityName`
|
||||
(`js→javascript`, `postgres→postgresql`, `k8s→kubernetes`), `isNoiseName`,
|
||||
`ConceptTracker.recordAnswer`/`getDueForReview`.
|
||||
- **Bitemporal:** both entities and relations carry `valid_from`/`valid_to`; `valid_to IS NULL`
|
||||
= active. Nothing is hard-deleted; rows are *retired* for audit history.
|
||||
|
||||
### 5.4 Identity / Awareness / Sessions
|
||||
- **Purpose:** persistent agent identity, volatile working memory, and conversation containers —
|
||||
all feeding the prompt without being frames.
|
||||
- **Files:** `mind/identity.ts`, `mind/awareness.ts`, `mind/sessions.ts`,
|
||||
`agent/src/orchestrator.ts`, `agent/src/context-loader.ts`
|
||||
- **Key functions:** `IdentityLayer.update` (column allowlist for SQLi defense) / `toContext`,
|
||||
`AwarenessLayer.add` / `getAll` (≤10, priority DESC, expiry via SQL WHERE) / `updateMetadata`
|
||||
(shallow JSON merge) / `toContext`, `SessionStore.ensureActive` (transaction + `id DESC`
|
||||
tiebreaker for same-second races) / `ensure` (idempotent for harvest), `loadRecentContext`.
|
||||
- **Cross-workspace rule:** identity is **always** from the personal mind; awareness is **merged**
|
||||
(personal first); frames/KG switch to the workspace mind when one is active.
|
||||
|
||||
### 5.5 Embeddings
|
||||
- **Purpose:** multi-tier provider fallback with tier-gating, quota, and an embedder-lock. See §6.
|
||||
- **Files:** `mind/embedding-provider.ts`, `mind/embeddings.ts`, `mind/inprocess-embedder.ts`,
|
||||
`mind/ollama-embedder.ts`, `mind/api-embedder.ts`, `mind/litellm-embedder.ts`
|
||||
- **Key functions:** `createEmbeddingProvider`, `probeProvider`, `createInProcessEmbedder`,
|
||||
`ensureEmbeddingFingerprint` (the embedder-lock), `recreateVecTables` (destructive),
|
||||
`normalizeDimensions`, `maxEmbedCharsForModel`, `reembedPerText`.
|
||||
|
||||
### 5.6 Harvest / ingestion
|
||||
- **Purpose:** turn external conversations/docs into frames + KG via 4-pass distillation. See §3(b).
|
||||
- **Files:** `harvest/pipeline.ts`, `harvest/types.ts`, `harvest/dedup.ts`, `harvest/raw-turns.ts`,
|
||||
`harvest/extract-memory-lanes.ts`, `harvest/extract-kg-entities.ts`, plus per-source adapters
|
||||
(`claude-adapter.ts`, `chatgpt-adapter.ts`, `gemini-adapter.ts`, `markdown-adapter.ts`,
|
||||
`pdf-adapter.ts`, `url-adapter.ts`, `plaintext-adapter.ts`, `perplexity-adapter.ts`,
|
||||
`claude-code-adapter.ts`, `universal-adapter.ts`).
|
||||
- **Key functions:** `HarvestPipeline.run`, `dedup`, `writeRawTurnFrames`, `extractMemoryLanes`,
|
||||
`writeMemoryLaneFrames`, `extractKgEntities`, `writeKgEntities`.
|
||||
|
||||
### 5.7 Cognify (live write wiring)
|
||||
- **Purpose:** passively learn from each turn; the live read+write loop inside the agent.
|
||||
- **Files:** `agent/src/cognify.ts`, `agent/src/orchestrator.ts`, `agent/src/pattern-write-back.ts`,
|
||||
`agent/src/entity-extractor.ts`, `server/src/local/routes/chat.ts`
|
||||
- **Key functions:** `recallMemory`, `buildSystemPrompt`, `cognify`, `autoSaveFromExchange`,
|
||||
`commitSurfacedSignals`, `upsertEntities`, `createCoOccurrenceRelations`, `createSemanticRelations`.
|
||||
|
||||
### 5.8 API / MCP surface
|
||||
- **Purpose:** dual external surface — REST for the UI, MCP tools for Claude agents/clients.
|
||||
- **REST files:** `server/src/local/routes/memory.ts` (`/api/memory/search`, `/api/memory/frames`,
|
||||
`/api/memory/stats`), `memory-center.ts` (Phase-2B `/api/memory` CRUD + merge + archive + trace),
|
||||
`identity.ts` (`/api/identity`), `knowledge.ts` (`/api/memory/graph`).
|
||||
- **MCP files:** `memory-mcp/src/tools/{memory,identity,awareness,knowledge,wiki,harvest,workspace,
|
||||
cleanup,ingest}.ts`, `memory-mcp/src/resources/memory.ts`, `memory-mcp/src/core/setup.ts`.
|
||||
- **Key functions:** `normalizeToMemory` (projects a frame + its JSON metadata blob into the shared
|
||||
`Memory` entity), `save_memory` / `recall_memory` (MCP), `getWorkspaceMind` (lazy per-workspace
|
||||
`MindDB`), `POST /api/memory/:id/merge` (C11: concatenate + archive originals, never hard-delete).
|
||||
|
||||
---
|
||||
|
||||
## 6. Embedding provider chain & the embedder-lock rule
|
||||
|
||||
**Why it matters operationally:** if the embedding model changes, the vector index silently
|
||||
becomes meaningless or the DB refuses to boot. The system guards this aggressively.
|
||||
|
||||
**Resolution chain** (`embedding-provider.ts`, auto-probe in strict order, halts at first working):
|
||||
|
||||
```
|
||||
inprocess ─▶ ollama ─▶ voyage ─▶ openai ─▶ litellm ─▶ mock
|
||||
```
|
||||
|
||||
| Provider | What it is | Notes |
|
||||
|---|---|---|
|
||||
| **inprocess** | `Xenova/all-MiniLM-L6-v2` via `@huggingface/transformers` | 384 native dims → normalized to 1024; ~23 MB, cached in `~/.waggle/models/`, **offline** |
|
||||
| **ollama** | POST `localhost:11434/api/embed`, `nomic-embed-text` | 30 s timeout |
|
||||
| **voyage** | POST `api.voyageai.com`, `voyage-3-lite` | needs API key; 15 s timeout |
|
||||
| **openai** | POST `api.openai.com`, `text-embedding-3-small` | needs API key; 15 s timeout |
|
||||
| **litellm** | proxy `/v1/embeddings` | optional Bearer; no explicit timeout |
|
||||
| **mock** | deterministic byte-hash → Float32Array | **zero semantic value**, last resort |
|
||||
|
||||
- **Tier enforcement:** `TIER_CAPABILITIES[tier].embeddingProviders` gates which providers a user
|
||||
may reach; non-allowed providers are skipped during auto-probe. Only active when `userTier` is
|
||||
passed; `WAGGLE_EVAL_MODE=1` disables tier gates entirely (eval-harness measurement validity).
|
||||
- **Quota:** monthly count in `embedding_usage` (`user_id`, `year_month`, `count`); `checkQuota()`
|
||||
throws `EmbeddingQuotaExceededError`; resets at UTC month boundary.
|
||||
- **Char capping:** `maxEmbedCharsForModel()` caps inputs (6K default, 8K for `*-8k` models) by
|
||||
model-NAME heuristic — *not* measured context. (D1 probe finding: `nomic-bert` has a hard
|
||||
2048-token limit despite its name.)
|
||||
- **Batch resilience:** on a batch embed failure, `reembedPerText()` retries one input at a time
|
||||
and degrades only the failing inputs to mock — the rest keep real vectors.
|
||||
|
||||
### The embedder-lock (fingerprint guard)
|
||||
|
||||
`ensureEmbeddingFingerprint()` (`db.ts`) runs on the first vector op and records
|
||||
`{provider, model, dim}` in the `meta` table. On subsequent ops:
|
||||
|
||||
| Condition | Result |
|
||||
|---|---|
|
||||
| same dim, same model | OK (`match`) |
|
||||
| same dim, **different model** | update meta + **warn** (`model-changed`). Vectors stay numerically valid; cross-model semantic similarity is *degraded*, not undefined. |
|
||||
| **different dim** | **throws `EmbeddingDimMismatchError`** — `vec0` virtual tables cannot be `ALTER`ed. |
|
||||
|
||||
**Remediation for a dimension change is destructive:** `MindDB.recreateVecTables(newDim)` DROPs
|
||||
both vec tables and reinitializes, after which **every frame must be re-embedded.** Treat
|
||||
`1024` as effectively load-bearing.
|
||||
|
||||
---
|
||||
|
||||
## 7. Where it lives & how to call it
|
||||
|
||||
### File map
|
||||
|
||||
```
|
||||
packages/hive-mind-core/src/
|
||||
├── mind/ ← the substrate (per-mind SQLite + layers)
|
||||
│ ├── db.ts schema.ts MindDB, migrations, append-only triggers
|
||||
│ ├── frames.ts FrameStore (I/P/B CRUD, dedup, compaction)
|
||||
│ ├── content-hash.ts hashFrameContent / stripHmPrefix
|
||||
│ ├── chunker.ts semantic paragraph chunking
|
||||
│ ├── search.ts scoring.ts HybridSearch (RRF) + computeRelevance
|
||||
│ ├── inprocess-reranker.ts cross-encoder rerank (soft-fail)
|
||||
│ ├── parse-date-window.ts resolve-relative-date.ts temporal lanes
|
||||
│ ├── raw-detail-lane.ts recall-context.ts recall assembly
|
||||
│ ├── knowledge.ts ontology.ts entity-normalizer.ts KG + canonicalization
|
||||
│ ├── concept-tracker.ts spaced-repetition (orthogonal)
|
||||
│ ├── identity.ts awareness.ts sessions.ts out-of-band layers
|
||||
│ ├── embedding-provider.ts embeddings.ts *-embedder.ts provider chain
|
||||
│ ├── reconcile.ts multi-source reconciliation
|
||||
│ └── evolution-runs.ts execution-traces.ts improvement-signals.ts (subsystems)
|
||||
└── harvest/ ← ingestion (adapters + 4-pass pipeline)
|
||||
├── pipeline.ts dedup.ts types.ts raw-types.ts
|
||||
├── raw-turns.ts extract-memory-lanes.ts extract-kg-entities.ts
|
||||
└── *-adapter.ts claude / chatgpt / gemini / markdown / pdf / url / …
|
||||
|
||||
packages/agent/src/ ← live wiring into the turn
|
||||
orchestrator.ts (recallMemory, buildSystemPrompt, autoSaveFromExchange)
|
||||
cognify.ts pattern-write-back.ts entity-extractor.ts context-loader.ts
|
||||
|
||||
packages/server/src/local/routes/ ← REST surface for the UI
|
||||
memory.ts memory-center.ts identity.ts knowledge.ts
|
||||
|
||||
packages/memory-mcp/src/ ← MCP surface for Claude agents
|
||||
tools/{memory,identity,awareness,knowledge,wiki,harvest,workspace,cleanup,ingest}.ts
|
||||
resources/memory.ts core/setup.ts
|
||||
```
|
||||
|
||||
### How to call it
|
||||
|
||||
- **From an agent (MCP):** `save_memory(content, importance?, source?, workspace?)` →
|
||||
I-frame + index. `recall_memory(query, limit?, workspace?, scope?, profile?)` → hybrid search
|
||||
(`scope` ∈ personal/current/all). Also `get_identity`/`set_identity`, `get_awareness`/
|
||||
`set_awareness`, `search_entities`/`save_entity`, `harvest_import`, `compile_wiki`.
|
||||
- **From the UI (REST):** `GET /api/memory?status=active`, `GET /api/memory/search?q=…`,
|
||||
`POST /api/memory/frames`, `GET /api/memory/graph?scope=all|personal|workspace`,
|
||||
`GET/POST /api/identity`. Workspace selectable via `?workspace=id`; mutations require an
|
||||
explicit `?mind=personal|workspace` (typos fail loudly with 400 — never silently widened).
|
||||
- **Programmatically:** open a `MindDB(dbPath)`, then construct `FrameStore`, `HybridSearch`,
|
||||
`KnowledgeGraph` against `db.raw`. Lazy workspace minds come from `getWorkspaceMind(id)`
|
||||
(singleton per workspace via `MultiMindCache`).
|
||||
|
||||
---
|
||||
|
||||
## 8. Gotchas & footguns
|
||||
|
||||
### Cross-cutting invariants (break these and memory silently rots)
|
||||
- **Every `memory_frames` INSERT must call `HybridSearch.indexFrame`.** Writing the row directly
|
||||
bypasses FTS + vec; the frame becomes invisible to search and there is no cheap rebuild script.
|
||||
- **Vectors are locked at 1024-dim.** `vec0` tables can't be `ALTER`ed. A dimension change means
|
||||
DROP + recreate + re-embed *everything* (`recreateVecTables`). The guard throws early at search
|
||||
time, but the fix is destructive.
|
||||
- **Mock embedder = zero semantics.** If the provider chain falls through to `mock`, search still
|
||||
"works" but ranks randomly. Detect via `getStatus().activeProvider === 'mock'`.
|
||||
- **Model change at same dim is allowed but degrades cross-model similarity** (a warning, not an
|
||||
error). Don't swap embedding models casually on a populated DB.
|
||||
- **Injection scan is all-or-nothing on recall.** A single poisoned hit zeroes the entire recall
|
||||
block. Harvest drops hostile items entirely — no soft sanitization anywhere.
|
||||
- **`install_audit` and `ai_interactions` are physically append-only** via DDL `BEFORE
|
||||
DELETE/UPDATE` triggers that `RAISE(ABORT)` — the DB *refuses* mutation (EU AI Act Art. 12).
|
||||
Substrate changes confined to `install_audit` have **nowhere to land on the OSS mirror** — don't
|
||||
treat them as a pending port (see CLAUDE.md §7.5).
|
||||
|
||||
### Frame/dedup subtleties
|
||||
- **`stripHmPrefix` is load-bearing for dedup (OQ-6).** Same-body captures from different sources
|
||||
must collapse regardless of the `[hm …]` prefix; using only `.trim()` regresses this. The hash
|
||||
is shared across insert/lookup/update/compaction/backfill so semantics never drift.
|
||||
- **P-frames append, never retract.** Patch application is string concatenation, not a diff. To
|
||||
correct/retract, `update()` the I-frame or mark it `deprecated`.
|
||||
- **content_hash INDEX is created in `runMigrations()` (after a guarded `ADD COLUMN`), not in
|
||||
SCHEMA_SQL.** Creating it in SCHEMA_SQL crashes boot on pre-D3 DBs (2026-06-12 regression).
|
||||
- **`setMetadata()` does not invalidate FTS or vec** — metadata is never indexed.
|
||||
- **`createIFrame` accepts an optional ISO-8601 `createdAt`** (T separator + timezone). Invalid
|
||||
values fall back to `datetime('now')`; the harvest path validates + logs so exported timestamps
|
||||
preserve original ordering.
|
||||
|
||||
### Knowledge-graph subtleties
|
||||
- **Use `findEntityByName()` (exact) for dedup, never `searchEntities()` (fuzzy LIKE).** Fuzzy
|
||||
search drops the exact match from top-K once similar names accumulate — this caused **3506
|
||||
duplicate "Phase" rows** in the OSS repo before the fix.
|
||||
- **`dedupeByName` keys on name *and* type.** "Marko" (person) and "Marko" (project) never merge.
|
||||
- **Bitemporal soft-delete needs periodic cleanup.** `valid_to IS NULL` = active; nothing is hard-
|
||||
deleted, so `dedupeByName` / retiring must run on-demand or rows accumulate.
|
||||
- **The "connected"/contextual score is 0 in production (KNOWN GAP).** `bfsDistances()` computes
|
||||
entity→entity distances but there is **no entity-id → frame-id bridge**, so the contextual
|
||||
dimension contributes nothing today (`scoring.ts §12`). A `kg_entity_frames` cross-ref table is
|
||||
hinted at in `frames.delete()`'s try-catch but is **not in the schema**.
|
||||
- **Entity upsert is case-insensitive but type-specific.** `John`/`john` merge; `john`(PERSON) and
|
||||
`john`(PRODUCT) stay separate.
|
||||
- **`isNoiseName()` runs twice** (parser + write seam) — don't assume one pass is enough.
|
||||
|
||||
### Recall / multi-mind subtleties
|
||||
- **Recall is appended outside `buildSystemPrompt()`** (unless PromptAssembler is on). If you're
|
||||
hunting "why isn't memory in the prompt", look at `chat.ts:767/776`, not the orchestrator's
|
||||
prompt builder.
|
||||
- **Catch-up mode skips the reranker** and uses importance-fetch instead of semantic search.
|
||||
- **Workspace recall renders before personal** (visual precedence).
|
||||
- **Frame IDs collide across personal + workspace minds** (separate autoincrements). Multi-mind
|
||||
reads tag results with `_mind`/`_workspace_name`; mutations require an explicit `mind` parameter.
|
||||
- **Identity cache key must hash the full JSON, not `updated_at`** — SQLite datetime has 1-second
|
||||
precision, so rapid edits within a second collide (`orchestrator.ts:303`).
|
||||
- **`/api/memory/graph?scope=all` offsets IDs by +100k per workspace** to avoid collisions in the
|
||||
merged viz; recomputed every request (expensive for many workspaces).
|
||||
- **Awareness is hard-capped at 10** (`LIMIT 10` in every read). Add an 11th and the oldest
|
||||
silently drops on next read; expiry is via SQL WHERE, not a background job.
|
||||
|
||||
### Stale claims in the existing `docs/memory-architecture.md` (April 2026)
|
||||
That doc predates the 2026-04-30 migration and should be read with these corrections:
|
||||
1. **Wrong package path.** It says the substrate lives in `packages/core/src/mind/`. It does
|
||||
**not** — it lives in `packages/hive-mind-core/src/mind/`. (`packages/core/src/mind/` does not
|
||||
exist on disk as of 2026-06-26.) Its own "Path discrepancy fixed" note is itself now stale.
|
||||
2. **"Schema version 1" / "five layers" undercount.** The current schema has **10+ table groups**
|
||||
(frames, FTS, vec, chunks+chunk-vec, KG, identity, awareness, sessions, improvement signals,
|
||||
install audit, procedures, AI interactions, execution traces, evolution runs, harvest sources,
|
||||
concept mastery). The "five memory layers" framing is a useful teaching simplification, not the
|
||||
physical schema.
|
||||
3. **Reranker omitted.** The doc describes RRF + scoring but predates the cross-encoder reranker
|
||||
(`inprocess-reranker.ts`) that the raw-detail and semantic lanes now use (soft-fails to RRF).
|
||||
4. **Embedding chain understated.** It lists `inprocess → ollama → voyage → openai → mock`; the
|
||||
current chain also includes a **litellm** stage before mock, plus tier-gating and quota.
|
||||
|
||||
Otherwise the old doc's descriptions of RRF fusion, scoring signals, dedup, and the dual-mind
|
||||
model remain conceptually accurate — only the paths and the layer/provider counts have drifted.
|
||||
228
docs/transfer/PROJECT-TRANSFER-ANALYSIS-2026-06-26.md
Normal file
228
docs/transfer/PROJECT-TRANSFER-ANALYSIS-2026-06-26.md
Normal file
@@ -0,0 +1,228 @@
|
||||
# Project Transfer Analysis — Hive Mind + Waggle OS
|
||||
|
||||
**Date:** 2026-06-26
|
||||
**Author:** Transfer-audit synthesis (6 parallel scout passes)
|
||||
**Audience:** Incoming developer taking over the Waggle OS + Hive Mind ecosystem
|
||||
**Scope:** `D:/Projects/waggle-os` (+ 3 worktrees), `D:/Projects/hive-mind`, `D:/Projects/hive-mind-test`, `D:/Projects/hive-mind-clients`, `D:/Projects/claude-hive-mind`, plus developer-local stores under `~/.waggle`, `~/.mem0`, and the Claude Code project memory.
|
||||
**Rev:** Completeness-critic reviewed (2026-06-26). Corrected: vault classified as *recoverable* (not irreplaceable); Ollama/`nomic-embed-text` reclassified from "optional" to a **hard benchmark prerequisite**; `Desktop\MEMORIES` cross-referenced into the local-data section; recall-stress gitignore rationale clarified as by-design.
|
||||
|
||||
---
|
||||
|
||||
## 1. Executive Summary
|
||||
|
||||
The Hive Mind + Waggle OS ecosystem transfers as **source code cleanly, but NOT as a turnkey reproducible whole**. Every line of application code, schema, migration, and benchmark harness lives in git and re-clones cleanly. Infrastructure (PostgreSQL, Redis, MinIO) is fully regenerable via `docker-compose up`. Public benchmark datasets (LoCoMo, LongMemEval, BEAM) are downloadable, and two flagship benchmark numbers (LoCoMo SOTA 87.66 and LoCoMo v5 OSS 67.8%) are fully reproducible given API keys. **However**, the institutional memory (1.6 MB of hand-written session handoffs and the SOTA results index), the LLM-generated benchmark answers/judgments (dollars and researcher-hours to regenerate), the developer's `.mind` / `hive-mind.db` memory databases (embedder-locked, not cheaply regenerable), and a concurrent session's uncommitted WIP in the main repo are all **local-only and irreplaceable** — they must be physically copied. The encrypted **vault is a distinct case: machine-local but *recoverable*** — the new developer simply re-enters their own keys; the `.vault-key` itself never transfers. No API credentials transfer; the new developer must provision their own ~22 keys/accounts.
|
||||
|
||||
> **Reproduce-everything verdict: PARTIAL.**
|
||||
> - Code, infra, public datasets, and 2 of 5 benchmarks: YES.
|
||||
> - LongMemEval (N=500) and recall-stress: PARTIAL (engineering/proprietary-query gaps).
|
||||
> - GAIA2: NO (blocked on an unresolved `ARE SIGALRM` platform bug).
|
||||
> - Institutional context + LLM-generated results: only via physical copy (not regenerable cheaply).
|
||||
|
||||
---
|
||||
|
||||
## 2. Repository Map
|
||||
|
||||
| Repo / Worktree | Remote URL | Public/Private | Role | Clones clean? |
|
||||
|---|---|---|---|---|
|
||||
| `D:/Projects/waggle-os` | `https://github.com/marolinik/waggle-os.git` (confirmed `git remote -v`) | PRIVATE (MIT license in repo) | Main monorepo — Tauri desktop + web + Fastify sidecar, 27 workspace packages | **NO — concurrent session WIP**: 52 modified + 22 untracked files at HEAD `18aebe1f`, 0 unpushed commits. Do NOT stage/commit/clean. WIP is reproducible only by its owner. |
|
||||
| `D:/Projects/hive-mind` | `https://github.com/marolinik/hive-mind.git` | PUBLIC (Apache-2.0, v0.4.0) | OSS memory substrate mirror (`packages/core` = mind/ + harvest/) + `benchmarks/locomo` harness. HEAD `bc4eba1` | YES — clean, no uncommitted changes |
|
||||
| `D:/Projects/hive-mind-test` | `https://github.com/marolinik/hive-mind-test.git` | (v0.1.0) | LoCoMo SOTA benchmark harness + answers/judgments (the 87.66 evidence). HEAD `05f2146` | YES — clean, but precious local data inside (see §5) |
|
||||
| `D:/Projects/hive-mind-clients` | `https://github.com/marolinik/hive-mind-clients-archive.git` | PUBLIC (Apache-2.0, v0.1.0) | Per-IDE silent-capture shims. HEAD `5b41eb5` | YES — clean |
|
||||
| `D:/Projects/claude-hive-mind` | (check `.git/config`) | (likely older variant of hive-mind) | Older copy/variant; **contains 32 MB personal data fixture** | YES — but holds personal-data zip to scrub on transfer |
|
||||
| `D:/Projects/waggle-os-w4` | worktree of waggle-os, branch `feature/w4-port` | PRIVATE | SOTA paper artifacts (arXiv LaTeX + docx + team briefing). HEAD `d146e906` | Clean: 0 uncommitted, 0 unpushed |
|
||||
| `D:/Projects/waggle-os-gaia2-wt` | worktree of waggle-os, branch `feature/gaia2-are-setup` | PRIVATE | GAIA2 Phase 3 evaluation (HALT verdict). HEAD `08a63ba7` | **3 unpushed commits** — review/push before discarding |
|
||||
| `D:/Projects/waggle-os-harness-bench` | worktree of waggle-os, branch `feature/harness-sota-bench` | PRIVATE | Premium harness eval (21/21 pillars, 2657/2657 suite). HEAD `ea078769` | **NO_UPSTREAM** — branch tracks no remote; would be LOST if worktree discarded without pushing |
|
||||
| `D:/Projects/waggle-os-ux-prototype` | **not git-tracked** | n/a | Standalone Vite+React UX prototype (122 MB incl. node_modules) | **NOT under version control** — copy as-is or treat as ephemeral |
|
||||
|
||||
Worktree list verified live via `git worktree list` (gaia2-wt, harness-bench, w4 all attached to the main repo's `.git`).
|
||||
|
||||
---
|
||||
|
||||
## 3. Data Volumes & Databases
|
||||
|
||||
| Path | Size | Contents | Reproducible? | Transfer method |
|
||||
|---|---|---|---|---|
|
||||
| `packages/marketplace/marketplace.db` | 14 MB | SQLite catalog: 40 sources, 120+ packages, 18 packs, FTS5, scan_history, security_config | YES — `npm run sync` in packages/marketplace (re-fetches ClawHub/SkillsMP/GitHub/LobeHub) | Travels in git clone; or regenerate via sync |
|
||||
| Docker volume `pgdata` (`DATABASE_URL=postgres://waggle:waggle_dev@localhost:5434/waggle`) | Variable (empty fresh) | Team server: users/teams/agents/tasks/messages/team_entities (Drizzle ORM, PG16) | YES — `docker-compose up`, Drizzle migrations auto-run from `packages/server/drizzle/` | Regenerate; do not copy |
|
||||
| Docker volume `redisdata` (`redis://localhost:6381`) | Variable (empty fresh) | Session cache, BullMQ job queue | YES — ephemeral by design | Regenerate |
|
||||
| Docker volume `miniodata`/`minio-data` (`localhost:9000`, bucket `waggle-files`) | Variable (empty fresh) | S3-compatible object storage; `minio-init` auto-creates bucket | YES — `docker-compose up` | Regenerate |
|
||||
| `waggle-os/.mind/` | 47 KB | Local hive-mind memory for this workspace (KG + entity cache) | NO (session-local content) — schema auto-creates | Copy if context valued; else regenerates on first use |
|
||||
| `~/.waggle/hive-mind.db` (per-project MemoryStore) | Varies (LoCoMo trio ~100 MB+) | FrameStore + HybridSearch (FTS5 + 1024-d sqlite-vec) + KnowledgeGraph + Identity/Awareness | NO — local-only; vectors tied to embedder model | Physically copy; **not portable if embedder changes** |
|
||||
| `~/.waggle/vault.json` + `~/.waggle/.vault-key` | <10 KB | AES-256-GCM encrypted API keys (canonical secret store) | NO — `.vault-key` is machine-local, never backed up | New dev re-enters keys via UI; if `.vault-key` lost, all entries unrecoverable |
|
||||
| `~/.mem0/history.db` | 888 KB (≈909 KB) | Mem0 SDK conversation history (cross-project, personal) | NO | **Do NOT transfer** (personal) |
|
||||
| `~/.mem0/migrations_qdrant/` | 18 KB | Qdrant vectors (1536-d Cosine, `mem0migrations` collection) | NO | **Do NOT transfer** (personal) |
|
||||
| `.claude/projects/D--Projects-waggle-os/memory/` | 1.6 MB (~100–250 `.md`) | Session handoffs, SOTA index, decisions, founder feedback pins | NO — hand-written | **Physically copy** (heart of handoff) — see §5 |
|
||||
| `packages/server/drizzle/` | 2 SQL + meta (~120 KB) | `0000_wild_glorian.sql`, `0001_redundant_sauron.sql` + journal | YES — source-controlled | Travels in git |
|
||||
| `hive-mind-test/scripts/locomo/data/` | ~144 MB dir (2.8 MB `locomo10.json` + ~15 MB judgments) | LoCoMo test set + answers (packc/packd/mem0/ours/theirs) + trio judgments | Dataset YES (public); answers/judgments NO ($$ token spend) | Copy answers/judgments; dataset auto-fetched |
|
||||
| `waggle-os/benchmarks/results/*.jsonl` (+ `.summary.json`) | ~36 MB, 68 files | Per-instance harness results (accuracy/latency/cost/failure_mode) | NO — real LLM calls cost $$ | Physically copy |
|
||||
| `waggle-os-gaia2-wt/benchmarks/gaia2/runs/*/` | multi-run | GAIA2 execution traces + scenario `trace.jsonl` | NO — $$ LLM + judge calls | Physically copy |
|
||||
| `waggle-os/.understand-anything/` | 7.1 MB | KG (4,067 nodes / 6,521 edges) + fingerprints + dashboard | YES — `/understand --full` (~17.9M tokens, ~2h, costly) | Copy or regenerate |
|
||||
|
||||
---
|
||||
|
||||
## 4. What the Developer Can Get Themselves (download or regenerate)
|
||||
|
||||
### 4.1 Public source repos (git clone)
|
||||
```bash
|
||||
git clone https://github.com/marolinik/hive-mind.git
|
||||
git clone https://github.com/marolinik/hive-mind-clients-archive.git # hive-mind-clients
|
||||
git clone https://github.com/marolinik/hive-mind-test.git
|
||||
git clone https://github.com/marolinik/waggle-os.git # PRIVATE — needs GitHub access
|
||||
```
|
||||
All use npm/pnpm workspaces, Node >= 20. After clone: `npm install && npm run build` (waggle-os: `npm run build:all`).
|
||||
|
||||
### 4.2 Native deps & toolchain (regenerate)
|
||||
- **Node.js 20+ (LTS)** — https://nodejs.org/ ; verify `node --version`.
|
||||
- **better-sqlite3 v12.6.2** — compiles via node-gyp on `npm install`; Windows needs **MSVC Build Tools + Python 3.x**.
|
||||
- **sqlite-vec-windows-x64 v0.1.9** — optionalDependency; npm skips on non-Windows. Source: https://github.com/asg017/sqlite-vec-releases
|
||||
- **Playwright browser** — `npx playwright install` (Chromium) on first E2E run.
|
||||
- **Local embedder** — `Xenova/all-MiniLM-L6-v2` (~23 MB) downloads to `~/.waggle/models/` on first `EMBEDDING_PROVIDER=local` boot.
|
||||
|
||||
### 4.3 Local infrastructure (regenerate)
|
||||
```bash
|
||||
docker-compose up -d # postgres :5434, redis :6381, minio :9000 (+ minio-init bucket)
|
||||
# Drizzle migrations auto-run on server start; force re-apply:
|
||||
docker volume rm waggle-os_pgdata && docker-compose up -d postgres
|
||||
```
|
||||
**Optional — LiteLLM proxy** (`litellm-proxy --config litellm-config.yaml --port 4000`): the app falls back to its built-in Anthropic proxy / Ollama if absent.
|
||||
|
||||
**REQUIRED for any hive-mind benchmark — Ollama** (NOT optional): `ollama pull nomic-embed-text`, then serve on `localhost:11434`. The embedder is **load-bearing** — the `.mind` / `hive-mind.db` vectors are 1024-d nomic; running a benchmark against a DB embedded with a *different* model silently corrupts retrieval (no error, just wrong results). Treat this as a hard prerequisite alongside API keys.
|
||||
|
||||
### 4.4 Public benchmark datasets (download + canonicalize)
|
||||
| Dataset | Fetch command | Then canonicalize |
|
||||
|---|---|---|
|
||||
| LoCoMo raw | `curl -L 'https://raw.githubusercontent.com/snap-research/locomo/main/data/locomo10.json' -o benchmarks/data/locomo10.json` (SHA-256 of canonical archive: `39e415e2…2a5b24`; raw `locomo10.json` pinned `79fa87e9…698ff4`) | `cd benchmarks/harness && tsx scripts/build-locomo-canonical.ts` → `locomo/locomo-1540.jsonl` (1531 instances) |
|
||||
| LongMemEval S | `curl -L 'https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned/resolve/main/longmemeval_s_cleaned.json' -o benchmarks/data/longmemeval_s_cleaned.json` (264 MB, SHA `d6f21ea9…c3a442`) | `tsx scripts/build-longmemeval-canonical.ts --variant s` (500 instances) |
|
||||
| BEAM | `git clone https://github.com/mohammadtavakoli78/BEAM.git` (parent dir) | `tsx scripts/build-beam-canonical.ts --beam-chats-path /path/to/BEAM/chats --chat-size 128K` (400 instances) |
|
||||
|
||||
Committed (no download needed): `preflight-locomo-50.json`, `failure-mode-calibration-10.jsonl`, all `*.meta.json`, harness config `benchmarks/harness/config/datasets.json`, build scripts `benchmarks/harness/scripts/build-*.ts`. The hive-mind OSS LoCoMo number is **free** offline: `cd D:/Projects/hive-mind/benchmarks/locomo && npm run build && node rescore.mjs` (zero API cost; verifies SHA-256-pinned artifacts).
|
||||
|
||||
---
|
||||
|
||||
## 5. Local-Only / IRREPLACEABLE Data (must physically copy)
|
||||
|
||||
This is the heart of the handoff. None of the below comes back from a `git clone`.
|
||||
|
||||
### 5.1 Institutional memory (highest priority)
|
||||
- **`C:/Users/MarkoMarkovic/.claude/projects/D--Projects-waggle-os/memory/`** (1.6 MB, ~100–250 `.md`). Single source of truth for what shipped / what's open / how to roll back. Includes `MEMORY.md` (START HERE index, daily), `project_session_handoff_*.md` (150+ sessions Apr–Jun 2026), `project_sota_results_index.md` (binding LoCoMo 87.66 evidence with commit SHAs + reproduction recipe), tier/KVARK/AI-Act strategy pins, and founder feedback pins. **Cannot regenerate** — copy the directory; or extract key insights into committed `docs/`.
|
||||
|
||||
### 5.2 LLM-generated benchmark answers/judgments/results ($ + time to regenerate)
|
||||
- **`hive-mind-test/scripts/locomo/data/{answers,judgments}/`** — the 87.66 SOTA evidence (packc/packd/mem0/ours/theirs answers, trio-strict judgments). Regenerable only with API keys + ~2h compute + token spend.
|
||||
- **`waggle-os/benchmarks/results/*.jsonl` + `*.summary.json`** (~36 MB, 68 files) — per-cell ablation results; each row is a paid LLM call. Gitignored.
|
||||
- **`waggle-os/benchmarks/calibration/v6-kappa-recal/*.jsonl`** + **`benchmarks/probes/judge-swap-validation/*.jsonl`** — judge-reliability (Fleiss' kappa) outputs across deepseek/kimi/minimax/zhipu. No regeneration script.
|
||||
- **`waggle-os/benchmarks/results/pilot-2026-04-26/`** (~40 MB) — agentic pilot (FAIL verdict, kept for audit).
|
||||
- **`waggle-os/benchmarks/results/v6-self-judge-rebench/`** (~50 KB) — apples-to-apples vs Mem0.
|
||||
- **`waggle-os-gaia2-wt/benchmarks/gaia2/runs/*/results.jsonl` + `search/scenario_*/trace.jsonl`** — GAIA2 execution traces (real LLM + judge $$).
|
||||
- **`waggle-os-gaia2-wt/benchmarks/gaia2/data/tasks-mini-2.jsonl`** (4.8 MB) — GAIA2 task set with **no traced public source** (origin untraced; cannot re-fetch).
|
||||
|
||||
> **⚠ Expected-but-ABSENT (verify before relying on it):** `C:/Users/MarkoMarkovic/Desktop/MEMORIES/` — `CLAUDE.md` §10 (M2/M3) cites a 30 MB Claude export + 437 MB Gemini export "ready for E-11 ingestion", but **the directory does not exist on this disk** (see §9). If those harvest corpora matter to the handoff, re-export them from the source Claude/Google accounts — they are recoverable from *no* repo or backup found here.
|
||||
|
||||
### 5.3 Generated DBs worth keeping
|
||||
- **`~/.waggle/hive-mind.db`** (and any project `.mind` DBs) — harvested memories + KG + identity. 1024-d sqlite-vec vectors are **embedder-locked** (re-embedding with a different model silently invalidates retrieval).
|
||||
- **`~/.waggle/vault.json` + `.vault-key`** — only if continuing on the SAME machine; otherwise re-enter keys.
|
||||
|
||||
### 5.4 Unpushed / unversioned work that vanishes if ignored
|
||||
- **`waggle-os` main repo concurrent WIP** (52 modified + 22 untracked at `18aebe1f`) — owner must review+commit; do not clobber.
|
||||
- **`waggle-os-harness-bench` (NO_UPSTREAM)** — push the branch or it dies with the worktree.
|
||||
- **`waggle-os-gaia2-wt`** — 3 unpushed commits to review/push.
|
||||
- **`waggle-os-ux-prototype/`** (122 MB, not git) — add to git or treat as ephemeral.
|
||||
- **`waggle-os/.planning/`** (1.6 MB, gitignored) — planning snapshots/decision logs.
|
||||
|
||||
### 5.5 Personal — DO NOT transfer (privacy)
|
||||
- **`~/.mem0/` (`history.db` + `migrations_qdrant/` + `config.json`)** — founder's cross-project personal memory. `config.json` is auto-regenerated by the Mem0 SDK on first run, so nothing under `~/.mem0/` is worth (or safe to) transfer.
|
||||
- **`hive-mind/test-fixtures/claude-export-2026-04-22-marko.zip`** (32 MB) and its duplicate in **`claude-hive-mind/test-fixtures/`** — founder's personal Claude chat archive. Scrub before any public push.
|
||||
|
||||
---
|
||||
|
||||
## 6. Credentials & Accounts the Developer Must Obtain
|
||||
|
||||
Secrets live in the encrypted vault (`~/.waggle/vault.json`) and/or `.env`/`.env.locomo-trio` (gitignored). **Never transfer the founder's values — provision your own.** `.env.example` is the safe template.
|
||||
|
||||
| Key name | File / store | External service | Purpose |
|
||||
|---|---|---|---|
|
||||
| `ANTHROPIC_API_KEY` | vault / `.env` | console.anthropic.com | Claude (default model + judge) |
|
||||
| `OPENAI_API_KEY` | vault / `.env` | platform.openai.com | GPT-4o/o3, embeddings, eval judge |
|
||||
| `GEMINI_API_KEY` / `GOOGLE_API_KEY` | vault / `.env` | ai.google.dev | Gemini 2.5/3.1 (judge) |
|
||||
| `XAI_API_KEY` | vault / `.env` | console.x.ai | Grok (v6 judge) |
|
||||
| `DEEPSEEK_API_KEY` | vault / `.env` | platform.deepseek.com | DeepSeek chat/reasoner |
|
||||
| `PERPLEXITY_API_KEY` | vault / `.env` | perplexity.ai/settings/api | Sonar web-grounded |
|
||||
| `OPENROUTER_API_KEY` | vault / `.env` | openrouter.ai/keys | **Critical fallback** for 10+ routes |
|
||||
| `MISTRAL_API_KEY` | vault / `.env` | console.mistral.ai | Mistral Large/Small/Codestral |
|
||||
| `MOONSHOT_API_KEY` | vault / `.env` | platform.moonshot.ai (intl) | Kimi K2 family |
|
||||
| `DASHSCOPE_API_KEY` | vault / `.env` | dashscope-intl.aliyuncs.com | **Qwen3.6-35B-A3B (LOCKED default)** — intl tenant only |
|
||||
| `MINIMAX_API_KEY` (+ `MINIMAX_GROUP_ID`) | vault / `.env` | minimaxi.com | MiniMax M1/M2.7 (v6 judge) |
|
||||
| `ZHIPU_API_KEY` | vault / `.env` | open.bigmodel.cn | GLM-4-Plus/GLM-5 |
|
||||
| `GENSPARK_API_KEY` | vault / `.env` | genspark.ai | Claude-compatible proxy |
|
||||
| `TAVILY_API_KEY` | vault / `.env` | tavily.com | Web search (not LLM) |
|
||||
| `LITELLM_MASTER_KEY` | `.env` | local (placeholder `sk-waggle-dev`) | Auth for local LiteLLM proxy |
|
||||
| `CLERK_SECRET_KEY` / `CLERK_PUBLISHABLE_KEY` / `VITE_CLERK_PUBLISHABLE_KEY` | `.env` | dashboard.clerk.com | Team server auth (optional for single-user) |
|
||||
| `STRIPE_SECRET_KEY` / `STRIPE_WEBHOOK_SECRET` / `STRIPE_PRICE_PRO_MONTHLY`/`_ANNUAL`/`STRIPE_PRICE_TEAMS_MONTHLY`/`_ANNUAL` | `.env` | dashboard.stripe.com | Billing (optional for dev) |
|
||||
| `DATABASE_URL` | `.env` | local PG (docker) | `postgres://waggle:waggle_dev@localhost:5434/waggle` |
|
||||
| `REDIS_URL` | `.env` | local Redis (docker) | `redis://localhost:6381` |
|
||||
| `MINIO_ENDPOINT`/`MINIO_ACCESS_KEY`/`MINIO_SECRET_KEY`/`MINIO_BUCKET` | `.env` | local MinIO (docker) | `localhost:9000`, `waggle-files` (dev creds `waggle`/`waggle_s3_dev`) |
|
||||
| `OLLAMA_URL` / `OLLAMA_MODEL` | `.env.locomo-trio` | local Ollama | `localhost:11434` + `nomic-embed-text` — **must match DB embedder (1024-d) or retrieval silently breaks** |
|
||||
|
||||
> Benchmark reproduction additionally needs `hive-mind-test/.env.locomo-trio` (ANTHROPIC + OPENAI + MINIMAX + DASHSCOPE + Ollama). The file is marked "treat as compromised and rotate post-run."
|
||||
|
||||
---
|
||||
|
||||
## 7. Benchmark Reproducibility Assessment
|
||||
|
||||
> **Prerequisite for EVERY hive-mind benchmark below:** Ollama serving `nomic-embed-text` (1024-d) on `localhost:11434` (see §4.3) **+** the relevant API keys (§6). The embedder must match what each DB was built with or retrieval silently breaks. This is not documented in any benchmark README today — it is the single most common silent failure for a new dev.
|
||||
|
||||
| Benchmark | Headline | Status | Doc / harness path |
|
||||
|---|---|---|---|
|
||||
| LoCoMo SOTA | 87.66% (Memori protocol, N=1540, GPT-4.1-mini) | **COMPLETE** | `hive-mind-test/scripts/locomo/` + `RESULT-backlog-closeout-2026-06-15.md` |
|
||||
| LoCoMo v5 OSS | 67.8% trio-strict (N=320) | **COMPLETE** | `hive-mind/benchmarks/locomo/` + `METHODOLOGY.md` |
|
||||
| LongMemEval | 75.2% trio-strict (N=100 probe) | **PARTIAL** | `hive-mind-test/scripts/longmemeval/` + `docs/plans/PILLAR2-MEMORY-LONGMEMEVAL-PLAN-2026-05-22.md` |
|
||||
| GAIA2 | preregistered, no live runs | **MISSING (blocked)** | `benchmarks/preregistration/manifest-v8-gaia2-preregistration.md`, `manifest-v8.2-final.md` |
|
||||
| recall-stress | precision@3 gate | **PARTIAL** | `hive-mind/benchmarks/recall-stress/` + `README.md` |
|
||||
|
||||
**LoCoMo SOTA (COMPLETE).** Scripts 40/41/42 + harness fixes (PACK packing, `--measure`) + gotchas (judge-trio resume trap, Mem0 qdrant lock) documented; answers/judgments committed. Repro: `cd hive-mind-test/scripts/locomo && node --env-file=../../.env.locomo-trio 40-cell-retrieval-gpt41mini.mjs` (env `PROMPT_MODE=ours PROFILES=1 DATEWIN=1 EPISODIC=1 RAWDETAIL=1`) `&& node 41-judge-memori-gpt41mini.mjs && node 42-report-memori.mjs`. Gaps: NONE except API keys + Ollama. Effort: 1–2h setup + ~2h compute.
|
||||
|
||||
**LoCoMo v5 OSS (COMPLETE).** Offline number is free via `node benchmarks/locomo/rescore.mjs` (zero cost, SHA-256 artifact integrity check). Full live pipeline (00–37) needs workspace build + API keys. Gaps: NONE for offline rescore.
|
||||
|
||||
**LongMemEval (PARTIAL).** Method proven (ablation 41.6%→52.5%→68.3%→75.2%); `run-longmemeval.mjs`, `run-blend.mjs`, `judge-trio.mjs` exist; dataset SHA-pinned. Gaps: (1) fetch script not implemented (only "adapt 00-fetch" in plan); (2) per-question workspace isolation not coded; (3) full N=500 never executed (only N=100 probe); (4) Qwen-local lane designed, not built; (5) abstention judge in plan, not verified in code. Remediation: ~2–3 days engineering for N=500 scale-out. **Start here:** the dataset is SHA-pinned and `run-longmemeval.mjs` / `run-blend.mjs` / `judge-trio.mjs` already exist in `hive-mind-test/scripts/longmemeval/` — fork `hive-mind-test/scripts/locomo/00-fetch-dataset.mjs` for the fetch step and add per-question workspace isolation to the existing runners; do **not** rewrite from scratch.
|
||||
|
||||
**GAIA2 (MISSING / blocked).** Preregistration thorough (4-cell ablation, ARE verifier + Llama-3.3-70B soft judge, health-check predicates; v8.2 makes LongMemEval V1 Track A0). Gaps: (1) **hard blocker** `module 'signal' has no attribute 'SIGALRM'` in smoke run 2026-04-30; (2) zero live runs, only smoke scaffolds; (3) no adapter code; (4) judge κ calibration unrun; (5) cost/timeline unvalidated. Phase 3 (separate, in `waggle-os-gaia2-wt`) DID close with +19.25pp Fisher p=8.07e-18, but cost $4.09/invocation → HALT. Remediation: fix SIGALRM (ARE platform) + build adapter + Docker — open-ended.
|
||||
|
||||
**recall-stress (PARTIAL).** Harness complete: `node benchmarks/recall-stress/run.mjs --queries <path> --profile … --min-precision … --max-seconds …`. Gaps: **`queries.local.json` is gitignored by design** (permanent and intentional — it shields a proprietary query corpus from the public mirror; this is *not* an accidental omission, so don't expect it to ever ship in git); only `queries.example.json` template ships. No public query set means historical recall-stress numbers are **not** reproducible from git alone. Regression thresholds are examples, not preregistered. Remediation: ~1 day to author domain-specific local queries. **Cross-cutting gap:** Ollama (`nomic-embed-text` 1024-d at `localhost:11434`) is assumed by all hive-mind benchmarks but has NO setup docs in any README.
|
||||
|
||||
---
|
||||
|
||||
## 8. Recommended Transfer Checklist
|
||||
|
||||
1. **Clone the public repos** (clean): `hive-mind`, `hive-mind-clients-archive`, `hive-mind-test`. Run `npm install && npm run build` in each.
|
||||
2. **Resolve the main repo's dirty state FIRST.** Have the concurrent-session owner review + commit the 52 modified + 22 untracked files at `18aebe1f`, or stash, before anyone else touches `waggle-os`. Then clone `waggle-os` (private — provision GitHub access).
|
||||
3. **Rescue unpushed worktree work:** push `feature/harness-sota-bench` (NO_UPSTREAM) and the 3 unpushed commits on `feature/gaia2-are-setup`; decide fate of `waggle-os-ux-prototype` (add to git or archive).
|
||||
4. **Provision credentials** (§6): copy `.env.example` → `.env`, obtain ~15 LLM keys + Clerk + Stripe + local DB/MinIO creds. Set keys in the Waggle vault via UI (Settings → API Keys). For benchmarks, create `hive-mind-test/.env.locomo-trio`.
|
||||
5. **Stand up infra:** `docker-compose up -d` (PG 5434 / Redis 6381 / MinIO 9000); install Ollama + `ollama pull nomic-embed-text`; build native deps (MSVC + Python 3.x on Windows).
|
||||
6. **Physically copy irreplaceable data** (§5): the `.claude/.../memory/` directory (institutional context), benchmark answers/judgments in `hive-mind-test/scripts/locomo/data/`, `waggle-os/benchmarks/results/`, GAIA2 runs, and `~/.waggle/hive-mind.db` if continuing the same memory corpus. **Scrub** the personal `claude-export-*-marko.zip` fixtures and do NOT copy `~/.mem0/`.
|
||||
7. **Verify code health:** `npm run build:all`, `npx tsc --noEmit` on `packages/agent`, `packages/server`, `app`; `npm run test -- --run`; `npm run lint`. (Note: `npm run build` typechecks only `apps/web`; the sidecar runs via `tsx` transpile-only — typecheck `packages/server` separately.)
|
||||
8. **Validate one cheap benchmark:** `cd hive-mind/benchmarks/locomo && node rescore.mjs` (offline, free) to confirm the 67.8% reproduces.
|
||||
9. **Document the gaps** the new dev inherits: LongMemEval N=500 engineering, GAIA2 SIGALRM blocker, recall-stress local queries, missing `Desktop\MEMORIES` exports.
|
||||
10. **Regenerate orientation aids** if helpful: `/understand --full` rebuilds `.understand-anything/` KG + dashboard (costly ~17.9M tokens).
|
||||
|
||||
---
|
||||
|
||||
## 9. Risks & Open Gaps
|
||||
|
||||
**Security / privacy**
|
||||
- Live API keys exist in `waggle-os/.env`, `hive-mind-test/.env.locomo-trio` (file self-flags keys as compromised), and `.env.locomo-trio` comments. **Rotate all on transfer.** Confirm both files remain gitignored.
|
||||
- Personal founder data: `claude-export-2026-04-22-marko.zip` (32 MB) lives in BOTH `hive-mind/test-fixtures/` and `claude-hive-mind/test-fixtures/` — must not reach any public mirror. `~/.mem0/` is personal — never transfer.
|
||||
- `~/.waggle/.vault-key` is machine-local and never backed up; losing it makes the vault unrecoverable. New dev re-enters keys.
|
||||
- recall-stress `queries.local.json` is gitignored specifically to keep a proprietary corpus out of the repo — keep it that way.
|
||||
|
||||
**Reproducibility / completeness gaps**
|
||||
- **`Desktop\MEMORIES` is MISSING.** CLAUDE.md §10 (M2/M3) claims a 30 MB Claude export + 437 MB Gemini export are "ready for E-11 ingestion," but the directory does not exist on disk. Do not assume availability — re-export from source accounts if needed.
|
||||
- **GAIA2 task dataset origin untraced** (`tasks-mini-2.jsonl`, 4.8 MB) — no public URL/fetch script found; if lost, cannot re-acquire.
|
||||
- **BEAM upstream URL** only inferred from an error string (`github.com/mohammadtavakoli78/BEAM.git`); confirm canonical source.
|
||||
- **Ollama embedder setup** (nomic-embed-text, 1024-d, :11434) is a hard prerequisite for every hive-mind benchmark but is undocumented in any README — embedder mismatch silently corrupts retrieval.
|
||||
- **Sidecar type errors ship undetected** — `npm run build` typechecks only `apps/web`; the Fastify sidecar runs transpile-only via `tsx`. Always run `npx tsc --noEmit --project packages/server/tsconfig.json`.
|
||||
- **Concurrent-session WIP** in the main repo is the single biggest operational hazard: 74 changed files at `18aebe1f` with no commits. Any clean/reset destroys them.
|
||||
|
||||
**Documentation gaps surfaced by scouts**
|
||||
- No `00-fetch-dataset` automation for LongMemEval; build scripts must be run manually post-clone.
|
||||
- Hive Mind canonical upstream URLs partly inferred — verify each repo's `.git/config` before relying on them.
|
||||
- `waggle-os/.lovable/` and `.agents/` contents unverified (build cache vs. artifacts).
|
||||
- OSS sync scripts (`oss-subtree-split.sh` guard-only, `oss-drift-check.sh`) — verify functional before any OSS release; per CLAUDE.md §7.5 a raw split would leak proprietary IP.
|
||||
Reference in New Issue
Block a user