8.1 KiB
OSS Drift Triage — waggle-os ↔ marolinik/hive-mind (2026-06-11)
Trigger: the first scripts/oss-drift-check.sh run (§7.5 policy ratification arc)
surfaced drift far beyond the W4.2 reranker incident: 3 OSS-only source files +
42 differing files. This doc is the file-by-file direction triage (4-agent recon,
monorepo HEAD vs OSS origin/master) and the reverse-port execution record.
Key discovery: the OSS repo was never a byte-identical subtree-split product —
it was a hand-extraction (May 2026) with deliberate transforms (extraction
headers, [hive-mind:] branding, import rewrites, scrubbed prompt examples), and
both sides evolved independently since. Blob-equality checks are useless;
semantic per-file triage was required. The §7.5 "monorepo sole source" policy
now governs; this triage is the one-time consolidation.
Verdict matrix (45 files)
PACKAGING-ONLY — no action (19)
perplexity/markdown/pdf/plaintext/url adapters, chunk-utils, prompts (scrubbed examples), index (harvest barrel), run-store, source-store, sessions, awareness, identity, concept-tracker, reconcile, api/inprocess/litellm/ollama embedders, multi-mind-cache.
MONO-AHEAD — mirror is stale, fixed by next re-split (6)
chatgpt-adapter, gemini-adapter, universal-adapter (W4.4 captions + raw-types), dedup (harvestSetHash), types (HARVEST_FRAME_CONTENT_CAP), scoring (W4.2 created_at decay — OSS still has the last_accessed decay bug). Plus all 11 ONLY-IN-MONO modules (temporal stack, raw-turns/raw-detail-lane, extract-memory-lanes, multi-mind, proprietary-excluded evolution/traces/signals).
REVERSE-PORT — OSS has real functionality the monorepo lacks
| # | Source | What | Size | Status |
|---|---|---|---|---|
| R1 | logger.ts | stderr routing for info/debug — mono logger writes to stdout; hive-mind-mcp-server is stdio-transport, so pipeline log.info calls can corrupt the MCP stream (LIVE latent bug). + CoreLogger type export. |
S | PORTED |
| R2 | knowledge.ts | findEntityByName() exact-match dedup fix (LIKE top-K drops exact match → runaway dups, e.g. 3506 "Phase" rows) + dedupeByName() transactional merge + safeParseProps(). |
M | PORTED |
| R3 | entity-normalizer.ts | Write-time noise filter: isNoiseName()/isLikelyAcronym() + STOP_TOKENS + TECH_ALLOWLIST — blocks low-signal names entering the KG. |
S | PORTED |
| R4 | workspace-manager.ts | ensure(id, options) idempotent create-by-trusted-id — mono MCP save_memory to a not-yet-created workspace has no auto-create path. |
S | PORTED |
| R5 | embedding-provider.ts | maxEmbedCharsForModel()/capEmbedText() (input capping) + reembedPerText() (per-text batch-failure recovery — mono degrades whole batch to mock noise). |
M | PORTED |
| R6 | claude-adapter.ts | 2026-04-22 Claude export streams: memories (conversations_memory + project_memories) + design_chats[] + enriched project-doc parsing — mono silently drops two whole export streams. |
M | PORTED |
| R7 | db.ts | Embedding-fingerprint guard (ensureEmbeddingFingerprint + EmbeddingDimMismatchError + recreateVecTables) — refuses loudly on dim mismatch; mono has zero protection against mixed-dim vector corruption. |
M | PORTED |
D-items — founder GO 2026-06-11 ("do all 3"), ALL PORTED same session
| # | What | Status |
|---|---|---|
| D1 | Chunk-level retrieval stack — chunker.ts + memory_frame_chunks(+_vec, dim-parameterized) schema + indexChunksForFrame + chunk-vec lane in HybridSearch (over-fetch ×5, best-chunk-per-frame dedup, clean fallback to whole-frame vectors) + rechunkAllFrames backfill. |
PORTED + DEFAULT-ON (2026-06-12) after the long-frame needle probe (kill switch WAGGLE_CHUNK_RETRIEVAL=0). See probe record below. |
| D2 | LLM KG entity extraction (replaces the capitalized-n-gram regex as the KG quality path) | PORTED — prompt/parser/batching from OSS llm-extractor; executors rehomed onto LLMCallFn 'fast'. Runs in the daily memory-lane cron AFTER the lane pass, same frame window + shared watermark. Writes dedup via findEntityByName (R2) + filter via isNoiseName (R3); injection-scanned. |
| D3 | content_hash indexed dedup column | PORTED — with MONO semantics (sha256 over stripHmPrefix(content).trim(), content-hash.ts): the OSS trim-only hash would have regressed OQ-6 provenance-insensitive dedup. findDuplicate now O(1) indexed, NO recency window (old LIMIT-500 scan silently missed older dups). Idempotent migration + backfill. |
Forward-port queue (mono → OSS, next re-split)
W4 arc (all of it), scoring created_at fix, since/until fencepost fixes,
likeFallbackSearch, escapeLikeTerm (knowledge), W4.4 captions + raw-types,
HARVEST_FRAME_CONTENT_CAP, harvestSetHash. Note: mono adapters depend on
harvest/raw-types.ts which the OSS tree lacks — re-split must carry it.
Execution record
- 2026-06-11: triage run (4 parallel agents over monorepo HEAD vs oss origin/master).
- 2026-06-11: R1-R7 reverse-ported (see commits on main).
- 2026-06-11 (later): founder GO "do all 3" → D1+D2+D3 ported same session (3 implementation agents + direct work; D1 flag-gated default-OFF).
- 2026-06-12 — D1 eval gate run → DEFAULT-ON. Pre-registered probe
(
benchmarks/chunk-probe/run-probe.mjs, $0, all-local):- LoCoMo REJECTED as the ruler — its frames max ~1,000 chars (below the 2,000-char chunk threshold): every frame yields 1 chunk ≡ whole frame, so a LoCoMo A/B measures noise by construction. Epistemic check before spend.
- Honest corpus: COPY of the real personal mind (471 frames, 129 >2k chars, max 25.7k), both cells re-embedded from scratch (Ollama nomic-embed-text-8k/1024; original vectors were mock-fingerprinted).
- Verbatim deep-position needles (50–90% frame depth), n=52, paired cells:
whole-frame
vectorSearchvs chunkvectorSearchChunks. - Result: hit@5 chunk 46/52 vs whole-frame 17/52; discordant pairs 30-vs-1 (McNemar p≈2e-8). Within-embed-cap stratum: chunk 17/20 vs 13/20 (wins on fair ground); beyond-cap: 29/32 vs 4/32 (whole-frame is structurally blind past the embedder's token window).
- Side-finding (R5 hardening): model names lie —
nomic-embed-text-8kis architecture-capped at 2048 tokens (nomic-bert); the OSS 24k-char heuristic 400s and mock-poisons long frames.maxEmbedCharsForModel8k-branch reduced 24k→8k chars. - Per-needle detail:
benchmarks/chunk-probe/data/probe-result.json(data dir gitignored — contains a personal-mind copy).
- 2026-06-12 — backfill wiring SHIPPED (
packages/server/src/local/ vector-backfill.ts): one-time per-mind vector repair (mock-fingerprint cure: recreate + re-embed) +rechunkAllFrames, wired at boot (personal) and in the daily memory_lane_extract cron (all minds); idempotent meta flag, skip-without-flag while the embedder is mock (daily retry). - 2026-06-12 — forward-port SHIPPED as
marolinik/hive-mind PR #14
(branch
feature/mono-parity-2026-06-12; merge = founder gate). Three port agents: core fixes (scoring created_at decay, search date-window + LIKE-fallback + chunk flagHIVE_MIND_CHUNK_RETRIEVAL+ auto-index + rechunkAllFrames, content-hash hm-stripped semantics + one-time rehash migration, embed-cap 24k→8k), harvest parity (captions ×4 + raw-types + content cap + harvestSetHash + extract-memory-lanes + raw-turns), new modules (temporal stack + raw-detail-lane). OSS gates: tsc 0, full repo 654/654 (+~90 new co-located tests). - Drift-check after the port (against the parity branch): structural
gaps CLOSED. Residual ONLY-IN-OSS =
llm-extractor.ts(deliberately kept — the OSS CLI consumes it; mono's D2 rehomed it as extract-kg-entities). Residual ONLY-IN-MONO = proprietary exclusions (evolution-runs/execution-traces/improvement-signals) + mono-specific modules (multi-mind, extract-kg-entities). The long DIFFERS list is PERMANENT-BY-DESIGN cosmetics (branding, env-var names, scrubbed examples, extraction headers) — the check's actionable signals are the ONLY-IN-* buckets plus manual DIFFERS inspection after substrate arcs.