waggle-os/packages/hive-mind-core/README.md

@waggle/hive-mind-core

The substrate. SQLite + sqlite-vec hybrid search, bitemporal knowledge graph, frame compression, identity layer, awareness layer, embedder providers, harvest pipeline.

What this is

hive-mind-core is the persistence and retrieval substrate that powers Waggle OS's memory layer. The public Apache-2.0 distribution lives in marolinik/hive-mind and is produced from this canonical source by a maintainer-curated forward-port.

Publication boundary: this monorepo package is private and must never be published or pushed as a raw subtree split. It contains Waggle-only files and interleaved install_audit schema/migration logic. The curated forward-port adapts the public layout and imports, then removes all excluded material.

What's inside

Module	Purpose
mind/db.ts	MindDB — better-sqlite3 + sqlite-vec hybrid backend
mind/frames.ts	FrameStore — I/P/B frame types + compaction + dedup
mind/sessions.ts	SessionStore — session lifecycle + ensureActive
mind/search.ts	HybridSearch — FTS5 + vec0 fused via Reciprocal Rank Fusion
mind/knowledge.ts	KnowledgeGraph — entity/relation graph + bitemporal validity
mind/identity.ts	IdentityLayer — personal identity persistence
mind/awareness.ts	AwarenessLayer — active task/state tracking
mind/scoring.ts	Scoring profiles (recency, popularity, relevance, importance)
mind/reconcile.ts	Index reconciliation (FTS, vec, orphan cleanup)
mind/ontology.ts	Entity ontology + validation
mind/concept-tracker.ts	Concept mastery tracking
mind/entity-normalizer.ts	Entity name normalization + dedup
mind/{api,inprocess,litellm,ollama}-embedder.ts	Embedder providers + provider factory
mind/embedding-provider.ts	createEmbeddingProvider — runtime embedder selection + quota
harvest/pipeline.ts	HarvestPipeline — universal ingestion into frames
harvest/dedup.ts	Cross-source dedup
harvest/{chatgpt,claude,claude-code,gemini,perplexity,markdown,plaintext,pdf,url,universal}-adapter.ts	Per-source ingest adapters
harvest/source-store.ts, run-store.ts	Harvest source + run persistence
injection-scanner.ts	scanForInjection — prompt-injection detection
logger.ts	createCoreLogger — minimal structured logger

Publication status

• The public mirror is Apache-2.0 and excludes Waggle-only governance and audit material.
• Benchmark and performance claims belong in versioned reports with their model, dataset, scorer, revision, and receipts; this README does not make an unqualified SOTA claim.
• Run scripts/oss-drift-check.sh and the public mirror's own test and lint gates before every curated release.

Status

Migrated from marolinik/hive-mind into the canonical marolinik/waggle-os monorepo at packages/hive-mind-core/ on 2026-04-30. Future development happens here. scripts/oss-subtree-split.sh is an inspection/curation starting point only; the public mirror is updated through a reviewed, curated forward-port.

License: Apache-2.0.