Files
waggle-os/packages/memory-mem0/bench/reference-set.json
shumeiko.it f046a20b34 test(memory): compare search quality of the old engine and mem0 on a reference set (P1.5)
Add a reference set (100 invented memories across personal memory and two
workspaces, 67 queries: paraphrase, keyword, multi-fact, scope, no-answer)
and a tsx benchmark that loads it into the old engine (MindDB, FrameStore,
HybridSearch, with and without the cross-encoder reranker), the mem0 store
and a plain-cosine baseline, all on Waggle's in-process embedder. It reports
recall@1, recall@5, MRR, latency and misses per engine and query type. The
mem0 half runs on its own (--engines mem0,cosine) once the old engine is
removed. Ollama and the user's own key are optional embedders; the bench
refuses network access to anything else and never reads OPENAI_API_KEY.

Results in docs/oss/research/memory-search-quality.md: mem0 matches the old
engine's chat-recall path (recall@1 0.72, recall@5 0.89 vs 0.90, MRR 0.82
vs 0.83) at a fraction of its latency and is acceptable for v1.

No new dependencies.
2026-10-24 11:56:35 +02:00

28 KiB