The bench compared mem0 with MindDB, FrameStore and HybridSearch from the hive-mind substrate, which @waggle/core no longer exports and which left the repository. It now runs mem0, mem0-rerank and cosine; the recorded comparison stays in docs/oss/research/memory-search-quality.md.
20 KiB
Memory search quality: the old engine compared with mem0 (P1.5)
Measured on 2026-10-24 to decide whether mem0 search is good enough to replace the old
engine (MindDB + FrameStore + HybridSearch from the hive-mind substrate) before the old
engine is deleted. ADR-0007 asked for this comparison on a reference set.
The reference set and the benchmark live in packages/memory-mem0/bench/ and keep working
after the old engine is gone (see "How to re-run").
Summary
- On the 61 answerable queries, the mem0 store scores the same as the old engine as Waggle uses it for chat recall (hybrid search followed by a cross-encoder reranker): recall@1 0.72 against 0.72, recall@5 0.89 against 0.90, MRR 0.82 against 0.83. mem0 answers in about 21 ms, the old engine with its reranker in about 145 ms.
- The old engine without the reranker, which is what the agent's
search_memorytool runs, is clearly worse: recall@1 0.49, recall@5 0.81, MRR 0.65. Its importance boost and its keyword lane push general "important" memories above the right answer. - mem0 is acceptable for v1. Its weak spots are questions phrased without the memory's words (recall@1 0.52, where plain cosine similarity reaches 0.71), questions that need several memories (recall@5 0.69), and a search cost that grows with the whole store, not only with the scopes searched.
Setup
| Code | branch oss/community-edition at f3eb6b6, plus the bench added with this document |
| mem0 | mem0ai 3.3.1 through @waggle/memory-mem0 (createMem0MemoryStore) |
| Embedder | Waggle's in-process embedder, Xenova/all-MiniLM-L6-v2 (fp32, 384 dimensions zero-padded to 1024), @huggingface/transformers 3.8.1, model cache ~/.waggle/models. These are the sidecar's settings (WaggleConfig.getEmbeddingConfig, targetDimensions: 1024) |
| Reranker | Xenova/ms-marco-MiniLM-L-6-v2 through createInProcessReranker, cache ~/.waggle/models/reranker, as the sidecar configures it |
| Machine | Apple M1, 8 cores, 16 GB, macOS 26.6, Node 22.23.2 |
| Network | every fetch to another host was refused by the bench; none was attempted. mem0 telemetry was off and ~/.mem0 was not created |
Ollama was not running on the machine (localhost:11434 refused the connection), so only the
in-process embedder was measured. The variant with the user's own embedding key was not run;
see "How to re-run".
Reference set
reference-set.json describes a fictional freelance designer, Mila Sorensen, with 44
personal memories and two client workspaces: atlas (30 memories, a route-planning app for a
freight company) and harbor (26 memories, a dashboard for a food bank). Every memory has a
kind and an importance: 3 critical, 9 important, 84 normal, 4 temporary. The set contains
near-duplicates (two dark-mode preferences, two statements of each project deadline) and
distractors (a friend's cat allergy next to Mila's peanut allergy; a dark-mode decision in
each workspace next to Mila's personal preference; a different database in each workspace).
All names and facts are invented.
The 67 queries:
| Type | n | What it tests |
|---|---|---|
| paraphrase | 21 | No content word in common with the memory to find. Checked with SQLite's porter stemmer, the one the old engine's full-text index uses |
| keyword | 17 | An exact name, code or identifier: ATL-142, #F28C28, fernwork/harbor-dashboard, Dr. Okafor |
| multi-fact | 12 | Two to four memories together answer the question |
| scope | 11 | The same question asked in different workspaces has different answers; personal memories compete |
| no-answer | 6 | Nothing in memory answers the question |
Personal questions are mostly in the first person ("What kind of pet do I have?"), as a user
asks in chat; chat recall searches with the user's message. A few use the name, as the agent
does when it writes its own search_memory query.
Engines
All engines share one embedder instance, so they see the same vectors.
| Engine | What runs |
|---|---|
old |
One .mind file for personal memory and one per workspace, as in the sidecar. Memories are written with FrameStore.createIFrame and indexed with HybridSearch.indexFrame. Each searched mind runs HybridSearch.search(query, { limit: 10, profile: 'balanced' }); results of all searched minds are merged by finalScore. This is the search behind the search_memory tool |
old-rerank |
The same with the in-process cross-encoder reranker, as Orchestrator.recallMemory uses it for chat recall by default |
mem0 |
createMem0MemoryStore, remember({ scope, text, kind, importance }), search(text, { scopes, limit: 10 }), which is what the adapter offers callers |
mem0-rerank |
mem0's top 30 re-ordered by the same reranker. Not part of the adapter; measured as a possible improvement |
cosine |
Baseline: every memory of the searched scopes ranked by cosine similarity alone, without any keyword signal |
Metrics
- recall@k: the share of a query's relevant memories found in the first k results, out of
the most that fit (
min(k, relevant)). recall@1 is therefore "the first result is relevant", and recall@5 for a three-memory question is "all three are in the top five". - MRR: mean of 1 / rank of the first relevant result within the top 10, 0 when none is there.
- Both are averaged over the 61 answerable queries. One query moves recall@1 overall by 0.016; differences below about 0.05 are a few queries and should be read as ties.
- Abstain separation: the probability that the top score of a query whose first result is correct is higher than the top score of a no-answer query. 1.0 would mean a score threshold can tell "memory has an answer" from "memory has none".
- Latency: wall time of one
searchcall, query embedding included, median and 95th percentile over the 67 queries after one warm-up query.
Results
The relevance numbers were identical in all four runs; only latency varied.
| Engine | recall@1 | recall@5 | MRR | median ms | p95 ms | abstain separation |
|---|---|---|---|---|---|---|
| old | 0.49 | 0.81 | 0.65 | 5.9 | 7.3 | 0.35 |
| old-rerank | 0.72 | 0.90 | 0.83 | 143.5 | 261.8 | 0.65 |
| mem0 | 0.72 | 0.89 | 0.82 | 21.0 | 39.5 | 0.79 |
| mem0-rerank | 0.72 | 0.90 | 0.83 | 148.5 | 256.7 | 0.65 |
| cosine | 0.74 | 0.88 | 0.82 | 2.3 | 5.5 | 0.47 |
The latency columns come from the first of the four runs. Other agents were working on the
same machine during the later runs (load average 30 to 60 on 8 cores), and the medians rose
to 7–20 ms for old, 36–47 ms for mem0 and 170–560 ms for the reranker variants. In
every run mem0 took 2 to 5 times as long as old, and the reranker variants were the
slowest.
By query type (recall@1 / recall@5 / MRR):
| Type | n | old | old-rerank | mem0 | mem0-rerank | cosine |
|---|---|---|---|---|---|---|
| paraphrase | 21 | 0.19 / 0.76 / 0.40 | 0.57 / 0.90 / 0.72 | 0.52 / 0.90 / 0.70 | 0.57 / 0.90 / 0.72 | 0.71 / 0.90 / 0.80 |
| keyword | 17 | 0.94 / 1.00 / 0.96 | 0.88 / 1.00 / 0.94 | 1.00 / 1.00 / 1.00 | 0.88 / 1.00 / 0.94 | 0.88 / 1.00 / 0.94 |
| multi-fact | 12 | 0.42 / 0.61 / 0.61 | 0.75 / 0.80 / 0.82 | 0.75 / 0.69 / 0.79 | 0.75 / 0.80 / 0.82 | 0.67 / 0.67 / 0.73 |
| scope | 11 | 0.45 / 0.82 / 0.66 | 0.73 / 0.86 / 0.86 | 0.64 / 0.91 / 0.80 | 0.73 / 0.86 / 0.86 | 0.64 / 0.86 / 0.78 |
By the number of scopes searched:
| Scopes | n | old | old-rerank | mem0 | mem0-rerank | cosine |
|---|---|---|---|---|---|---|
| one | 25 | 0.60 / 0.89 / 0.75 | 0.80 / 0.95 / 0.87 | 0.80 / 0.99 / 0.87 | 0.80 / 0.95 / 0.87 | 0.88 / 0.99 / 0.93 |
| two or more | 36 | 0.42 / 0.75 / 0.58 | 0.67 / 0.87 / 0.79 | 0.67 / 0.82 / 0.78 | 0.67 / 0.87 / 0.79 | 0.64 / 0.80 / 0.75 |
The bench prints a per-query table of misses (rank of the first relevant memory, recall@5 and the top result per engine) and the top result of every no-answer query. The examples below come from it.
What mem0 3.3.1 does when it searches
Read in dist/oss/index.js of the installed package (line numbers refer to that file) and
checked with mem0's explain: true score breakdown on this set.
- Vector similarity. The query is embedded once. The
memoryvector store reads every row of itsvectorstable, parses each JSON payload, keeps the rows whoseuser_idandagent_idmatch, and computes cosine similarity in JavaScript (lines 1143–1174). The table holds all scopes, so the whole store is read even when one small workspace is searched. mem0 keeps the bestmax(topK × 4, 60)rows as candidates (line 18031); the adapter asks fortopK = max(limit × 5, 100), so up to 400 per scope. - BM25. A second full read of the table computes BM25 (k1 1.5, b 0.75) over
textLemmatized, which mem0 stores with each memory: lower-case ASCII words without stop words, stemmed with the Porter stemmer of thenaturalpackage if it is installed (lines 16171–16237).naturalis not installed in Waggle, so mem0 falls back to a simple suffix stripper. Non-ASCII letters split words (Škodabecomeskoda) on both sides, so names still match. The raw score is mapped to 0–1 with a sigmoid whose midpoint depends on the query length (lines 16855–16871). Only memories already among the vector candidates receive it. - Entity boost. Entities found in the query (quoted text, proper nouns, identifiers,
noun phrases through
compromise) are looked up in a second store,vectors_entities.db, and add up to 0.5 to the memories linked to them (lines 18060–18127). Links are only written when mem0 extracts facts with a model (infer: true). The adapter stores text withinfer: false, which callscreateMemoryand links nothing (lines 17575–17592, 18423), so the boost was 0 for all 67 queries. Memories that P1.5 extracts from conversations will get links. - Combination. Each candidate scores
(semantic + bm25 + entity) / maxPossible, wheremaxPossibleis 1, plus 1 if any candidate of this call has a BM25 score, plus 0.5 if any has an entity boost (lines 16873–16919). The threshold applies to the semantic part only; the adapter passes 0.
There is no importance, recency or access-count signal. ADR-0007 is right that mem0 search is "hybrid vector and BM25", with three qualifications: BM25 only re-ranks the vector candidates, the entity part is inactive for memories written as text, and semantic and keyword scores have equal weight. BM25 contributed to the ranking in the first searched scope of 40 of the 67 queries.
The adapter searches each scope with its own call and merges by score. Because
maxPossible is computed per call, scores from two scopes can be divided by different
numbers: this happened in 12 of the 39 queries that search several scopes. Re-ranking those
with a common divisor changed the first result of one query (q40), from a relevant memory to
an irrelevant one, so the effect on this set is nil.
Where each engine wins and loses
Old engine without the reranker
HybridSearch fuses a full-text lane and a vector lane with reciprocal rank fusion
(1 / (60 + rank) per lane) and multiplies the result by a relevance factor. With the
'balanced' profile and new memories, that factor is 0.80 for normal, 0.90 for important and
1.00 for critical memories. The gap between rank 1 and rank 2 in a lane is under 2 %, so a
12–25 % importance bonus outweighs most differences in rank. 21 of the 31 wrong first
results were critical or important memories, which make up 12 of the 100 memories. For
example, "What kind of pet do I have?" returns the peanut allergy (critical) first and the
greyhound fourth; "What do I charge clients?" returns the decision to take no new clients
(important) first.
The full-text lane joins all words of the query with OR. Its stop-word list has no pronouns or names, so any shared common word gives a memory a full vote. In a first draft of this set, personal questions were in the third person ("What kind of pet does she have?"); the old engine then ranked a relevant memory first for 1 of the 21 paraphrase queries, because "she" appears in many memories. With first-person questions it is 4 of 21.
Exact names are its strength: recall@1 0.94 on keyword queries, through the full-text index.
Old engine with the reranker (chat recall)
The cross-encoder replaces the fused score with its own, which removes the importance bias:
recall@1 rises to 0.72. It costs 130–140 ms per search on 100 memories and makes its own
mistakes. "Would I enjoy a beef burger?" ranks a science-fiction memory first and loses the
vegetarian-diet memory from the top 10; "Why was a lightweight graphing package picked?"
ranks the standing-desk decision first and loses the Chart.js decision; two keyword queries
(When can I phone Grace Mbeki?, fernwork/harbor-dashboard) drop from rank 1 to rank 2.
The model is trained on web passages, not on short personal facts.
mem0
mem0 ties the old engine with the reranker overall.
It is best on keyword queries (recall@1 1.00): BM25 finds ATL-142, #F28C28, PostGIS
and fernwork.si even where the vector alone puts another memory first. It also separates
answerable from unanswerable questions best (0.79, against 0.47 for plain cosine over the
same vectors, so the difference comes from BM25), though not well enough for a threshold
(see "Shared failures").
It loses on paraphrases: recall@1 0.52, where the cosine baseline over the same vectors
reaches 0.71. BM25 weighs as much as the vector score, so one shared ordinary word lifts a
distractor above a better semantic match. "How do I like replies to be formatted?" puts
"She likes to be addressed as Mila" first through "like"; "What computer do I work on?"
puts "wears glasses for screen work" first through "work". Lowering the BM25 weight does not
help overall. Re-scoring mem0's candidates as semantic + w × bm25, from the explain: true
breakdown (a one-off check, not part of the bench), gave:
| w | recall@1 | paraphrase recall@1 | keyword recall@1 |
|---|---|---|---|
| 0 (cosine) | 0.74 | 0.71 | 0.88 |
| 0.25 | 0.72 | 0.67 | 0.88 |
| 0.5 | 0.72 | 0.57 | 0.94 |
| 1 (mem0's weights) | 0.70 | 0.52 | 1.00 |
The weight trades paraphrase accuracy for keyword accuracy without changing the total. The last row differs from the bench's 0.72 by one query (q40), because the re-scoring used one divisor for all scopes.
Questions that need several memories are its other weak spot (recall@5 0.69 against 0.80 with the reranker). "What has been decided about the Harbor tech stack?" finds none of the four relevant memories in the top five, because every Harbor memory is about Harbor; "How big are the budgets of Mila's current projects?" puts the personal decision about "current projects" first through BM25.
In scope queries, a personal memory sometimes wins over the workspace answer: "When is the deadline?" in Harbor returns the personal first-quarter plan first and the board-meeting deadline third; "Who is the client contact?" in Harbor returns "emergency contact" first through BM25. Isolation between workspaces holds in every engine: no query returned a memory from a workspace it did not search.
Shared failures
- "Is there anything I must never eat?" ranks the vegetarian diet above the peanut allergy in every engine (allergy at rank 2 or 3).
- "Does the app have dark mode?" in either workspace ranks Mila's personal dark-mode preference above the workspace's decision in every engine.
- "What did we adjust for people wearing mittens?" ranks "wears glasses for screen work" first everywhere; the glove decision is second or third.
- Every engine returns something for questions memory cannot answer. With mem0, 19 of the 44 correct first results score at or below the highest no-answer score (0.39 for "How many employees does Kestrel Freight have?"), so a fixed threshold cannot decide that memory has no answer.
Latency and store size
At 100 memories mem0 needs about 21 ms per search, the old engine 6 ms and the reranker
variants about 145 ms. mem0's cost grows with the size of the whole store, because both
passes read every row of the vectors table. A separate probe on the same machine (a
deterministic stand-in embedder, so that only the store's work was timed, 21 queries each):
| Store | Median search |
|---|---|
| 1,000 memories in one scope, 1024 dimensions | 66 ms |
| 5,000 memories in one scope, 1024 dimensions | 336 ms |
| 5,000 memories in one scope, 384 dimensions | 136 ms |
| 5,000 memories in personal, search of a 20-memory workspace | 224 ms |
The machine was busy with other work during the probe, so the absolute numbers are rough; the ratios come from runs made one after another. A search across personal memory and one workspace makes two such passes.
Conclusion
mem0 search is acceptable for v1. On this set it matches the old engine's chat-recall path,
it is clearly better than the old engine's search_memory path, it needs no reranker model,
and it runs offline with Waggle's embedder.
What would improve it, in order of value for effort:
- Store vectors at the embedder's native size. The sidecar pads MiniLM's 384 dimensions to 1024. Zero padding does not change cosine similarity, so the ranking stays the same, while mem0's search pass gets about 2.5 times faster. The adapter has no check like the old engine's embedding fingerprint: a store created with one embedder and read with another of the same size gives degraded results without an error. Whatever size is chosen, the store should record the embedder and refuse a mismatch.
- Do not tune the BM25 weight. The table above shows a trade, not a gain.
- Leave the reranker out of v1. It brings mem0 to the same numbers as the old
engine's chat recall (recall@5 0.90, MRR 0.83), costs about 130 ms per search, and lives
in the substrate that is being deleted, so it would have to be copied into
@waggle/corefirst. - Revisit the vector store when stores grow. At a few hundred to a few thousand
memories the full scan is fine. Past that, a store with an index, or a single mem0 call
across all searched scopes (one pass instead of one per scope, not tested here), would
be needed. ADR-0007 already notes the same for
list,countandforgetMatching. - Do not use the score to decide that memory has no answer. Callers should rank, not threshold.
What this bench does not cover:
- Time. Neither store can backdate a memory in the bench (mem0 rejects timestamps), so the old engine's recency decay and questions about dates are not measured.
- The rest of chat recall. The importance lane, prefix lanes and date window in
Orchestrator.recallMemorysit outsideHybridSearch; P1.5 maps them to contract filters. - Extracted memories. All memories are written as text; memories extracted by a model would also get mem0's entity boost.
- Other embedders and languages. One English model was measured.
- The merge across minds. Production renders personal and workspace results in separate sections; the bench merges them by score for every engine to get one ranking.
- Size. 100 memories and 67 queries written by one author.
How to re-run
From the repository root, after npm ci and npm run build:packages. The old engine left
the repository in P2.2, so the bench runs mem0, mem0-rerank and cosine:
# All engines (mem0, mem0-rerank, cosine)
npx tsx packages/memory-mem0/bench/search-quality.ts
# Without the reranker model
npx tsx packages/memory-mem0/bench/search-quality.ts --engines mem0,cosine
# Per-query results as JSON
npx tsx packages/memory-mem0/bench/search-quality.ts --out /tmp/search-quality.json
The in-process model must be in ~/.waggle/models (the sidecar downloads it on first start)
or in the directory named by WAGGLE_MODELS_DIR; the bench refuses network access and does
not download it.
Other embedders:
# A running local Ollama; the bench neither starts it nor pulls models
OLLAMA_EMBED_MODEL=nomic-embed-text npx tsx packages/memory-mem0/bench/search-quality.ts --embedder ollama
# The user's own key. OPENAI_API_KEY is ignored on purpose, so a key that happens to be
# set in the environment is never used.
BENCH_EMBEDDING_API_KEY=<key> BENCH_EMBEDDING_MODEL=text-embedding-3-small \
npx tsx packages/memory-mem0/bench/search-quality.ts --embedder openai
BENCH_EMBEDDING_BASE_URL points the key variant at another OpenAI-compatible endpoint.
With a key, only that endpoint may be reached.
To extend the set, add memories and queries to reference-set.json; a query lists every
memory that answers it in relevant, and an empty list marks a question without an answer.