# Memory search quality: the old engine compared with mem0 (P1.5) Measured on 2026-10-24 to decide whether mem0 search is good enough to replace the old engine (`MindDB` + `FrameStore` + `HybridSearch` from the hive-mind substrate) before the old engine is deleted. ADR-0007 asked for this comparison on a reference set. The reference set and the benchmark live in `packages/memory-mem0/bench/` and keep working after the old engine is gone (see "How to re-run"). ## Summary - On the 61 answerable queries, the mem0 store scores the same as the old engine as Waggle uses it for chat recall (hybrid search followed by a cross-encoder reranker): recall@1 0.72 against 0.72, recall@5 0.89 against 0.90, MRR 0.82 against 0.83. mem0 answers in about 21 ms, the old engine with its reranker in about 145 ms. - The old engine without the reranker, which is what the agent's `search_memory` tool runs, is clearly worse: recall@1 0.49, recall@5 0.81, MRR 0.65. Its importance boost and its keyword lane push general "important" memories above the right answer. - mem0 is acceptable for v1. Its weak spots are questions phrased without the memory's words (recall@1 0.52, where plain cosine similarity reaches 0.71), questions that need several memories (recall@5 0.69), and a search cost that grows with the whole store, not only with the scopes searched. ## Setup | | | |---|---| | Code | branch `oss/community-edition` at `f3eb6b6`, plus the bench added with this document | | mem0 | `mem0ai` 3.3.1 through `@waggle/memory-mem0` (`createMem0MemoryStore`) | | Embedder | Waggle's in-process embedder, `Xenova/all-MiniLM-L6-v2` (fp32, 384 dimensions zero-padded to 1024), `@huggingface/transformers` 3.8.1, model cache `~/.waggle/models`. These are the sidecar's settings (`WaggleConfig.getEmbeddingConfig`, `targetDimensions: 1024`) | | Reranker | `Xenova/ms-marco-MiniLM-L-6-v2` through `createInProcessReranker`, cache `~/.waggle/models/reranker`, as the sidecar configures it | | Machine | Apple M1, 8 cores, 16 GB, macOS 26.6, Node 22.23.2 | | Network | every `fetch` to another host was refused by the bench; none was attempted. mem0 telemetry was off and `~/.mem0` was not created | Ollama was not running on the machine (`localhost:11434` refused the connection), so only the in-process embedder was measured. The variant with the user's own embedding key was not run; see "How to re-run". ### Reference set `reference-set.json` describes a fictional freelance designer, Mila Sorensen, with 44 personal memories and two client workspaces: `atlas` (30 memories, a route-planning app for a freight company) and `harbor` (26 memories, a dashboard for a food bank). Every memory has a kind and an importance: 3 critical, 9 important, 84 normal, 4 temporary. The set contains near-duplicates (two dark-mode preferences, two statements of each project deadline) and distractors (a friend's cat allergy next to Mila's peanut allergy; a dark-mode decision in each workspace next to Mila's personal preference; a different database in each workspace). All names and facts are invented. The 67 queries: | Type | n | What it tests | |---|---|---| | paraphrase | 21 | No content word in common with the memory to find. Checked with SQLite's porter stemmer, the one the old engine's full-text index uses | | keyword | 17 | An exact name, code or identifier: `ATL-142`, `#F28C28`, `fernwork/harbor-dashboard`, `Dr. Okafor` | | multi-fact | 12 | Two to four memories together answer the question | | scope | 11 | The same question asked in different workspaces has different answers; personal memories compete | | no-answer | 6 | Nothing in memory answers the question | Personal questions are mostly in the first person ("What kind of pet do I have?"), as a user asks in chat; chat recall searches with the user's message. A few use the name, as the agent does when it writes its own `search_memory` query. ### Engines All engines share one embedder instance, so they see the same vectors. | Engine | What runs | |---|---| | `old` | One `.mind` file for personal memory and one per workspace, as in the sidecar. Memories are written with `FrameStore.createIFrame` and indexed with `HybridSearch.indexFrame`. Each searched mind runs `HybridSearch.search(query, { limit: 10, profile: 'balanced' })`; results of all searched minds are merged by `finalScore`. This is the search behind the `search_memory` tool | | `old-rerank` | The same with the in-process cross-encoder reranker, as `Orchestrator.recallMemory` uses it for chat recall by default | | `mem0` | `createMem0MemoryStore`, `remember({ scope, text, kind, importance })`, `search(text, { scopes, limit: 10 })`, which is what the adapter offers callers | | `mem0-rerank` | mem0's top 30 re-ordered by the same reranker. Not part of the adapter; measured as a possible improvement | | `cosine` | Baseline: every memory of the searched scopes ranked by cosine similarity alone, without any keyword signal | ### Metrics - recall@k: the share of a query's relevant memories found in the first k results, out of the most that fit (`min(k, relevant)`). recall@1 is therefore "the first result is relevant", and recall@5 for a three-memory question is "all three are in the top five". - MRR: mean of 1 / rank of the first relevant result within the top 10, 0 when none is there. - Both are averaged over the 61 answerable queries. One query moves recall@1 overall by 0.016; differences below about 0.05 are a few queries and should be read as ties. - Abstain separation: the probability that the top score of a query whose first result is correct is higher than the top score of a no-answer query. 1.0 would mean a score threshold can tell "memory has an answer" from "memory has none". - Latency: wall time of one `search` call, query embedding included, median and 95th percentile over the 67 queries after one warm-up query. ## Results The relevance numbers were identical in all four runs; only latency varied. | Engine | recall@1 | recall@5 | MRR | median ms | p95 ms | abstain separation | |---|---|---|---|---|---|---| | old | 0.49 | 0.81 | 0.65 | 5.9 | 7.3 | 0.35 | | old-rerank | 0.72 | 0.90 | 0.83 | 143.5 | 261.8 | 0.65 | | mem0 | 0.72 | 0.89 | 0.82 | 21.0 | 39.5 | 0.79 | | mem0-rerank | 0.72 | 0.90 | 0.83 | 148.5 | 256.7 | 0.65 | | cosine | 0.74 | 0.88 | 0.82 | 2.3 | 5.5 | 0.47 | The latency columns come from the first of the four runs. Other agents were working on the same machine during the later runs (load average 30 to 60 on 8 cores), and the medians rose to 7–20 ms for `old`, 36–47 ms for `mem0` and 170–560 ms for the reranker variants. In every run mem0 took 2 to 5 times as long as `old`, and the reranker variants were the slowest. By query type (recall@1 / recall@5 / MRR): | Type | n | old | old-rerank | mem0 | mem0-rerank | cosine | |---|---|---|---|---|---|---| | paraphrase | 21 | 0.19 / 0.76 / 0.40 | 0.57 / 0.90 / 0.72 | 0.52 / 0.90 / 0.70 | 0.57 / 0.90 / 0.72 | 0.71 / 0.90 / 0.80 | | keyword | 17 | 0.94 / 1.00 / 0.96 | 0.88 / 1.00 / 0.94 | 1.00 / 1.00 / 1.00 | 0.88 / 1.00 / 0.94 | 0.88 / 1.00 / 0.94 | | multi-fact | 12 | 0.42 / 0.61 / 0.61 | 0.75 / 0.80 / 0.82 | 0.75 / 0.69 / 0.79 | 0.75 / 0.80 / 0.82 | 0.67 / 0.67 / 0.73 | | scope | 11 | 0.45 / 0.82 / 0.66 | 0.73 / 0.86 / 0.86 | 0.64 / 0.91 / 0.80 | 0.73 / 0.86 / 0.86 | 0.64 / 0.86 / 0.78 | By the number of scopes searched: | Scopes | n | old | old-rerank | mem0 | mem0-rerank | cosine | |---|---|---|---|---|---|---| | one | 25 | 0.60 / 0.89 / 0.75 | 0.80 / 0.95 / 0.87 | 0.80 / 0.99 / 0.87 | 0.80 / 0.95 / 0.87 | 0.88 / 0.99 / 0.93 | | two or more | 36 | 0.42 / 0.75 / 0.58 | 0.67 / 0.87 / 0.79 | 0.67 / 0.82 / 0.78 | 0.67 / 0.87 / 0.79 | 0.64 / 0.80 / 0.75 | The bench prints a per-query table of misses (rank of the first relevant memory, recall@5 and the top result per engine) and the top result of every no-answer query. The examples below come from it. ## What mem0 3.3.1 does when it searches Read in `dist/oss/index.js` of the installed package (line numbers refer to that file) and checked with mem0's `explain: true` score breakdown on this set. 1. **Vector similarity.** The query is embedded once. The `memory` vector store reads every row of its `vectors` table, parses each JSON payload, keeps the rows whose `user_id` and `agent_id` match, and computes cosine similarity in JavaScript (lines 1143–1174). The table holds all scopes, so the whole store is read even when one small workspace is searched. mem0 keeps the best `max(topK × 4, 60)` rows as candidates (line 18031); the adapter asks for `topK = max(limit × 5, 100)`, so up to 400 per scope. 2. **BM25.** A second full read of the table computes BM25 (k1 1.5, b 0.75) over `textLemmatized`, which mem0 stores with each memory: lower-case ASCII words without stop words, stemmed with the Porter stemmer of the `natural` package if it is installed (lines 16171–16237). `natural` is not installed in Waggle, so mem0 falls back to a simple suffix stripper. Non-ASCII letters split words (`Škoda` becomes `koda`) on both sides, so names still match. The raw score is mapped to 0–1 with a sigmoid whose midpoint depends on the query length (lines 16855–16871). Only memories already among the vector candidates receive it. 3. **Entity boost.** Entities found in the query (quoted text, proper nouns, identifiers, noun phrases through `compromise`) are looked up in a second store, `vectors_entities.db`, and add up to 0.5 to the memories linked to them (lines 18060–18127). Links are only written when mem0 extracts facts with a model (`infer: true`). The adapter stores text with `infer: false`, which calls `createMemory` and links nothing (lines 17575–17592, 18423), so the boost was 0 for all 67 queries. Memories that P1.5 extracts from conversations will get links. 4. **Combination.** Each candidate scores `(semantic + bm25 + entity) / maxPossible`, where `maxPossible` is 1, plus 1 if any candidate of this call has a BM25 score, plus 0.5 if any has an entity boost (lines 16873–16919). The threshold applies to the semantic part only; the adapter passes 0. There is no importance, recency or access-count signal. ADR-0007 is right that mem0 search is "hybrid vector and BM25", with three qualifications: BM25 only re-ranks the vector candidates, the entity part is inactive for memories written as text, and semantic and keyword scores have equal weight. BM25 contributed to the ranking in the first searched scope of 40 of the 67 queries. The adapter searches each scope with its own call and merges by score. Because `maxPossible` is computed per call, scores from two scopes can be divided by different numbers: this happened in 12 of the 39 queries that search several scopes. Re-ranking those with a common divisor changed the first result of one query (q40), from a relevant memory to an irrelevant one, so the effect on this set is nil. ## Where each engine wins and loses ### Old engine without the reranker `HybridSearch` fuses a full-text lane and a vector lane with reciprocal rank fusion (`1 / (60 + rank)` per lane) and multiplies the result by a relevance factor. With the 'balanced' profile and new memories, that factor is 0.80 for normal, 0.90 for important and 1.00 for critical memories. The gap between rank 1 and rank 2 in a lane is under 2 %, so a 12–25 % importance bonus outweighs most differences in rank. 21 of the 31 wrong first results were critical or important memories, which make up 12 of the 100 memories. For example, "What kind of pet do I have?" returns the peanut allergy (critical) first and the greyhound fourth; "What do I charge clients?" returns the decision to take no new clients (important) first. The full-text lane joins all words of the query with OR. Its stop-word list has no pronouns or names, so any shared common word gives a memory a full vote. In a first draft of this set, personal questions were in the third person ("What kind of pet does she have?"); the old engine then ranked a relevant memory first for 1 of the 21 paraphrase queries, because "she" appears in many memories. With first-person questions it is 4 of 21. Exact names are its strength: recall@1 0.94 on keyword queries, through the full-text index. ### Old engine with the reranker (chat recall) The cross-encoder replaces the fused score with its own, which removes the importance bias: recall@1 rises to 0.72. It costs 130–140 ms per search on 100 memories and makes its own mistakes. "Would I enjoy a beef burger?" ranks a science-fiction memory first and loses the vegetarian-diet memory from the top 10; "Why was a lightweight graphing package picked?" ranks the standing-desk decision first and loses the Chart.js decision; two keyword queries (`When can I phone Grace Mbeki?`, `fernwork/harbor-dashboard`) drop from rank 1 to rank 2. The model is trained on web passages, not on short personal facts. ### mem0 mem0 ties the old engine with the reranker overall. It is best on keyword queries (recall@1 1.00): BM25 finds `ATL-142`, `#F28C28`, `PostGIS` and `fernwork.si` even where the vector alone puts another memory first. It also separates answerable from unanswerable questions best (0.79, against 0.47 for plain cosine over the same vectors, so the difference comes from BM25), though not well enough for a threshold (see "Shared failures"). It loses on paraphrases: recall@1 0.52, where the cosine baseline over the same vectors reaches 0.71. BM25 weighs as much as the vector score, so one shared ordinary word lifts a distractor above a better semantic match. "How do I like replies to be formatted?" puts "She likes to be addressed as Mila" first through "like"; "What computer do I work on?" puts "wears glasses for screen work" first through "work". Lowering the BM25 weight does not help overall. Re-scoring mem0's candidates as `semantic + w × bm25`, from the `explain: true` breakdown (a one-off check, not part of the bench), gave: | w | recall@1 | paraphrase recall@1 | keyword recall@1 | |---|---|---|---| | 0 (cosine) | 0.74 | 0.71 | 0.88 | | 0.25 | 0.72 | 0.67 | 0.88 | | 0.5 | 0.72 | 0.57 | 0.94 | | 1 (mem0's weights) | 0.70 | 0.52 | 1.00 | The weight trades paraphrase accuracy for keyword accuracy without changing the total. The last row differs from the bench's 0.72 by one query (q40), because the re-scoring used one divisor for all scopes. Questions that need several memories are its other weak spot (recall@5 0.69 against 0.80 with the reranker). "What has been decided about the Harbor tech stack?" finds none of the four relevant memories in the top five, because every Harbor memory is about Harbor; "How big are the budgets of Mila's current projects?" puts the personal decision about "current projects" first through BM25. In scope queries, a personal memory sometimes wins over the workspace answer: "When is the deadline?" in Harbor returns the personal first-quarter plan first and the board-meeting deadline third; "Who is the client contact?" in Harbor returns "emergency contact" first through BM25. Isolation between workspaces holds in every engine: no query returned a memory from a workspace it did not search. ### Shared failures - "Is there anything I must never eat?" ranks the vegetarian diet above the peanut allergy in every engine (allergy at rank 2 or 3). - "Does the app have dark mode?" in either workspace ranks Mila's personal dark-mode preference above the workspace's decision in every engine. - "What did we adjust for people wearing mittens?" ranks "wears glasses for screen work" first everywhere; the glove decision is second or third. - Every engine returns something for questions memory cannot answer. With mem0, 19 of the 44 correct first results score at or below the highest no-answer score (0.39 for "How many employees does Kestrel Freight have?"), so a fixed threshold cannot decide that memory has no answer. ## Latency and store size At 100 memories mem0 needs about 21 ms per search, the old engine 6 ms and the reranker variants about 145 ms. mem0's cost grows with the size of the whole store, because both passes read every row of the `vectors` table. A separate probe on the same machine (a deterministic stand-in embedder, so that only the store's work was timed, 21 queries each): | Store | Median search | |---|---| | 1,000 memories in one scope, 1024 dimensions | 66 ms | | 5,000 memories in one scope, 1024 dimensions | 336 ms | | 5,000 memories in one scope, 384 dimensions | 136 ms | | 5,000 memories in personal, search of a 20-memory workspace | 224 ms | The machine was busy with other work during the probe, so the absolute numbers are rough; the ratios come from runs made one after another. A search across personal memory and one workspace makes two such passes. ## Conclusion mem0 search is acceptable for v1. On this set it matches the old engine's chat-recall path, it is clearly better than the old engine's `search_memory` path, it needs no reranker model, and it runs offline with Waggle's embedder. What would improve it, in order of value for effort: 1. **Store vectors at the embedder's native size.** The sidecar pads MiniLM's 384 dimensions to 1024. Zero padding does not change cosine similarity, so the ranking stays the same, while mem0's search pass gets about 2.5 times faster. The adapter has no check like the old engine's embedding fingerprint: a store created with one embedder and read with another of the same size gives degraded results without an error. Whatever size is chosen, the store should record the embedder and refuse a mismatch. 2. **Do not tune the BM25 weight.** The table above shows a trade, not a gain. 3. **Leave the reranker out of v1.** It brings mem0 to the same numbers as the old engine's chat recall (recall@5 0.90, MRR 0.83), costs about 130 ms per search, and lives in the substrate that is being deleted, so it would have to be copied into `@waggle/core` first. 4. **Revisit the vector store when stores grow.** At a few hundred to a few thousand memories the full scan is fine. Past that, a store with an index, or a single mem0 call across all searched scopes (one pass instead of one per scope, not tested here), would be needed. ADR-0007 already notes the same for `list`, `count` and `forgetMatching`. 5. **Do not use the score to decide that memory has no answer.** Callers should rank, not threshold. What this bench does not cover: - Time. Neither store can backdate a memory in the bench (mem0 rejects timestamps), so the old engine's recency decay and questions about dates are not measured. - The rest of chat recall. The importance lane, prefix lanes and date window in `Orchestrator.recallMemory` sit outside `HybridSearch`; P1.5 maps them to contract filters. - Extracted memories. All memories are written as text; memories extracted by a model would also get mem0's entity boost. - Other embedders and languages. One English model was measured. - The merge across minds. Production renders personal and workspace results in separate sections; the bench merges them by score for every engine to get one ranking. - Size. 100 memories and 67 queries written by one author. ## How to re-run From the repository root, after `npm ci` and `npm run build:packages`: ```bash # All engines this checkout has (old, old-rerank, mem0, mem0-rerank, cosine) npx tsx packages/memory-mem0/bench/search-quality.ts # After the old engine is removed npx tsx packages/memory-mem0/bench/search-quality.ts --engines mem0,cosine # Per-query results as JSON npx tsx packages/memory-mem0/bench/search-quality.ts --out /tmp/search-quality.json ``` The in-process model must be in `~/.waggle/models` (the sidecar downloads it on first start) or in the directory named by `WAGGLE_MODELS_DIR`; the bench refuses network access and does not download it. Other embedders: ```bash # A running local Ollama; the bench neither starts it nor pulls models OLLAMA_EMBED_MODEL=nomic-embed-text npx tsx packages/memory-mem0/bench/search-quality.ts --embedder ollama # The user's own key. OPENAI_API_KEY is ignored on purpose, so a key that happens to be # set in the environment is never used. BENCH_EMBEDDING_API_KEY= BENCH_EMBEDDING_MODEL=text-embedding-3-small \ npx tsx packages/memory-mem0/bench/search-quality.ts --embedder openai ``` `BENCH_EMBEDDING_BASE_URL` points the key variant at another OpenAI-compatible endpoint. With a key, only that endpoint may be reached. To extend the set, add memories and queries to `reference-set.json`; a query lists every memory that answers it in `relevant`, and an empty list marks a question without an answer.