The bench compared mem0 with MindDB, FrameStore and HybridSearch from the hive-mind substrate, which @waggle/core no longer exports and which left the repository. It now runs mem0, mem0-rerank and cosine; the recorded comparison stays in docs/oss/research/memory-search-quality.md.
353 lines
20 KiB
Markdown
353 lines
20 KiB
Markdown
# Memory search quality: the old engine compared with mem0 (P1.5)
|
||
|
||
Measured on 2026-10-24 to decide whether mem0 search is good enough to replace the old
|
||
engine (`MindDB` + `FrameStore` + `HybridSearch` from the hive-mind substrate) before the old
|
||
engine is deleted. ADR-0007 asked for this comparison on a reference set.
|
||
|
||
The reference set and the benchmark live in `packages/memory-mem0/bench/` and keep working
|
||
after the old engine is gone (see "How to re-run").
|
||
|
||
## Summary
|
||
|
||
- On the 61 answerable queries, the mem0 store scores the same as the old engine as Waggle
|
||
uses it for chat recall (hybrid search followed by a cross-encoder reranker): recall@1
|
||
0.72 against 0.72, recall@5 0.89 against 0.90, MRR 0.82 against 0.83. mem0 answers in
|
||
about 21 ms, the old engine with its reranker in about 145 ms.
|
||
- The old engine without the reranker, which is what the agent's `search_memory` tool runs,
|
||
is clearly worse: recall@1 0.49, recall@5 0.81, MRR 0.65. Its importance boost and its
|
||
keyword lane push general "important" memories above the right answer.
|
||
- mem0 is acceptable for v1. Its weak spots are questions phrased without the memory's words
|
||
(recall@1 0.52, where plain cosine similarity reaches 0.71), questions that need several
|
||
memories (recall@5 0.69), and a search cost that grows with the whole store, not only with
|
||
the scopes searched.
|
||
|
||
## Setup
|
||
|
||
| | |
|
||
|---|---|
|
||
| Code | branch `oss/community-edition` at `f3eb6b6`, plus the bench added with this document |
|
||
| mem0 | `mem0ai` 3.3.1 through `@waggle/memory-mem0` (`createMem0MemoryStore`) |
|
||
| Embedder | Waggle's in-process embedder, `Xenova/all-MiniLM-L6-v2` (fp32, 384 dimensions zero-padded to 1024), `@huggingface/transformers` 3.8.1, model cache `~/.waggle/models`. These are the sidecar's settings (`WaggleConfig.getEmbeddingConfig`, `targetDimensions: 1024`) |
|
||
| Reranker | `Xenova/ms-marco-MiniLM-L-6-v2` through `createInProcessReranker`, cache `~/.waggle/models/reranker`, as the sidecar configures it |
|
||
| Machine | Apple M1, 8 cores, 16 GB, macOS 26.6, Node 22.23.2 |
|
||
| Network | every `fetch` to another host was refused by the bench; none was attempted. mem0 telemetry was off and `~/.mem0` was not created |
|
||
|
||
Ollama was not running on the machine (`localhost:11434` refused the connection), so only the
|
||
in-process embedder was measured. The variant with the user's own embedding key was not run;
|
||
see "How to re-run".
|
||
|
||
### Reference set
|
||
|
||
`reference-set.json` describes a fictional freelance designer, Mila Sorensen, with 44
|
||
personal memories and two client workspaces: `atlas` (30 memories, a route-planning app for a
|
||
freight company) and `harbor` (26 memories, a dashboard for a food bank). Every memory has a
|
||
kind and an importance: 3 critical, 9 important, 84 normal, 4 temporary. The set contains
|
||
near-duplicates (two dark-mode preferences, two statements of each project deadline) and
|
||
distractors (a friend's cat allergy next to Mila's peanut allergy; a dark-mode decision in
|
||
each workspace next to Mila's personal preference; a different database in each workspace).
|
||
All names and facts are invented.
|
||
|
||
The 67 queries:
|
||
|
||
| Type | n | What it tests |
|
||
|---|---|---|
|
||
| paraphrase | 21 | No content word in common with the memory to find. Checked with SQLite's porter stemmer, the one the old engine's full-text index uses |
|
||
| keyword | 17 | An exact name, code or identifier: `ATL-142`, `#F28C28`, `fernwork/harbor-dashboard`, `Dr. Okafor` |
|
||
| multi-fact | 12 | Two to four memories together answer the question |
|
||
| scope | 11 | The same question asked in different workspaces has different answers; personal memories compete |
|
||
| no-answer | 6 | Nothing in memory answers the question |
|
||
|
||
Personal questions are mostly in the first person ("What kind of pet do I have?"), as a user
|
||
asks in chat; chat recall searches with the user's message. A few use the name, as the agent
|
||
does when it writes its own `search_memory` query.
|
||
|
||
### Engines
|
||
|
||
All engines share one embedder instance, so they see the same vectors.
|
||
|
||
| Engine | What runs |
|
||
|---|---|
|
||
| `old` | One `.mind` file for personal memory and one per workspace, as in the sidecar. Memories are written with `FrameStore.createIFrame` and indexed with `HybridSearch.indexFrame`. Each searched mind runs `HybridSearch.search(query, { limit: 10, profile: 'balanced' })`; results of all searched minds are merged by `finalScore`. This is the search behind the `search_memory` tool |
|
||
| `old-rerank` | The same with the in-process cross-encoder reranker, as `Orchestrator.recallMemory` uses it for chat recall by default |
|
||
| `mem0` | `createMem0MemoryStore`, `remember({ scope, text, kind, importance })`, `search(text, { scopes, limit: 10 })`, which is what the adapter offers callers |
|
||
| `mem0-rerank` | mem0's top 30 re-ordered by the same reranker. Not part of the adapter; measured as a possible improvement |
|
||
| `cosine` | Baseline: every memory of the searched scopes ranked by cosine similarity alone, without any keyword signal |
|
||
|
||
### Metrics
|
||
|
||
- recall@k: the share of a query's relevant memories found in the first k results, out of
|
||
the most that fit (`min(k, relevant)`). recall@1 is therefore "the first result is
|
||
relevant", and recall@5 for a three-memory question is "all three are in the top five".
|
||
- MRR: mean of 1 / rank of the first relevant result within the top 10, 0 when none is
|
||
there.
|
||
- Both are averaged over the 61 answerable queries. One query moves recall@1 overall by
|
||
0.016; differences below about 0.05 are a few queries and should be read as ties.
|
||
- Abstain separation: the probability that the top score of a query whose first result is
|
||
correct is higher than the top score of a no-answer query. 1.0 would mean a score
|
||
threshold can tell "memory has an answer" from "memory has none".
|
||
- Latency: wall time of one `search` call, query embedding included, median and 95th
|
||
percentile over the 67 queries after one warm-up query.
|
||
|
||
## Results
|
||
|
||
The relevance numbers were identical in all four runs; only latency varied.
|
||
|
||
| Engine | recall@1 | recall@5 | MRR | median ms | p95 ms | abstain separation |
|
||
|---|---|---|---|---|---|---|
|
||
| old | 0.49 | 0.81 | 0.65 | 5.9 | 7.3 | 0.35 |
|
||
| old-rerank | 0.72 | 0.90 | 0.83 | 143.5 | 261.8 | 0.65 |
|
||
| mem0 | 0.72 | 0.89 | 0.82 | 21.0 | 39.5 | 0.79 |
|
||
| mem0-rerank | 0.72 | 0.90 | 0.83 | 148.5 | 256.7 | 0.65 |
|
||
| cosine | 0.74 | 0.88 | 0.82 | 2.3 | 5.5 | 0.47 |
|
||
|
||
The latency columns come from the first of the four runs. Other agents were working on the
|
||
same machine during the later runs (load average 30 to 60 on 8 cores), and the medians rose
|
||
to 7–20 ms for `old`, 36–47 ms for `mem0` and 170–560 ms for the reranker variants. In
|
||
every run mem0 took 2 to 5 times as long as `old`, and the reranker variants were the
|
||
slowest.
|
||
|
||
By query type (recall@1 / recall@5 / MRR):
|
||
|
||
| Type | n | old | old-rerank | mem0 | mem0-rerank | cosine |
|
||
|---|---|---|---|---|---|---|
|
||
| paraphrase | 21 | 0.19 / 0.76 / 0.40 | 0.57 / 0.90 / 0.72 | 0.52 / 0.90 / 0.70 | 0.57 / 0.90 / 0.72 | 0.71 / 0.90 / 0.80 |
|
||
| keyword | 17 | 0.94 / 1.00 / 0.96 | 0.88 / 1.00 / 0.94 | 1.00 / 1.00 / 1.00 | 0.88 / 1.00 / 0.94 | 0.88 / 1.00 / 0.94 |
|
||
| multi-fact | 12 | 0.42 / 0.61 / 0.61 | 0.75 / 0.80 / 0.82 | 0.75 / 0.69 / 0.79 | 0.75 / 0.80 / 0.82 | 0.67 / 0.67 / 0.73 |
|
||
| scope | 11 | 0.45 / 0.82 / 0.66 | 0.73 / 0.86 / 0.86 | 0.64 / 0.91 / 0.80 | 0.73 / 0.86 / 0.86 | 0.64 / 0.86 / 0.78 |
|
||
|
||
By the number of scopes searched:
|
||
|
||
| Scopes | n | old | old-rerank | mem0 | mem0-rerank | cosine |
|
||
|---|---|---|---|---|---|---|
|
||
| one | 25 | 0.60 / 0.89 / 0.75 | 0.80 / 0.95 / 0.87 | 0.80 / 0.99 / 0.87 | 0.80 / 0.95 / 0.87 | 0.88 / 0.99 / 0.93 |
|
||
| two or more | 36 | 0.42 / 0.75 / 0.58 | 0.67 / 0.87 / 0.79 | 0.67 / 0.82 / 0.78 | 0.67 / 0.87 / 0.79 | 0.64 / 0.80 / 0.75 |
|
||
|
||
The bench prints a per-query table of misses (rank of the first relevant memory, recall@5
|
||
and the top result per engine) and the top result of every no-answer query. The examples
|
||
below come from it.
|
||
|
||
## What mem0 3.3.1 does when it searches
|
||
|
||
Read in `dist/oss/index.js` of the installed package (line numbers refer to that file) and
|
||
checked with mem0's `explain: true` score breakdown on this set.
|
||
|
||
1. **Vector similarity.** The query is embedded once. The `memory` vector store reads every
|
||
row of its `vectors` table, parses each JSON payload, keeps the rows whose `user_id` and
|
||
`agent_id` match, and computes cosine similarity in JavaScript (lines 1143–1174). The
|
||
table holds all scopes, so the whole store is read even when one small workspace is
|
||
searched. mem0 keeps the best `max(topK × 4, 60)` rows as candidates (line 18031); the
|
||
adapter asks for `topK = max(limit × 5, 100)`, so up to 400 per scope.
|
||
2. **BM25.** A second full read of the table computes BM25 (k1 1.5, b 0.75) over
|
||
`textLemmatized`, which mem0 stores with each memory: lower-case ASCII words without stop
|
||
words, stemmed with the Porter stemmer of the `natural` package if it is installed
|
||
(lines 16171–16237). `natural` is not installed in Waggle, so mem0 falls back to a
|
||
simple suffix stripper. Non-ASCII letters split words (`Škoda` becomes `koda`) on both
|
||
sides, so names still match. The raw score is mapped to 0–1 with a sigmoid whose
|
||
midpoint depends on the query length (lines 16855–16871). Only memories already among
|
||
the vector candidates receive it.
|
||
3. **Entity boost.** Entities found in the query (quoted text, proper nouns, identifiers,
|
||
noun phrases through `compromise`) are looked up in a second store,
|
||
`vectors_entities.db`, and add up to 0.5 to the memories linked to them (lines
|
||
18060–18127). Links are only written when mem0 extracts facts with a model
|
||
(`infer: true`). The adapter stores text with `infer: false`, which calls `createMemory`
|
||
and links nothing (lines 17575–17592, 18423), so the boost was 0 for all 67 queries.
|
||
Memories that P1.5 extracts from conversations will get links.
|
||
4. **Combination.** Each candidate scores `(semantic + bm25 + entity) / maxPossible`, where
|
||
`maxPossible` is 1, plus 1 if any candidate of this call has a BM25 score, plus 0.5 if
|
||
any has an entity boost (lines 16873–16919). The threshold applies to the semantic part
|
||
only; the adapter passes 0.
|
||
|
||
There is no importance, recency or access-count signal. ADR-0007 is right that mem0 search
|
||
is "hybrid vector and BM25", with three qualifications: BM25 only re-ranks the vector
|
||
candidates, the entity part is inactive for memories written as text, and semantic and
|
||
keyword scores have equal weight. BM25 contributed to the ranking in the first searched scope
|
||
of 40 of the 67 queries.
|
||
|
||
The adapter searches each scope with its own call and merges by score. Because
|
||
`maxPossible` is computed per call, scores from two scopes can be divided by different
|
||
numbers: this happened in 12 of the 39 queries that search several scopes. Re-ranking those
|
||
with a common divisor changed the first result of one query (q40), from a relevant memory to
|
||
an irrelevant one, so the effect on this set is nil.
|
||
|
||
## Where each engine wins and loses
|
||
|
||
### Old engine without the reranker
|
||
|
||
`HybridSearch` fuses a full-text lane and a vector lane with reciprocal rank fusion
|
||
(`1 / (60 + rank)` per lane) and multiplies the result by a relevance factor. With the
|
||
'balanced' profile and new memories, that factor is 0.80 for normal, 0.90 for important and
|
||
1.00 for critical memories. The gap between rank 1 and rank 2 in a lane is under 2 %, so a
|
||
12–25 % importance bonus outweighs most differences in rank. 21 of the 31 wrong first
|
||
results were critical or important memories, which make up 12 of the 100 memories. For
|
||
example, "What kind of pet do I have?" returns the peanut allergy (critical) first and the
|
||
greyhound fourth; "What do I charge clients?" returns the decision to take no new clients
|
||
(important) first.
|
||
|
||
The full-text lane joins all words of the query with OR. Its stop-word list has no pronouns
|
||
or names, so any shared common word gives a memory a full vote. In a first draft of this
|
||
set, personal questions were in the third person ("What kind of pet does she have?"); the
|
||
old engine then ranked a relevant memory first for 1 of the 21 paraphrase queries, because
|
||
"she" appears in many memories. With first-person questions it is 4 of 21.
|
||
|
||
Exact names are its strength: recall@1 0.94 on keyword queries, through the full-text index.
|
||
|
||
### Old engine with the reranker (chat recall)
|
||
|
||
The cross-encoder replaces the fused score with its own, which removes the importance bias:
|
||
recall@1 rises to 0.72. It costs 130–140 ms per search on 100 memories and makes its own
|
||
mistakes. "Would I enjoy a beef burger?" ranks a science-fiction memory first and loses the
|
||
vegetarian-diet memory from the top 10; "Why was a lightweight graphing package picked?"
|
||
ranks the standing-desk decision first and loses the Chart.js decision; two keyword queries
|
||
(`When can I phone Grace Mbeki?`, `fernwork/harbor-dashboard`) drop from rank 1 to rank 2.
|
||
The model is trained on web passages, not on short personal facts.
|
||
|
||
### mem0
|
||
|
||
mem0 ties the old engine with the reranker overall.
|
||
|
||
It is best on keyword queries (recall@1 1.00): BM25 finds `ATL-142`, `#F28C28`, `PostGIS`
|
||
and `fernwork.si` even where the vector alone puts another memory first. It also separates
|
||
answerable from unanswerable questions best (0.79, against 0.47 for plain cosine over the
|
||
same vectors, so the difference comes from BM25), though not well enough for a threshold
|
||
(see "Shared failures").
|
||
|
||
It loses on paraphrases: recall@1 0.52, where the cosine baseline over the same vectors
|
||
reaches 0.71. BM25 weighs as much as the vector score, so one shared ordinary word lifts a
|
||
distractor above a better semantic match. "How do I like replies to be formatted?" puts
|
||
"She likes to be addressed as Mila" first through "like"; "What computer do I work on?"
|
||
puts "wears glasses for screen work" first through "work". Lowering the BM25 weight does not
|
||
help overall. Re-scoring mem0's candidates as `semantic + w × bm25`, from the `explain: true`
|
||
breakdown (a one-off check, not part of the bench), gave:
|
||
|
||
| w | recall@1 | paraphrase recall@1 | keyword recall@1 |
|
||
|---|---|---|---|
|
||
| 0 (cosine) | 0.74 | 0.71 | 0.88 |
|
||
| 0.25 | 0.72 | 0.67 | 0.88 |
|
||
| 0.5 | 0.72 | 0.57 | 0.94 |
|
||
| 1 (mem0's weights) | 0.70 | 0.52 | 1.00 |
|
||
|
||
The weight trades paraphrase accuracy for keyword accuracy without changing the total. The
|
||
last row differs from the bench's 0.72 by one query (q40), because the re-scoring used one
|
||
divisor for all scopes.
|
||
|
||
Questions that need several memories are its other weak spot (recall@5 0.69 against 0.80
|
||
with the reranker). "What has been decided about the Harbor tech stack?" finds none of the
|
||
four relevant memories in the top five, because every Harbor memory is about Harbor;
|
||
"How big are the budgets of Mila's current projects?" puts the personal decision about
|
||
"current projects" first through BM25.
|
||
|
||
In scope queries, a personal memory sometimes wins over the workspace answer:
|
||
"When is the deadline?" in Harbor returns the personal first-quarter plan first and the
|
||
board-meeting deadline third; "Who is the client contact?" in Harbor returns "emergency
|
||
contact" first through BM25. Isolation between workspaces holds in every engine: no query
|
||
returned a memory from a workspace it did not search.
|
||
|
||
### Shared failures
|
||
|
||
- "Is there anything I must never eat?" ranks the vegetarian diet above the peanut allergy
|
||
in every engine (allergy at rank 2 or 3).
|
||
- "Does the app have dark mode?" in either workspace ranks Mila's personal dark-mode
|
||
preference above the workspace's decision in every engine.
|
||
- "What did we adjust for people wearing mittens?" ranks "wears glasses for screen work"
|
||
first everywhere; the glove decision is second or third.
|
||
- Every engine returns something for questions memory cannot answer. With mem0, 19 of the
|
||
44 correct first results score at or below the highest no-answer score (0.39 for "How many
|
||
employees does Kestrel Freight have?"), so a fixed threshold cannot decide that memory has
|
||
no answer.
|
||
|
||
## Latency and store size
|
||
|
||
At 100 memories mem0 needs about 21 ms per search, the old engine 6 ms and the reranker
|
||
variants about 145 ms. mem0's cost grows with the size of the whole store, because both
|
||
passes read every row of the `vectors` table. A separate probe on the same machine (a
|
||
deterministic stand-in embedder, so that only the store's work was timed, 21 queries each):
|
||
|
||
| Store | Median search |
|
||
|---|---|
|
||
| 1,000 memories in one scope, 1024 dimensions | 66 ms |
|
||
| 5,000 memories in one scope, 1024 dimensions | 336 ms |
|
||
| 5,000 memories in one scope, 384 dimensions | 136 ms |
|
||
| 5,000 memories in personal, search of a 20-memory workspace | 224 ms |
|
||
|
||
The machine was busy with other work during the probe, so the absolute numbers are rough;
|
||
the ratios come from runs made one after another. A search across personal memory and one
|
||
workspace makes two such passes.
|
||
|
||
## Conclusion
|
||
|
||
mem0 search is acceptable for v1. On this set it matches the old engine's chat-recall path,
|
||
it is clearly better than the old engine's `search_memory` path, it needs no reranker model,
|
||
and it runs offline with Waggle's embedder.
|
||
|
||
What would improve it, in order of value for effort:
|
||
|
||
1. **Store vectors at the embedder's native size.** The sidecar pads MiniLM's 384
|
||
dimensions to 1024. Zero padding does not change cosine similarity, so the ranking stays
|
||
the same, while mem0's search pass gets about 2.5 times faster. The adapter has no check
|
||
like the old engine's embedding fingerprint: a store created with one embedder and read
|
||
with another of the same size gives degraded results without an error. Whatever size is
|
||
chosen, the store should record the embedder and refuse a mismatch.
|
||
2. **Do not tune the BM25 weight.** The table above shows a trade, not a gain.
|
||
3. **Leave the reranker out of v1.** It brings mem0 to the same numbers as the old
|
||
engine's chat recall (recall@5 0.90, MRR 0.83), costs about 130 ms per search, and lives
|
||
in the substrate that is being deleted, so it would have to be copied into `@waggle/core`
|
||
first.
|
||
4. **Revisit the vector store when stores grow.** At a few hundred to a few thousand
|
||
memories the full scan is fine. Past that, a store with an index, or a single mem0 call
|
||
across all searched scopes (one pass instead of one per scope, not tested here), would
|
||
be needed. ADR-0007 already notes the same for `list`, `count` and `forgetMatching`.
|
||
5. **Do not use the score to decide that memory has no answer.** Callers should rank, not
|
||
threshold.
|
||
|
||
What this bench does not cover:
|
||
|
||
- Time. Neither store can backdate a memory in the bench (mem0 rejects timestamps), so
|
||
the old engine's recency decay and questions about dates are not measured.
|
||
- The rest of chat recall. The importance lane, prefix lanes and date window in
|
||
`Orchestrator.recallMemory` sit outside `HybridSearch`; P1.5 maps them to contract
|
||
filters.
|
||
- Extracted memories. All memories are written as text; memories extracted by a model would
|
||
also get mem0's entity boost.
|
||
- Other embedders and languages. One English model was measured.
|
||
- The merge across minds. Production renders personal and workspace results in separate
|
||
sections; the bench merges them by score for every engine to get one ranking.
|
||
- Size. 100 memories and 67 queries written by one author.
|
||
|
||
## How to re-run
|
||
|
||
From the repository root, after `npm ci` and `npm run build:packages`. The old engine left
|
||
the repository in P2.2, so the bench runs mem0, mem0-rerank and cosine:
|
||
|
||
```bash
|
||
# All engines (mem0, mem0-rerank, cosine)
|
||
npx tsx packages/memory-mem0/bench/search-quality.ts
|
||
|
||
# Without the reranker model
|
||
npx tsx packages/memory-mem0/bench/search-quality.ts --engines mem0,cosine
|
||
|
||
# Per-query results as JSON
|
||
npx tsx packages/memory-mem0/bench/search-quality.ts --out /tmp/search-quality.json
|
||
```
|
||
|
||
The in-process model must be in `~/.waggle/models` (the sidecar downloads it on first start)
|
||
or in the directory named by `WAGGLE_MODELS_DIR`; the bench refuses network access and does
|
||
not download it.
|
||
|
||
Other embedders:
|
||
|
||
```bash
|
||
# A running local Ollama; the bench neither starts it nor pulls models
|
||
OLLAMA_EMBED_MODEL=nomic-embed-text npx tsx packages/memory-mem0/bench/search-quality.ts --embedder ollama
|
||
|
||
# The user's own key. OPENAI_API_KEY is ignored on purpose, so a key that happens to be
|
||
# set in the environment is never used.
|
||
BENCH_EMBEDDING_API_KEY=<key> BENCH_EMBEDDING_MODEL=text-embedding-3-small \
|
||
npx tsx packages/memory-mem0/bench/search-quality.ts --embedder openai
|
||
```
|
||
|
||
`BENCH_EMBEDDING_BASE_URL` points the key variant at another OpenAI-compatible endpoint.
|
||
With a key, only that endpoint may be reached.
|
||
|
||
To extend the set, add memories and queries to `reference-set.json`; a query lists every
|
||
memory that answers it in `relevant`, and an empty list marks a question without an answer.
|