test(memory): compare search quality of the old engine and mem0 on a reference set (P1.5)

Add a reference set (100 invented memories across personal memory and two
workspaces, 67 queries: paraphrase, keyword, multi-fact, scope, no-answer)
and a tsx benchmark that loads it into the old engine (MindDB, FrameStore,
HybridSearch, with and without the cross-encoder reranker), the mem0 store
and a plain-cosine baseline, all on Waggle's in-process embedder. It reports
recall@1, recall@5, MRR, latency and misses per engine and query type. The
mem0 half runs on its own (--engines mem0,cosine) once the old engine is
removed. Ollama and the user's own key are optional embedders; the bench
refuses network access to anything else and never reads OPENAI_API_KEY.

Results in docs/oss/research/memory-search-quality.md: mem0 matches the old
engine's chat-recall path (recall@1 0.72, recall@5 0.89 vs 0.90, MRR 0.82
vs 0.83) at a fraction of its latency and is acceptable for v1.

No new dependencies.
This commit is contained in:
shumeiko.it
2026-10-24 11:55:39 +02:00
parent 784ca770b1
commit f046a20b34
3 changed files with 1107 additions and 0 deletions

View File

@@ -0,0 +1,351 @@
# Memory search quality: the old engine compared with mem0 (P1.5)
Measured on 2026-10-24 to decide whether mem0 search is good enough to replace the old
engine (`MindDB` + `FrameStore` + `HybridSearch` from the hive-mind substrate) before the old
engine is deleted. ADR-0007 asked for this comparison on a reference set.
The reference set and the benchmark live in `packages/memory-mem0/bench/` and keep working
after the old engine is gone (see "How to re-run").
## Summary
- On the 61 answerable queries, the mem0 store scores the same as the old engine as Waggle
uses it for chat recall (hybrid search followed by a cross-encoder reranker): recall@1
0.72 against 0.72, recall@5 0.89 against 0.90, MRR 0.82 against 0.83. mem0 answers in
about 21 ms, the old engine with its reranker in about 145 ms.
- The old engine without the reranker, which is what the agent's `search_memory` tool runs,
is clearly worse: recall@1 0.49, recall@5 0.81, MRR 0.65. Its importance boost and its
keyword lane push general "important" memories above the right answer.
- mem0 is acceptable for v1. Its weak spots are questions phrased without the memory's words
(recall@1 0.52, where plain cosine similarity reaches 0.71), questions that need several
memories (recall@5 0.69), and a search cost that grows with the whole store, not only with
the scopes searched.
## Setup
| | |
|---|---|
| Code | branch `oss/community-edition` at `f3eb6b6`, plus the bench added with this document |
| mem0 | `mem0ai` 3.3.1 through `@waggle/memory-mem0` (`createMem0MemoryStore`) |
| Embedder | Waggle's in-process embedder, `Xenova/all-MiniLM-L6-v2` (fp32, 384 dimensions zero-padded to 1024), `@huggingface/transformers` 3.8.1, model cache `~/.waggle/models`. These are the sidecar's settings (`WaggleConfig.getEmbeddingConfig`, `targetDimensions: 1024`) |
| Reranker | `Xenova/ms-marco-MiniLM-L-6-v2` through `createInProcessReranker`, cache `~/.waggle/models/reranker`, as the sidecar configures it |
| Machine | Apple M1, 8 cores, 16 GB, macOS 26.6, Node 22.23.2 |
| Network | every `fetch` to another host was refused by the bench; none was attempted. mem0 telemetry was off and `~/.mem0` was not created |
Ollama was not running on the machine (`localhost:11434` refused the connection), so only the
in-process embedder was measured. The variant with the user's own embedding key was not run;
see "How to re-run".
### Reference set
`reference-set.json` describes a fictional freelance designer, Mila Sorensen, with 44
personal memories and two client workspaces: `atlas` (30 memories, a route-planning app for a
freight company) and `harbor` (26 memories, a dashboard for a food bank). Every memory has a
kind and an importance: 3 critical, 9 important, 84 normal, 4 temporary. The set contains
near-duplicates (two dark-mode preferences, two statements of each project deadline) and
distractors (a friend's cat allergy next to Mila's peanut allergy; a dark-mode decision in
each workspace next to Mila's personal preference; a different database in each workspace).
All names and facts are invented.
The 67 queries:
| Type | n | What it tests |
|---|---|---|
| paraphrase | 21 | No content word in common with the memory to find. Checked with SQLite's porter stemmer, the one the old engine's full-text index uses |
| keyword | 17 | An exact name, code or identifier: `ATL-142`, `#F28C28`, `fernwork/harbor-dashboard`, `Dr. Okafor` |
| multi-fact | 12 | Two to four memories together answer the question |
| scope | 11 | The same question asked in different workspaces has different answers; personal memories compete |
| no-answer | 6 | Nothing in memory answers the question |
Personal questions are mostly in the first person ("What kind of pet do I have?"), as a user
asks in chat; chat recall searches with the user's message. A few use the name, as the agent
does when it writes its own `search_memory` query.
### Engines
All engines share one embedder instance, so they see the same vectors.
| Engine | What runs |
|---|---|
| `old` | One `.mind` file for personal memory and one per workspace, as in the sidecar. Memories are written with `FrameStore.createIFrame` and indexed with `HybridSearch.indexFrame`. Each searched mind runs `HybridSearch.search(query, { limit: 10, profile: 'balanced' })`; results of all searched minds are merged by `finalScore`. This is the search behind the `search_memory` tool |
| `old-rerank` | The same with the in-process cross-encoder reranker, as `Orchestrator.recallMemory` uses it for chat recall by default |
| `mem0` | `createMem0MemoryStore`, `remember({ scope, text, kind, importance })`, `search(text, { scopes, limit: 10 })`, which is what the adapter offers callers |
| `mem0-rerank` | mem0's top 30 re-ordered by the same reranker. Not part of the adapter; measured as a possible improvement |
| `cosine` | Baseline: every memory of the searched scopes ranked by cosine similarity alone, without any keyword signal |
### Metrics
- recall@k: the share of a query's relevant memories found in the first k results, out of
the most that fit (`min(k, relevant)`). recall@1 is therefore "the first result is
relevant", and recall@5 for a three-memory question is "all three are in the top five".
- MRR: mean of 1 / rank of the first relevant result within the top 10, 0 when none is
there.
- Both are averaged over the 61 answerable queries. One query moves recall@1 overall by
0.016; differences below about 0.05 are a few queries and should be read as ties.
- Abstain separation: the probability that the top score of a query whose first result is
correct is higher than the top score of a no-answer query. 1.0 would mean a score
threshold can tell "memory has an answer" from "memory has none".
- Latency: wall time of one `search` call, query embedding included, median and 95th
percentile over the 67 queries after one warm-up query.
## Results
The relevance numbers were identical in all four runs; only latency varied.
| Engine | recall@1 | recall@5 | MRR | median ms | p95 ms | abstain separation |
|---|---|---|---|---|---|---|
| old | 0.49 | 0.81 | 0.65 | 5.9 | 7.3 | 0.35 |
| old-rerank | 0.72 | 0.90 | 0.83 | 143.5 | 261.8 | 0.65 |
| mem0 | 0.72 | 0.89 | 0.82 | 21.0 | 39.5 | 0.79 |
| mem0-rerank | 0.72 | 0.90 | 0.83 | 148.5 | 256.7 | 0.65 |
| cosine | 0.74 | 0.88 | 0.82 | 2.3 | 5.5 | 0.47 |
The latency columns come from the first of the four runs. Other agents were working on the
same machine during the later runs (load average 30 to 60 on 8 cores), and the medians rose
to 7–20 ms for `old`, 36–47 ms for `mem0` and 170–560 ms for the reranker variants. In
every run mem0 took 2 to 5 times as long as `old`, and the reranker variants were the
slowest.
By query type (recall@1 / recall@5 / MRR):
| Type | n | old | old-rerank | mem0 | mem0-rerank | cosine |
|---|---|---|---|---|---|---|
| paraphrase | 21 | 0.19 / 0.76 / 0.40 | 0.57 / 0.90 / 0.72 | 0.52 / 0.90 / 0.70 | 0.57 / 0.90 / 0.72 | 0.71 / 0.90 / 0.80 |
| keyword | 17 | 0.94 / 1.00 / 0.96 | 0.88 / 1.00 / 0.94 | 1.00 / 1.00 / 1.00 | 0.88 / 1.00 / 0.94 | 0.88 / 1.00 / 0.94 |
| multi-fact | 12 | 0.42 / 0.61 / 0.61 | 0.75 / 0.80 / 0.82 | 0.75 / 0.69 / 0.79 | 0.75 / 0.80 / 0.82 | 0.67 / 0.67 / 0.73 |
| scope | 11 | 0.45 / 0.82 / 0.66 | 0.73 / 0.86 / 0.86 | 0.64 / 0.91 / 0.80 | 0.73 / 0.86 / 0.86 | 0.64 / 0.86 / 0.78 |
By the number of scopes searched:
| Scopes | n | old | old-rerank | mem0 | mem0-rerank | cosine |
|---|---|---|---|---|---|---|
| one | 25 | 0.60 / 0.89 / 0.75 | 0.80 / 0.95 / 0.87 | 0.80 / 0.99 / 0.87 | 0.80 / 0.95 / 0.87 | 0.88 / 0.99 / 0.93 |
| two or more | 36 | 0.42 / 0.75 / 0.58 | 0.67 / 0.87 / 0.79 | 0.67 / 0.82 / 0.78 | 0.67 / 0.87 / 0.79 | 0.64 / 0.80 / 0.75 |
The bench prints a per-query table of misses (rank of the first relevant memory, recall@5
and the top result per engine) and the top result of every no-answer query. The examples
below come from it.
## What mem0 3.3.1 does when it searches
Read in `dist/oss/index.js` of the installed package (line numbers refer to that file) and
checked with mem0's `explain: true` score breakdown on this set.
1. **Vector similarity.** The query is embedded once. The `memory` vector store reads every
row of its `vectors` table, parses each JSON payload, keeps the rows whose `user_id` and
`agent_id` match, and computes cosine similarity in JavaScript (lines 1143–1174). The
table holds all scopes, so the whole store is read even when one small workspace is
searched. mem0 keeps the best `max(topK × 4, 60)` rows as candidates (line 18031); the
adapter asks for `topK = max(limit × 5, 100)`, so up to 400 per scope.
2. **BM25.** A second full read of the table computes BM25 (k1 1.5, b 0.75) over
`textLemmatized`, which mem0 stores with each memory: lower-case ASCII words without stop
words, stemmed with the Porter stemmer of the `natural` package if it is installed
(lines 16171–16237). `natural` is not installed in Waggle, so mem0 falls back to a
simple suffix stripper. Non-ASCII letters split words (`Škoda` becomes `koda`) on both
sides, so names still match. The raw score is mapped to 0–1 with a sigmoid whose
midpoint depends on the query length (lines 16855–16871). Only memories already among
the vector candidates receive it.
3. **Entity boost.** Entities found in the query (quoted text, proper nouns, identifiers,
noun phrases through `compromise`) are looked up in a second store,
`vectors_entities.db`, and add up to 0.5 to the memories linked to them (lines
18060–18127). Links are only written when mem0 extracts facts with a model
(`infer: true`). The adapter stores text with `infer: false`, which calls `createMemory`
and links nothing (lines 17575–17592, 18423), so the boost was 0 for all 67 queries.
Memories that P1.5 extracts from conversations will get links.
4. **Combination.** Each candidate scores `(semantic + bm25 + entity) / maxPossible`, where
`maxPossible` is 1, plus 1 if any candidate of this call has a BM25 score, plus 0.5 if
any has an entity boost (lines 16873–16919). The threshold applies to the semantic part
only; the adapter passes 0.
There is no importance, recency or access-count signal. ADR-0007 is right that mem0 search
is "hybrid vector and BM25", with three qualifications: BM25 only re-ranks the vector
candidates, the entity part is inactive for memories written as text, and semantic and
keyword scores have equal weight. BM25 contributed to the ranking in the first searched scope
of 40 of the 67 queries.
The adapter searches each scope with its own call and merges by score. Because
`maxPossible` is computed per call, scores from two scopes can be divided by different
numbers: this happened in 12 of the 39 queries that search several scopes. Re-ranking those
with a common divisor changed the first result of one query (q40), from a relevant memory to
an irrelevant one, so the effect on this set is nil.
## Where each engine wins and loses
### Old engine without the reranker
`HybridSearch` fuses a full-text lane and a vector lane with reciprocal rank fusion
(`1 / (60 + rank)` per lane) and multiplies the result by a relevance factor. With the
'balanced' profile and new memories, that factor is 0.80 for normal, 0.90 for important and
1.00 for critical memories. The gap between rank 1 and rank 2 in a lane is under 2 %, so a
12–25 % importance bonus outweighs most differences in rank. 21 of the 31 wrong first
results were critical or important memories, which make up 12 of the 100 memories. For
example, "What kind of pet do I have?" returns the peanut allergy (critical) first and the
greyhound fourth; "What do I charge clients?" returns the decision to take no new clients
(important) first.
The full-text lane joins all words of the query with OR. Its stop-word list has no pronouns
or names, so any shared common word gives a memory a full vote. In a first draft of this
set, personal questions were in the third person ("What kind of pet does she have?"); the
old engine then ranked a relevant memory first for 1 of the 21 paraphrase queries, because
"she" appears in many memories. With first-person questions it is 4 of 21.
Exact names are its strength: recall@1 0.94 on keyword queries, through the full-text index.
### Old engine with the reranker (chat recall)
The cross-encoder replaces the fused score with its own, which removes the importance bias:
recall@1 rises to 0.72. It costs 130–140 ms per search on 100 memories and makes its own
mistakes. "Would I enjoy a beef burger?" ranks a science-fiction memory first and loses the
vegetarian-diet memory from the top 10; "Why was a lightweight graphing package picked?"
ranks the standing-desk decision first and loses the Chart.js decision; two keyword queries
(`When can I phone Grace Mbeki?`, `fernwork/harbor-dashboard`) drop from rank 1 to rank 2.
The model is trained on web passages, not on short personal facts.
### mem0
mem0 ties the old engine with the reranker overall.
It is best on keyword queries (recall@1 1.00): BM25 finds `ATL-142`, `#F28C28`, `PostGIS`
and `fernwork.si` even where the vector alone puts another memory first. It also separates
answerable from unanswerable questions best (0.79, against 0.47 for plain cosine over the
same vectors, so the difference comes from BM25), though not well enough for a threshold
(see "Shared failures").
It loses on paraphrases: recall@1 0.52, where the cosine baseline over the same vectors
reaches 0.71. BM25 weighs as much as the vector score, so one shared ordinary word lifts a
distractor above a better semantic match. "How do I like replies to be formatted?" puts
"She likes to be addressed as Mila" first through "like"; "What computer do I work on?"
puts "wears glasses for screen work" first through "work". Lowering the BM25 weight does not
help overall. Re-scoring mem0's candidates as `semantic + w × bm25`, from the `explain: true`
breakdown (a one-off check, not part of the bench), gave:
| w | recall@1 | paraphrase recall@1 | keyword recall@1 |
|---|---|---|---|
| 0 (cosine) | 0.74 | 0.71 | 0.88 |
| 0.25 | 0.72 | 0.67 | 0.88 |
| 0.5 | 0.72 | 0.57 | 0.94 |
| 1 (mem0's weights) | 0.70 | 0.52 | 1.00 |
The weight trades paraphrase accuracy for keyword accuracy without changing the total. The
last row differs from the bench's 0.72 by one query (q40), because the re-scoring used one
divisor for all scopes.
Questions that need several memories are its other weak spot (recall@5 0.69 against 0.80
with the reranker). "What has been decided about the Harbor tech stack?" finds none of the
four relevant memories in the top five, because every Harbor memory is about Harbor;
"How big are the budgets of Mila's current projects?" puts the personal decision about
"current projects" first through BM25.
In scope queries, a personal memory sometimes wins over the workspace answer:
"When is the deadline?" in Harbor returns the personal first-quarter plan first and the
board-meeting deadline third; "Who is the client contact?" in Harbor returns "emergency
contact" first through BM25. Isolation between workspaces holds in every engine: no query
returned a memory from a workspace it did not search.
### Shared failures
- "Is there anything I must never eat?" ranks the vegetarian diet above the peanut allergy
in every engine (allergy at rank 2 or 3).
- "Does the app have dark mode?" in either workspace ranks Mila's personal dark-mode
preference above the workspace's decision in every engine.
- "What did we adjust for people wearing mittens?" ranks "wears glasses for screen work"
first everywhere; the glove decision is second or third.
- Every engine returns something for questions memory cannot answer. With mem0, 19 of the
44 correct first results score at or below the highest no-answer score (0.39 for "How many
employees does Kestrel Freight have?"), so a fixed threshold cannot decide that memory has
no answer.
## Latency and store size
At 100 memories mem0 needs about 21 ms per search, the old engine 6 ms and the reranker
variants about 145 ms. mem0's cost grows with the size of the whole store, because both
passes read every row of the `vectors` table. A separate probe on the same machine (a
deterministic stand-in embedder, so that only the store's work was timed, 21 queries each):
| Store | Median search |
|---|---|
| 1,000 memories in one scope, 1024 dimensions | 66 ms |
| 5,000 memories in one scope, 1024 dimensions | 336 ms |
| 5,000 memories in one scope, 384 dimensions | 136 ms |
| 5,000 memories in personal, search of a 20-memory workspace | 224 ms |
The machine was busy with other work during the probe, so the absolute numbers are rough;
the ratios come from runs made one after another. A search across personal memory and one
workspace makes two such passes.
## Conclusion
mem0 search is acceptable for v1. On this set it matches the old engine's chat-recall path,
it is clearly better than the old engine's `search_memory` path, it needs no reranker model,
and it runs offline with Waggle's embedder.
What would improve it, in order of value for effort:
1. **Store vectors at the embedder's native size.** The sidecar pads MiniLM's 384
dimensions to 1024. Zero padding does not change cosine similarity, so the ranking stays
the same, while mem0's search pass gets about 2.5 times faster. The adapter has no check
like the old engine's embedding fingerprint: a store created with one embedder and read
with another of the same size gives degraded results without an error. Whatever size is
chosen, the store should record the embedder and refuse a mismatch.
2. **Do not tune the BM25 weight.** The table above shows a trade, not a gain.
3. **Leave the reranker out of v1.** It brings mem0 to the same numbers as the old
engine's chat recall (recall@5 0.90, MRR 0.83), costs about 130 ms per search, and lives
in the substrate that is being deleted, so it would have to be copied into `@waggle/core`
first.
4. **Revisit the vector store when stores grow.** At a few hundred to a few thousand
memories the full scan is fine. Past that, a store with an index, or a single mem0 call
across all searched scopes (one pass instead of one per scope, not tested here), would
be needed. ADR-0007 already notes the same for `list`, `count` and `forgetMatching`.
5. **Do not use the score to decide that memory has no answer.** Callers should rank, not
threshold.
What this bench does not cover:
- Time. Neither store can backdate a memory in the bench (mem0 rejects timestamps), so
the old engine's recency decay and questions about dates are not measured.
- The rest of chat recall. The importance lane, prefix lanes and date window in
`Orchestrator.recallMemory` sit outside `HybridSearch`; P1.5 maps them to contract
filters.
- Extracted memories. All memories are written as text; memories extracted by a model would
also get mem0's entity boost.
- Other embedders and languages. One English model was measured.
- The merge across minds. Production renders personal and workspace results in separate
sections; the bench merges them by score for every engine to get one ranking.
- Size. 100 memories and 67 queries written by one author.
## How to re-run
From the repository root, after `npm ci` and `npm run build:packages`:
```bash
# All engines this checkout has (old, old-rerank, mem0, mem0-rerank, cosine)
npx tsx packages/memory-mem0/bench/search-quality.ts
# After the old engine is removed
npx tsx packages/memory-mem0/bench/search-quality.ts --engines mem0,cosine
# Per-query results as JSON
npx tsx packages/memory-mem0/bench/search-quality.ts --out /tmp/search-quality.json
```
The in-process model must be in `~/.waggle/models` (the sidecar downloads it on first start)
or in the directory named by `WAGGLE_MODELS_DIR`; the bench refuses network access and does
not download it.
Other embedders:
```bash
# A running local Ollama; the bench neither starts it nor pulls models
OLLAMA_EMBED_MODEL=nomic-embed-text npx tsx packages/memory-mem0/bench/search-quality.ts --embedder ollama
# The user's own key. OPENAI_API_KEY is ignored on purpose, so a key that happens to be
# set in the environment is never used.
BENCH_EMBEDDING_API_KEY=<key> BENCH_EMBEDDING_MODEL=text-embedding-3-small \
npx tsx packages/memory-mem0/bench/search-quality.ts --embedder openai
```
`BENCH_EMBEDDING_BASE_URL` points the key variant at another OpenAI-compatible endpoint.
With a key, only that endpoint may be reached.
To extend the set, add memories and queries to `reference-set.json`; a query lists every
memory that answers it in `relevant`, and an empty list marks a question without an answer.