waggle-os/docs/briefs/2026-04-26-retrieval-v2-embeddings-audit-brief.md

# Retrieval V2 + Embeddings Audit Brief

**Date:** 2026-04-26
**Author:** PM
**Status:** Authored awaiting Marko ratification + downstream sequencing
**Target executor:** CC-3 (or CC-1 after Phase 5 re-pilot completion; or CC-2 after Step 3 sync workflow live)
**Scope corollary:** Korak 2 iz 14-step launch plan (memory retrieval V2 + embeddings audit)

---

## §1 — Goal

Production memory retrieval (V1 — `packages/core/src/mind/search.ts` HybridSearch) underperforms substrate ceiling on canonical Stage 3 v6 N=400 LoCoMo benchmark. V2 work closes the gap.

**Empirical anchors (LOCKED, both methodologies cited explicitly per `feedback_config_inheritance_audit.md` Extension 2):**

| Cell | Trio-strict (canonical paper claim) | Self-judge (apples-to-apples per Mem0 methodology) |
|------|-------------------------------------|---------------------------------------------------|
| oracle-context | 33.50% (134/400) | 74.0% (296/400) |
| full-context | 27.25% (109/400) | (TBD if not separately re-judged) |
| retrieval (V1) | **22.25% (89/400)** | (TBD — was reported as ~48% in prior PM docs but methodology unclear) |
| no-context | 3.00% (12/400) | (TBD) |

**Critical finding from Stage 3 v6 trio-strict numbers:** V1 retrieval (22.25%) is *below* full-context (27.25%) by 5pp. This means **V1 retrieval actively hurts vs. giving model entire conversation history**, on synthesis tasks where conversation fits in context window. Retrieval has no production deployment justification at current quality unless context window forces it.

**V2 targets (binding goals — both methodologies):**

- Trio-strict: V1 22.25% → V2 ≥30% (close at least 70% of 11.25pp gap to oracle ceiling)
- Trio-strict: V2 must **beat full-context (27.25%)** as deployment threshold — below this, retrieval is net-negative
- Self-judge: re-judge V1 retrieval JSONL with current self-judge methodology to establish proper baseline; V2 target = close 70% of gap to 74% oracle ceiling
- Production deployment threshold: V2 retrieval must beat full-context on BOTH methodologies, not just one

---

## §2 — Current state audit (V1 implementation)

### 2.1 — search.ts (HybridSearch RRF fusion)

**Implementation summary (verified via direct code read):**

```
keyword (FTS5) || vector (sqlite-vec)  → RRF fusion (K=60)  → relevance multiplier  → final score
```

- RRF_K = 60 hardcoded — standard literature value, no empirical tuning evidence in repo
- Keyword query construction: OR-based (W3.6 commit) with stop word filtering (60+ English stop words) and min word length > 2
- Vector search: sqlite-vec MATCH with k=limit*3 fallback for GOP-scoped, k=limit for global
- No learned reranker (cross-encoder, MS-MARCO style)
- No query expansion (synonyms, entity extraction, KG-aware reformulation)
- No semantic-only fallback path; if both keyword and vector fail, returns empty

**Identified weak points:**

1. **Stop word list is English-only.** Marko's conversational corpus likely includes Serbian (per CLAUDE.md ekavica preference). Multi-language stop word handling is missing. Plus stop word removal is destructive — `"ko je rekao šta"` (`"who said what"` in Serbian) becomes empty after filtering.

2. **Min word length > 2 filters legitimate 2-char tokens** (AI, OK, LM, v6, JS, TS). Domain-specific terminology lost.

3. **OR-based query** maximizes recall but tanks precision. No phrase matching (e.g., `"Q4 strategy"` as bigram), no fuzzy matching for typos/variants.

4. **No re-ranking.** RRF + relevance multiplication is baseline algorithm. Modern memory systems use cross-encoder rerankers on top-K candidates (e.g., cohere-rerank-v3 or local fine-tuned BGE-rerank).

5. **No query expansion.** Question `"what did Marko say about pricing"` does not trigger entity extraction (`Marko` → entity ID for KG traversal) or topic expansion (`pricing` → related concepts).

6. **GOP scope is good** for session-isolated retrieval (per amendment v2 §4 of pilot brief), but does not bridge knowledge graph traversal across sessions when query references cross-session entities.

### 2.2 — scoring.ts (relevance multiplier)

**Implementation summary:**

```
final_score = rrf_score × relevance_score
relevance_score = (temporal × w_t) + (popularity × w_p) + (contextual × w_c) + (importance × w_i)
```

Four hardcoded profiles:
- `balanced`: 0.4/0.2/0.2/0.2
- `recent`: 0.6/0.1/0.2/0.1
- `important`: 0.1/0.1/0.2/0.6
- `connected`: 0.1/0.1/0.6/0.2

Sub-scores:
- temporal: exponential decay, 30-day half-life, 7-day recency boost (1.0 ceiling)
- popularity: `1 + log10(1+access_count) × 0.1` (max ~0.5 boost on 1000 accesses)
- contextual: BFS graph distance (0 → 1.0, 1 → 0.7, 2 → 0.4, 3 → 0.2, else 0)
- importance: `IMPORTANCE_WEIGHTS` (critical=2.0, important=1.5, normal=1.0, temporary=0.7, deprecated=0.3)

**Identified weak points:**

1. **Contextual signal often unused in practice.** `graphDistances` is optional in `ScoringContext`. If caller doesn't pre-compute graph traversal and pass results, contextual = 0 across the board. HybridSearch.search() default flow does NOT compute graph distances — meaning the connected profile's 0.6 weight is multiplied by 0 most of the time. Effectively dead code for typical retrieval calls.

2. **Per-task-type weight tuning is non-existent.** Synthesis tasks may benefit from `important` profile (high importance weight for marked critical knowledge). Coordination tasks may benefit from `recent` (latest thread state matters). But selection requires explicit caller specification — no auto-routing per query type.

3. **Multiplication of rrf × relevance** amplifies bias. If rrf score is high but relevance is borderline (e.g., 0.5), final = rrf * 0.5 — half rank. Additive `rrf + α·relevance` may be more robust. No empirical evidence to justify multiplication choice in repo.

4. **30-day temporal half-life is arbitrary.** Some conversational memory has 365-day relevance (long-running consulting engagement). Some has 24-hour relevance (active incident). Per-user or per-domain tuning missing.

5. **Importance weights are hardcoded.** No empirical calibration against user feedback (did user act on results marked critical vs normal?).

### 2.3 — entity-normalizer.ts

**Implementation summary:** 10 hardcoded alias groups (postgres/postgresql/pg, javascript/js, typescript/ts, kubernetes/k8s, etc.).

**Identified weak points:**

1. **Domain coverage is thin.** Project-specific entities (KVARK, hive-mind, Waggle, Egzakta, LM TEK, ChainSight, Helix, Quanta, NorthLane, etc.) not in list. No mechanism to extend list per-deployment.

2. **No fuzzy matching.** `"GitHub"` → `"github"` works (case-only). `"Git Hub"` → no match. Lemmatization absent (`"Markov"` → `"Marko"` for possessive forms).

3. **No entity extraction from text.** Entity normalizer normalizes entities given as input; does not extract candidates from unstructured text. Knowledge graph population depends on external entity extraction (probably LLM call in harvest pipeline).

### 2.4 — embedding-provider.ts (provider chain + tier gating)

**Implementation summary:**

- Provider chain (auto-detect): `inprocess` → `ollama` → `voyage` → `openai` → `mock` fallback
- Default models per provider:
  - inprocess: `Xenova/all-MiniLM-L6-v2` (384 native dimensions)
  - ollama: `nomic-embed-text` (768 native)
  - voyage: `voyage-3-lite` (TBD native dims)
  - openai: `text-embedding-3-small` (1536 native, configurable)
- Default `targetDimensions = 1024` — pads or truncates from native to 1024
- Tier gating + monthly quota tracking (Solo/Pro/Teams capabilities)
- Mock fallback is deterministic but **semantically meaningless** (TextEncoder bytes / 128); explicitly documented as last resort

**Identified weak points:**

1. **Default inprocess model `Xenova/all-MiniLM-L6-v2` is small (22M params, 384 native dims).** Modern competitive embedding models are 100M-1B+ parameters: BGE-base-en-v1.5 (110M, 768 dims), nomic-embed-text-v1.5 (137M, 768 dims), gte-Qwen2-1.5B-instruct (1.5B, 1536 dims). MiniLM-L6 is 2019-era technology; conversational memory benchmarks favor newer models by 5-10pp.

2. **Padding/truncation to 1024 is suboptimal.** If native is 384, padding to 1024 with zeros wastes 60% of vector space and dilutes similarity scores. If native is 1536, truncation to 1024 throws away 33% of learned signal. Native-dim retention with per-model configuration is better.

3. **Mock fallback path is invisible to caller in production failures.** If real provider fails mid-run, embedder silently switches to mock. Retrieval quality collapses but no surfaced telemetry. Phase 5 of agent fix sprint (re-pilot) MUST verify this didn't happen during Stage 3 v6 — if mock fallback was active for any portion, V1 retrieval baseline 22.25% may be over-pessimistic (real provider would do better).

4. **No re-embedding strategy on model upgrade.** If we upgrade from MiniLM-L6 to BGE-base, all stored embeddings are stale. No migration path; manual re-embed required.

5. **Stage 3 v6 actual embedder used during retrieval cell run is not documented in summary.** Need audit of `stage3-n400-v6-final-analysis.md` or equivalent to confirm which provider was active. If mock, retrieval baseline is misleading.

### 2.5 — Memory sync state for retrieval files

Per Memory Sync Audit (`decisions/2026-04-26-memory-sync-audit.md`), waggle-os and hive-mind have divergent `mind/` substrate:

- `search.ts` waggle-os 8440 bytes vs hive-mind 8689 bytes (hive-mind +3%)
- `scoring.ts` waggle-os 2887 vs hive-mind 3386 (hive-mind +17%)
- `entity-normalizer.ts` waggle-os 1167 vs hive-mind 1627 (hive-mind +39%)
- `embedding-provider.ts` waggle-os 16694 vs hive-mind 11047 (waggle-os +51%, but waggle-os has tier-gating + quota tracking that hive-mind doesn't per EXTRACTION.md scrub)

V2 work MUST start after Memory Sync Step 3 (CI/CD sync workflow) is live. Otherwise V2 changes will land in one repo and divergence will grow.

---

## §3 — Five V2 directions (binding work scope)

Per arxiv §5.3 + §7 Future Work + this audit:

### 3.1 — Direction A: Embedding model upgrade

**Hypothesis:** Default `Xenova/all-MiniLM-L6-v2` (2019-era, 384-dim) under-performs modern models by 5-10pp on conversational memory recall.

**Implementation:**
- Add `bge-base-en-v1.5` (or `bge-large-en-v1.5` for higher quality) as new in-process option via `@xenova/transformers` or `transformers.js`
- Add `gte-Qwen2-1.5B-instruct` via Ollama for users with GPU
- Add `nomic-embed-text-v1.5` as Ollama default upgrade (already supported, but version pinned)
- Per-model `nativeDimensions` exposed; abandon padding/truncation in favor of model-native dim throughout pipeline
- Add per-user `embedding_model_version` field in DB schema; migration: if field changes, re-embed all frames in background
- Telemetry hook: surface mock-fallback events to caller (so Phase 5 re-pilot can verify real embedder was used)

**Empirical validation:** ablation N=20-30 on retrieval cell with each candidate model (MiniLM-L6 baseline, BGE-base, BGE-large, nomic-v1.5, gte-Qwen2) — measure trio-strict accuracy delta and per-task-type breakdown.

**Cost:** ~$5-10 per ablation run × 5 models = $25-50.

### 3.2 — Direction B: Hybrid scoring weight optimization

**Hypothesis:** Fixed 4-profile scoring weights are suboptimal. Per-task-type tuning (factoid vs synthesis vs coordination vs decision support) plus learned per-user preference can lift retrieval recall.

**Implementation:**
- Add `task_type` parameter to `SearchOptions` (factoid / synthesis / coordination / decision-support / unknown)
- Per-task-type default scoring weights (initial: hand-tuned from Stage 3 v6 per-question-type breakdown if available, otherwise empirical sweep)
- Learnable weights per-user via implicit feedback (which retrieved memory was actually used in agent response → boost; which was ignored → demote). Out-of-scope for first V2; shipping fixed-per-task-type weights initially.
- Replace multiplication `rrf × relevance` with additive `rrf + α·relevance`; tune α in ablation.
- Auto-detect task type from question structure (heuristic: question word + length + presence of named entities → classifier). Phase 2 work; initial V2 ships with explicit `task_type` parameter, auto-detection deferred.

**Empirical validation:** ablation per profile combination on N=20-30 synthetic per-task-type subset of LoCoMo.

**Cost:** ~$10-20 per ablation × 5-10 weight combinations = $50-200.

### 3.3 — Direction C: Temporal-aware retrieval

**Hypothesis:** Bitemporal queries (event-time vs state-time) require retrieval that respects temporal ordering and validity windows, not just lexical/semantic similarity.

**Implementation:**
- Extend HybridSearch.search() signature with `temporalIntent` parameter: `event-time-recent` / `state-time-as-of-X` / `temporal-range` / `non-temporal`
- For temporal queries, prepend timestamp-aware re-ranker on top of RRF candidates (boost candidates whose `created_at` matches query temporal scope)
- Test: Stage 3 v6 LoCoMo has temporal questions per question_type field; isolate them and measure delta with and without temporal-aware ranking
- Knowledge graph integration: temporal queries with named entities should trigger BFS over entity's temporal validity windows (uses bitemporal KG layer that already exists in `mind/knowledge.ts`)

**Empirical validation:** N=30-50 isolated temporal subset of LoCoMo; measure trio-strict accuracy delta.

**Cost:** ~$10-15 ablation.

### 3.4 — Direction D: Learned reranker on top-K candidates

**Hypothesis:** RRF top-50 candidates pruned to top-10 via learned reranker (cohere-rerank-v3 or local cross-encoder) lifts precision substantially.

**Implementation:**
- Add post-RRF reranker layer; configurable provider (cohere API, local BGE-reranker-v2-m3, openai-style)
- Reranker takes (query, candidate_doc) pairs, returns relevance score; resort RRF candidates by reranker score
- Optional: only invoke reranker for top-N RRF candidates (cost optimization)
- Tier gating: reranker may be Pro/Teams feature if cost adds up (cohere $1/1k searches per current pricing)

**Empirical validation:** N=30-50 ablation with and without reranker, measure precision@10 + trio-strict.

**Cost:** ~$15-25 ablation (reranker API calls add up).

### 3.5 — Direction E: Entity-aware retrieval + KG bridge

**Hypothesis:** Question entity extraction + knowledge graph traversal + explicit entity bridge to retrieval candidates lifts recall on multi-entity questions.

**Implementation:**
- Pre-retrieval: extract entities from question (LLM call: "list named entities and their types in this question, JSON output")
- Knowledge graph lookup: for each extracted entity, find canonical entity ID via entity-normalizer + KG search
- Boost RRF candidates that mention or relate to extracted entities (additive boost in scoring.ts contextual layer; reuses existing graph distance code)
- Side benefit: populates `graphDistances` in `ScoringContext`, which currently is unused in practice (per audit §2.2 finding 1)

**Empirical validation:** N=20-30 multi-entity subset of LoCoMo (LoCoMo has named-entity-rich questions per dataset).

**Cost:** ~$10-15 ablation + LLM extraction cost (~$0.01 per question).

---

## §4 — Phasing (A → B → C, no Tier oznake)

### Phase A — Audit + baseline reproduction (1 week, ~$20-30)

1. Confirm Stage 3 v6 retrieval cell baseline reproducibility on N=20 subset (smoke check before any change)
2. Confirm which embedder was active during Stage 3 v6 run (audit `stage3-n400-v6-final-analysis.md`)
3. Per-question-type breakdown of V1 retrieval failures (which question types fail most: temporal? multi-entity? long-context?)
4. Document baseline cost + latency profile per direction
5. Memo: `decisions/2026-04-XX-retrieval-v2-phase-a-baseline.md`

### Phase B — Per-direction ablations (2-3 weeks, ~$100-200)

Run each direction (A through E) in isolation on N=20-30:
- Direction A: 5 candidate embedding models × N=20 = ~$50-100
- Direction B: 5-10 weight combinations × N=20 = ~$50-100
- Direction C: temporal-aware reranker × N=30-50 = ~$15
- Direction D: 2-3 reranker variants × N=30 = ~$15-25
- Direction E: KG-bridge × N=20-30 = ~$15

Per-direction memo: `decisions/2026-04-XX-retrieval-v2-direction-{A..E}-results.md`

Halt-and-ratify: each direction memo includes "ship/extend/skip" recommendation. PM ratifies subset that ships in Phase C.

### Phase C — Full N=400 V2 reproduction sa best combination (1 week, ~$30-50)

1. Combine ratified directions into single V2 build
2. Pre-registered manifest v7 (Phase A baselines + Phase B directional gains + Phase C combined V2)
3. Trio-strict + self-judge re-evaluation on N=400 (both methodologies per Mixed-methodology baseline rule)
4. Compare V2 vs V1 baselines vs oracle ceiling
5. Acceptance: V2 trio-strict ≥30% AND V2 trio-strict beats full-context (27.25%) on majority of question types

If acceptance fails: halt + diagnose which direction(s) didn't combine well; iterate.

If acceptance passes: V2 ships to production (waggle-os/packages/core/src/mind/) + sync to hive-mind via Step 3 CI/CD workflow.

---

## §5 — Sequencing constraints

V2 work cannot start until:

1. **Memory sync Step 3 (CI/CD sync workflow) is LIVE.** Otherwise V2 changes land in one repo and divergence grows. (~Step 2-3 take ~1-2 weeks per CC-2 brief estimates.)

2. **Agent fix sprint Phase 5 (re-pilot) is LIVE or substrate baselines are independently reproducible.** Otherwise V2 baseline measurements may be confounded by simultaneous agent harness changes. Phase 1 already PASSED gate; Phase 2-5 are paths critical to V2 baseline stability. (~Phase 2 unification ~1 week, Phase 3 long-task ~1 week, Phase 4-5 re-pilot validation ~2 weeks.)

3. **Tooling decision: who runs V2.** Three options:
   - (a) Same CC-1 sesion that did agent fix Phase 1-5 → context warm but big context window after months of work
   - (b) New CC-3 session with fresh context + this brief as starting point
   - (c) CC-2 (memory sync) takes V2 after Step 3 done — context warm on substrate code
   
   PM recommendation: (b) CC-3 fresh session. Reasons: (1) avoids CC-1 context exhaustion, (2) avoids CC-2 task overlap with sync repair, (3) clean PM ratification chain per direction memo.

**Realistic V2 timeline:** earliest start 2-3 weeks from now (post-CC-1 Phase 5 + post-CC-2 Step 3). Phase A through C: 4-5 weeks total. **V2 production-ready ETA: 6-9 weeks from today.**

**PRE-LAUNCH SEQUENCING (RATIFIED 2026-04-26):** Marko ratified V2 work as launch prerequisite, not post-launch follow-up. Quote: "nema launcha dok se sve ne sredi". This means:

- Launch ETA shifts to 6-9 weeks from today (V2 completion + remaining 14-step launch plan items)
- Substrate-ceiling-led launch comms (substrate 74% self-judge oracle + V1 retrieval honest disclosure framing) is REPLACED by V2 retrieval results in launch comms
- arxiv §5.3 will publish V2 results at launch, not "limited V1 + V2 follow-up"
- Landing copy v3 §3 Claim 3 will cite V2 numbers + production deployment justification, not V1 honest disclosure
- Decision Matrix amendment 2026-04-26 PHF (PASS-WITH-HONEST-FRAMING) is augmented: PHF still binding for substrate claim methodology framing, but retrieval framing strengthens from "V1 honest, V2 in progress" to "V2 production-ready"

PM update to landing copy + arxiv + Decision Matrix to reflect this sequencing change is queued as separate stream (independent of CC-1/CC-2/CC-3 code work).

---

## §6 — Cost estimate

| Phase | Sub-component | Cost | Time |
|-------|--------------|------|------|
| A | Baseline reproduction + audit + per-question-type breakdown | $20-30 | 1 week |
| B | Direction A (embedding model ablation) | $50-100 | 1 week |
| B | Direction B (scoring weight ablation) | $50-100 | 1 week (parallel with A) |
| B | Direction C (temporal-aware) | $15 | 0.5 week |
| B | Direction D (learned reranker) | $15-25 | 0.5 week |
| B | Direction E (KG bridge + entity-aware) | $15 | 0.5 week |
| C | Full N=400 V2 reproduction (trio-strict + self-judge) | $30-50 | 1 week |
| **Total** | | **$195-335** | **4-5 weeks** |

Compare to Stage 3 v6 cost ($29.75) — V2 is ~7-10× more expensive but produces evidence for paper §5.3 V2 results section + production deployment justification.

---

## §7 — Acceptance criteria (binding)

V2 ships to production iff:

1. **Trio-strict V2 ≥ 30%** on N=400 (close 70% of 11.25pp gap to oracle 33.5%)
2. **Trio-strict V2 > 27.25%** (must beat full-context baseline, otherwise retrieval has no production deployment justification)
3. **Self-judge V2 ≥ 65%** on N=400 (close 70% of gap to oracle 74%)
4. **No critical regression** on any directional ablation (each direction memo confirms ship/extend/skip)
5. **Memory sync verified** (Step 3 CI/CD workflow shipped V2 to both repos, parity check passes)
6. **arxiv paper §5.3 updated** with V2 results table replacing "five identified directions" enumeration with empirical results
7. **Cost stayed within envelope** ($335 total) — if exceeded, halt + PM ratification

---

## §8 — Marko ratifications (resolved 2026-04-26)

All five questions resolved:

1. **V2 sequencing — PRE launch (not post).** Marko: "Nema launcha dok se sve ne sredi". V2 work is launch prerequisite, not follow-up. Implication: realistic launch ETA shifts to 6-9 weeks from today (CC-1 Phase 5 + CC-2 Step 3 + CC-3 Phase A-C sequential). This is consistent with Marko's earlier "datum je sada nebitan, izgubili smo dosta vremena" stance and produces a stronger product at launch.

2. **CC-3 fresh session for V2.** Ratified.

3. **Direction priority — ALL FIVE directions execute, no cuts.** Marko: "sve". Cost envelope $195-335 binding; any direction-level scope reduction requires explicit Marko ratification. Direction A embedding model + B scoring weights + C temporal + D learned reranker + E entity-aware KG bridge all in scope.

4. **Tier-gating ratified.** Pro tier for reranker (cohere ~$1/1k searches), Voyage/OpenAI embedding remain Free tier.

5. **Mock-fallback telemetry — internal observability only.** Ratified. Per-deployment audit log; not landing-facing transparency feature.

---

## §9 — Cross-references

- arxiv paper outline: `research/2026-04-26-arxiv-paper/00-paper-outline.md`
- arxiv paper skeleton §5.3: `research/2026-04-26-arxiv-paper/01-paper-skeleton.md`
- Memory sync audit: `decisions/2026-04-26-memory-sync-audit.md`
- Pilot verdict: `decisions/2026-04-26-pilot-verdict-FAIL.md`
- Decision Matrix amendment (PHF): `decisions/2026-04-26-decision-matrix-self-judge-reframe.md`
- Stage 3 v6 5-cell summary: `D:\Projects\waggle-os\benchmarks\results\stage3-n400-v6-final-5cell-summary.md`
- Manifest v6: `D:\Projects\waggle-os\benchmarks\preregistration\manifest-v6-preregistration.yaml`
- 14-step launch plan: in PM session memory (Korak 2 = this brief)
- Mixed-methodology baseline rule: `feedback_config_inheritance_audit.md` Extension 2