moving

docs/plans/W4-PRODUCTION-PORT-PLAN-2026-06-11.md

@@ -0,0 +1,161 @@
+# W4 — Production Parity Port Plan (2026-06-11)
+
+**Goal:** port the benchmark-proven LoCoMo retrieval stack (W3.3 FINAL: **87.66 overall,
++5.71pp vs Memori, z=4.42**; see `benchmarks/results/memori-phase22-RESULT.md`) into the
+Waggle OS production recall path. Pre-approved by Marko ("if results within projected we
+implement on production" — they are, above projection).
+
+**Evidence base per component** = the wave-gated z-tests (W1 answer policy → W2a profiles
+→ W3.1 date-window → W3.3 raw-detail + caption parity). Anti-goals from the arc apply
+verbatim (proposal §4): never strip write-time dating; no relevance-only episodic; no
+agentic multi-turn retrieval in the hot path; keep conditional abstention in production
+(never-refuse was benchmark-cell policy ONLY); no Neo4j/cloud.
+
+**Recon provenance:** 4-agent workflow `w4-port-recon` (2026-06-11) over
+`packages/agent`, `packages/hive-mind-core`, `packages/server`, benchmark harness
+`D:/Projects/hive-mind-test/scripts/locomo/`. Full gap matrix below.
+
+---
+
+## 0. Headline findings
+
+1. **Production auto-recall uses 1 of the benchmark's 7 lanes.**
+   `chat.ts:761 → orchestrator.recallMemory` (default opts) → HybridSearch FTS5+vec
+   RRF — that's it. No reranker, no distilled/episodic/profile lanes, no date windows,
+   no raw escalation. The benchmark's win is the *orchestration* (7 lanes, id-dedup,
+   fixed render order), not any single lane.
+2. **The OSS repo is AHEAD of the monorepo** — `inprocess-reranker.ts`
+   (Xenova/ms-marco-MiniLM-L-6-v2, transformers.js ONNX, ~22MB) + reranker options in
+   HybridSearch exist ONLY in `D:/Projects/hive-mind/packages/core`. This contradicts
+   the CLAUDE.md §7.5 "byte-identical subtree-split" assumption. **W4.2 reverse-ports
+   it; a §7.5 sync-policy decision is flagged for Marko.**
+3. **Ollama is NOT a hard dependency for anything.** LLM extraction passes route via
+   the existing `LLMCallFn` 'fast' tier → LiteLLM (Ollama = optional sovereign-local
+   routing target). Reranker is in-process CPU ONNX. Components #1/#2/#3/#7/#9/#10
+   are pure code.
+4. **Three production scoring/filter bugs must not be built on top of** (§3 below).
+
+---
+
+## 1. Gap matrix (impact × ease order)
+
+| # | Component | Verdict | Where it lands |
+|---|---|---|---|
+| 1 | Date rendering + TEMPORAL_GUIDANCE | **PARTIAL** — helpers shipped in `recall-context.ts`, ZERO consumers; guidance text is stale pre-W1 wording | orchestrator.ts:486-498/:529-537, tools.ts:217/:232, prompt-assembler.ts:280-282 |
+| 2 | Importance K=5 lane | **PARTIAL** — identical SQL exists (orchestrator.ts:422-432) but gated behind 13 catch-up regexes | make unconditional in recallMemory; dedup at :440-447 reused |
+| 3 | Date-window parser + Events-during-X | **MISSING** parser; substrate since/until exists w/ 2 defects | new pure module beside resolve-relative-date.ts; wire at orchestrator.ts:453-458 |
+| 4 | Cross-encoder reranker | **MISSING** in waggle-os (0 grep hits) — reverse-port from OSS `inprocess-reranker.ts` | optional peer dep @huggingface/transformers; caller-level seam covers merged personal+workspace pool |
+| 5 | Distilled-facts lane | **PARTIAL** — 4-pass HarvestPipeline (pipeline.ts:83-338) is DEAD CODE (zero call sites); no wholesale fetch lane | distillation cron (cron-store.ts/setup-crons.ts) + prefix-fetch lane + "Memory Facts" section |
+| 6 | Episodic events block | **MISSING** — only resolveRelativeDate shipped, wired into MCP harvest ONLY; sidecar + memory-mcp paths unwired (twin drift: memory-mcp passes NO timestamp) | port script-33 pass into pipeline; unify 3 ingest surfaces; chronological render; CE top-K needs #4 |
+| 7 | Caption-aware harvest | **MISSING** — all 4 adapters drop image content (exact production counterpart of the W3.3 4.5pp single-hop fix); exports ALREADY carry extracted text (Claude `extracted_content`, ChatGPT caption parts, Gemini inlineData) | chatgpt/claude/gemini/universal adapters; no vision model needed |
+| 8 | Profile cards | **MISSING** — IdentityLayer is single-user; wiki person pages never injected | script-35 pass keyed off KnowledgeGraph person entities; render-FIRST + seen-set exclusion |
+| 9 | Token packing | **PARTIAL** — PromptAssembler exists, flag OFF, **3 bugs block enabling**: double-inject (chat.ts:879), double-compute (orchestrator.ts:340-343), dateless renderFrames; fleet spawns get zero recall | fix bugs → route lanes through budget → flip flag |
+| 10 | RAWDETAIL escalation lane | **MISSING — hardest, do LAST.** Blockers: (a) per-turn verbatim dialogue is NOT stored (harvest collapses to 1 summary frame, 2000/10000-char truncation divergence), (b) needs #4, (c) no conversational-adjacency key (base_frame_id is I/P delta chains) | per-turn storage decision + turn-index metadata + lane port |
+
+## 2. Render order (port as a pure `buildContext`-style renderer in hive-mind-core)
+
+profiles → distilled facts → episodic (chronological) → Events-during-window →
+importance+semantic snippets (+ reference-date anchor) → raw excerpts.
+Natural location: beside `recall-context.ts` (its stated purpose), consumed by
+`orchestrator.recallMemory` — benchmark/production format parity by construction.
+
+## 3. Production bugs to fix in-line (NOT build on top of)
+
+1. **graphDistances never passed** → contextual score always 0 → 20% of 'balanced' /
+   60% of 'connected' weight permanently dead. Worse: `bfsDistances` returns ENTITY-id
+   keys where scoring looks up FRAME ids — wiring it naively silently fails.
+   Decision: build the frame-anchored bridge (docs/memory-architecture.md:296 describes
+   the intent) **or zero out the dead weight** in SCORING_PROFILES. Default: zero-out in
+   W4.2, bridge as follow-up (benchmark won without graph signal).
+2. **since/until zero deterministic callers**; SQL filter applied POST-fusion (filtered
+   frames consume lane slots → results shrink below limit) + until-fencepost
+   (string-compare excludes same-day frames). Fixed by #3.
+3. **'temporal' scoring decays on last_accessed** (touch() bumps it → constant noise on
+   historical corpora). Switch to created_at/event-date decay in W4.2 (resolved
+   created_at from #6 makes it meaningful).
+
+## 4. Non-negotiable constraints (every new lane)
+
+- `scanForInjection` over ALL recalled text (orchestrator.ts:517-527 blocks all recall
+  on hit; chat.ts:763-771 re-scans).
+- Anti-confabulation provenance preamble stays (orchestrator.ts:529-537).
+- temporary/deprecated post-filter respected (orchestrator.ts:466-471).
+- Conditional abstention preserved — do NOT port the benchmark's never-refuse prompt.
+- Profile frames importance='normal' (out of K5 lane), excluded from snippet lane.
+- Wholesale chronological episodic block is load-bearing — no top-K-only "optimization"
+  without the CE floor (P5 lesson).
+
+## 5. Phases (≤5 files each, commit + verify per phase)
+
+| Phase | Scope | Components | Verify |
+|---|---|---|---|
+| **W4.1** | Query-time quick wins (pure code) | #1 temporal render+W1 guidance text, #2 unconditional importance lane, #3 date-window parser + since/until substrate fixes | new unit tests; tsc agent+hive-mind-core; existing suites green |
+| **W4.2** | Reranker reverse-port + scoring bug fixes | #4 inprocess-reranker + HybridSearch options (from OSS); bugs 1+3 | reranker unit tests (OSS has them); scoring tests updated |
+| **W4.3** | Extraction passes + lanes | #5 distillation cron + facts lane, #6 episodic pass + ingest unification (3 surfaces), #8 profile cards; new renderer module | pipeline tests w/ mocked LLMCallFn; ingest-path tests incl. memory-mcp timestamp fix |
+| **W4.4** | Harvest input parity | #7 caption-aware adapters ×4; truncation reconciliation (2000 vs 10k) | adapter fixture tests w/ real export shapes |
+| **W4.5** | Budget + assembly | #9 PromptAssembler 3 bug fixes, route lanes through budget, flag flip; ≤1.5k token packing target | assembler tests; double-inject regression test; live smoke |
+| **W4.6** | RAWDETAIL (last) | #10 per-turn storage decision + turn-index key + lane | needs Marko sign-off on storage growth tradeoff first |
+
+Re-validation after W4.3 and W4.5: LongMemEval N=100 spot-check (knowledge-update +
+single-session categories) per proposal §5 — guards the production-policy variants
+(conditional abstention) against regression.
+
+## 6. Open decisions for Marko
+
+1. **§7.5 sync policy** — OSS repo evolved ahead (reranker). One-off reverse-port (W4.2
+   does this regardless) vs re-establishing the subtree-split invariant afterward.
+2. **W4.6 storage tradeoff** — per-turn raw dialogue storage grows the .mind footprint
+   substantially (LoCoMo: ~600 turns/conv). Gate W4.6 on explicit GO.
+3. **graphDistances** — zero-out (default) vs build the frame-anchored bridge now.
+4. **Distillation cron cadence + model tier** — 'fast' tier via LiteLLM default;
+   Ollama-only mode reserved for the sovereign story.
+
+## 7. Status log
+
+- 2026-06-11 (SHIPPED → origin/main): **W4.1a** `9487f0d` (temporal render +
+  W1 guidance + unconditional importance lane), **W4.1b** `eb8996f`
+  (date-window parser + since/until fencepost + slot-consumption fixes),
+  **W4.2** `f47ee8f` (reranker reverse-port, flag `WAGGLE_RERANKER=1` opt-in;
+  bug #3 created_at decay FIXED; bug #1 documented-not-zeroed — constant-0 is
+  ranking-neutral, zeroing would break graphDistances capability), **W4.3a-d**
+  `8cd841c`/`71f8abe`/`a6c1107`/`8289e53` (extract-memory-lanes passes +
+  [mind-*] frame conventions + recallMemory lane rendering + ingest
+  unification incl. memory-mcp no-timestamp bug + daily extraction cron).
+  All via worktree D:/Projects/waggle-os-w4 (branch feature/w4-port).
+  Remaining: W4.4 caption adapters, W4.5 PromptAssembler fixes + flag flips
+  + live smoke, W4.6 rawdetail (gated on storage decision).
+- 2026-06-11 (later, SHIPPED → origin/main): **W4.4** `1c337d7` (4 caption-aware
+  adapters + HARVEST_FRAME_CONTENT_CAP=10k unification across 3 surfaces),
+  **W4.5** `5a5fc0a` + `a6ef018` (double-inject + double-compute FIXED;
+  recallMemory's multi-lane block routes verbatim through the assembler budget;
+  LIVE SMOKE all-pass — real server + real ONNX reranker, 58-83ms warm recalls,
+  recall block exactly once; **WAGGLE_RERANKER now DEFAULT ON**, kill switch =0,
+  tests pinned off). **WAGGLE_PROMPT_ASSEMBLER stays opt-in** — flip pending
+  founder ratification (smoke validated the recall path, not assembler-wide
+  prompt reshaping in live LLM chats). Remaining: **W4.6 rawdetail only**
+  (gated on the per-turn raw-storage decision, §6.2).
+- 2026-06-11 (W4.6 SHIPPED — **port COMPLETE, 7/7 lanes**): Marko GO on the
+  storage tradeoff (§6.2, full — no retention cap). **W4.6a** raw-turn storage
+  + RAWDETAIL lane core (`harvest/raw-turns.ts` per-turn
+  `[mind-rawturn conv:<key> turn:<n> speaker:<s>]` frames, write-time injection
+  scan per turn; `mind/raw-detail-lane.ts` window/FTS pool → CE top-6 → ±1
+  dialogue neighbors), **W4.6b** recallMemory wiring (rendered LAST as
+  '## Raw dialogue excerpts (verbatim)', CE-gated, raw turns excluded from the
+  snippet lanes, kill switch WAGGLE_RAWDETAIL=0; speaker labels PARENTHESIZED —
+  colon-suffixed role labels collide with the injection scanner's
+  chat-template-smuggling patterns), **W4.6c** writes on all 3 harvest surfaces
+  + sidecar cognify-selection fix (explicit summary-frame ids replace the
+  getRecent recency window the interleaved raw turns would have polluted).
+  Suites: hive-mind-core+agent 3353/3353, server-local 868/868; tsc
+  agent/server/hive-mind-core/memory-mcp 0. Same-session decisions ratified:
+  assembler = smoke-then-flip; §7.5 = monorepo sole source + drift check;
+  graphDistances = leave documented (§6.3 closed as leave).
+
+- 2026-06-11: Plan written from w4-port-recon workflow output.
+- 2026-06-11 (W3.4 ablation DONE): **attribution resolved — captions alone +0.26 ns;
+  raw-detail lane on top +2.40 (z=1.95).** The lane is the delivery mechanism, captions
+  the payload. W4 consequence: **#7 and #10 are a coupled pair** — caption-aware
+  adapters deliver little recall value through existing lanes; schedule #7 WITH (or
+  immediately before) #10, or route caption text through the #5/#6 extraction passes
+  so distilled/episodic facts carry it. The W4.4 phase stays (input parity is still
+  correct), but its measured-win expectation moves to W4.6.