# Verbatim Provenance Archive (#7) — Design Spec

**Date:** 2026-06-30 · **Author:** Claude Opus 4.8 (1M) · **Audience:** Marko (founder) · **Status:** design — awaiting spec review
**Arc:** paperclip external-agent recon backlog → STEAL SOON / memory moat → #7
**Source backlog:** `docs/analysis/external-agent-launching-and-memory-comparison-2026-06-29.md` §8 #7

---

## 1. The finding that reshaped #7

The §8 backlog described #7 as "append-only verbatim archive, new file `harvest/raw-turns.ts`." **That file already exists** (W4.6, "Marko GO 2026-06-11"). On inspection, verbatim text is *already stored twice* in the substrate:

| System | What it stores | Immutable? | Coverage | Purpose |
|---|---|---|---|---|
| `harvest/raw-turns.ts` | per-turn dialogue text, as `[mind-rawturn …]` frames | **No** — deletable rows in `memory_frames` (cleanup/dedup/reconcile) | harvest imports **with messages** only; gated by `WAGGLE_RAWDETAIL` | retrieval (the RAWDETAIL recall lane) |
| `ai_interactions` (schema Layer 7) | model I/O (`input_text`/`output_text`) | **Yes** — DDL `BEFORE UPDATE/DELETE` triggers | live agent interactions | EU-AI-Act Art.12 event log |
| harvest route summary frame | `item.content.slice(0, 10_000)` (truncated preview) + `metadata.sourceId` | **No** — deletable | every harvest item | the searchable memory |

So "store verbatim" is **not** the gap. The irreducible gap is a **provenance anchor**: the ability to take any distilled/imported memory frame and reconstruct the **exact, full, never-mutable source it came from**. Today:
- The harvest summary frame is a **10K-char truncation** of `item.content` — long documents/conversations lose their tail.
- raw-turns are deletable and messages-only.
- No store guarantees the *full* source survives frame cleanup, and nothing carries an immutable integrity hash.

This is precisely the EU-AI-Act audit / "reconstruct the original" value #7 was picked for.

## 2. Goal & non-goals

**Goal:** an append-only, immutable, full-fidelity store of each harvested source item, with a provenance link from the frames it produced, and an audit/reconstruction query.

**Non-goals (YAGNI — explicitly out of scope for v0):**
- Capturing non-harvest ingest paths (`save_memory`, `agent_inferred`, `team_sync`, connector auto-fetch). Founder-chosen scope = **harvest/import only**. Most other paths are already verbatim (not lossy).
- Touching the retrieval path. **No change to `search.ts` / `scoring.ts` / the ranked corpus.** The 87.66% LoCoMo SOTA is regression-locked by construction (the archive is not part of the recall corpus). Anti-rec #1 honored.
- Replacing or modifying raw-turns or `ai_interactions`. This is **additive**.
- Retention/GC of the archive. It is append-only and grows; retention policy is a documented follow-up (mirrors the `ai_interactions` posture — storage growth accepted, pseudonymize-tombstone flow deferred).
- A UI surface. v0 is substrate + wiring + a query API; a Memory-Center "view original" button is a follow-up.

## 3. Design

### 3.1 New table `raw_archive` (hive-mind-core substrate)

```sql
CREATE TABLE IF NOT EXISTS raw_archive (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  archive_uid TEXT NOT NULL UNIQUE,          -- = content_sha256 (stable, idempotent natural key)
  source TEXT NOT NULL,                       -- import source (chatgpt/claude/gemini/url/pdf/…)
  source_ref TEXT,                            -- item.id (the UniversalImportItem id)
  title TEXT,                                 -- item.title (audit readability)
  content TEXT NOT NULL,                       -- FULL verbatim item.content (untruncated)
  content_sha256 TEXT NOT NULL,                -- integrity anchor
  injection_flagged INTEGER NOT NULL DEFAULT 0,-- 1 if scanForInjection flagged the content
  injection_flags TEXT NOT NULL DEFAULT '',    -- comma-joined flags when flagged
  source_timestamp TEXT,                       -- item.timestamp (original event time), if ISO
  created_at TEXT NOT NULL DEFAULT (datetime('now'))
);
CREATE INDEX IF NOT EXISTS idx_raw_archive_source_ref ON raw_archive (source, source_ref);
CREATE INDEX IF NOT EXISTS idx_raw_archive_created ON raw_archive (created_at DESC);

-- Append-only enforcement — identical posture to ai_interactions (schema Layer 7).
CREATE TRIGGER IF NOT EXISTS raw_archive_no_update
BEFORE UPDATE ON raw_archive
BEGIN SELECT RAISE(ABORT, 'raw_archive is append-only (verbatim provenance archive)'); END;
CREATE TRIGGER IF NOT EXISTS raw_archive_no_delete
BEFORE DELETE ON raw_archive
BEGIN SELECT RAISE(ABORT, 'raw_archive is append-only (verbatim provenance archive)'); END;
```

Added to **both** `SCHEMA_SQL` (fresh DBs) and an idempotent block in `db.ts runMigrations()` (existing DBs), matching the established pattern. Triggers are `CREATE … IF NOT EXISTS` so both paths are safe.

### 3.2 New module `mind/raw-archive.ts` — `RawArchive` store

```ts
export interface RawArchiveRow {
  id: number; archive_uid: string; source: string; source_ref: string | null;
  title: string | null; content: string; content_sha256: string;
  injection_flagged: 0 | 1; injection_flags: string;
  source_timestamp: string | null; created_at: string;
}
export interface ArchiveInput {
  source: string; sourceRef?: string; title?: string; content: string; sourceTimestamp?: string;
}
export class RawArchive {
  constructor(db: MindDB);
  /** Idempotent: archive_uid = sha256(content). INSERT OR IGNORE; returns the uid either way.
   *  Injection-scans the content and records the flag, but stores verbatim regardless
   *  (zero-loss forensic semantics — the archive is never fed to an LLM directly). */
  append(input: ArchiveInput): { archiveUid: string; created: boolean };
  getByUid(archiveUid: string): RawArchiveRow | undefined;
  /** Resolves frame.metadata.archiveUid → row. Returns undefined when the frame has no link. */
  reconstructSource(frameId: number): RawArchiveRow | undefined;
  list(opts?: { limit?: number; offset?: number; source?: string }): RawArchiveRow[];
  count(): number;
}
```

- **Idempotency:** `archive_uid = content_sha256`. Re-importing an unchanged item re-derives the same uid; `INSERT OR IGNORE` makes the second append a no-op (`created:false`). Matches the existing harvest idempotency story (source-level content hash + `createIFrame` dedup).
- **Injection posture:** scan-but-don't-drop. A zero-loss audit record must keep exactly what arrived — including a hostile payload (that's *evidence*). The row is never fed to an LLM; it's read only by `reconstructSource`/`list` for human/audit eyes. `injection_flagged` + `injection_flags` are recorded so any future consumer that *does* surface the content to a model re-scans first. (Contrast raw-turns, which drops, because those frames ARE fed to recall.)
- **Hashing:** `createHash('sha256')` from `node:crypto` over the **full, untouched** `content` (zero new dependency — `content-hash.ts` already uses `node:crypto` the same way). **Do NOT reuse `hashFrameContent`** — it `stripHmPrefix`'s + trims the body (provenance-insensitive dedup semantics), which would hash a *mangled* body and break the "integrity hash of the exact verbatim" guarantee. Add a small `hashRaw(content)` helper (or inline the 1-liner) in `raw-archive.ts`.

### 3.3 Link mechanism — frame metadata (zero migration on the hot table)

The harvest route already stamps the summary frame:
```ts
frameStore.setMetadata(frame.id, JSON.stringify({ kind, confidence, status: 'unreviewed', sourceId: item.id }));
```
We add `archiveUid` to that **same** object — no new `setMetadata` call, no schema change to `memory_frames` (the `metadata` JSON column already exists). The `archiveUid` sits alongside the existing `sourceId`. `reconstructSource(frameId)` reads `JSON.parse(frame.metadata).archiveUid`.

Rationale for metadata-JSON over a bridge table: the link is 1:1 (one summary frame per item on this route), read-on-demand (audit query, not hot-path), and the column already exists — a bridge table would add a migration + join for no query benefit at this scope.

### 3.4 Wiring — server harvest route (`packages/server/src/local/routes/harvest.ts`)

In the existing per-item loop (the `for (const item of items)` block, ~L424–485), once per item:
1. `const { archiveUid } = rawArchive.append({ source: item.source, sourceRef: item.id, title: item.title, content: item.content, sourceTimestamp: providedTimestamp });` — **full** `item.content`, before truncation.
2. Include `archiveUid` in the metadata object already built at the `setMetadata` call (~L468).

This is the primary desktop ingest path and the only wiring point for v0. The two MCP harvest tools (`memory-mcp`, `hive-mind-mcp-server`) are a **documented follow-up** — same `RawArchive.append` call in their item loops; deferred to keep v0 a single reviewable surface.

## 4. Data flow

```
harvest import (UniversalImportItem)
   │
   ├─ rawArchive.append({full item.content})  ──►  raw_archive row (immutable, sha256, injection-flagged)
   │        returns archiveUid                          ▲
   │                                                    │ metadata.archiveUid
   ├─ createIFrame(content.slice(0,10K))  ──►  memory_frames summary frame ─┘
   │        + setMetadata({…, sourceId, archiveUid})
   │
   └─ writeRawTurnFrames(item)  ──►  [mind-rawturn…] frames (unchanged; retrieval lane)

audit / reconstruction:
   reconstructSource(frameId) → frame.metadata.archiveUid → raw_archive row (full verbatim + integrity hash)
```

## 5. Error handling

- `append()` is best-effort-safe: an injection-flagged item still stores (flag recorded). A DB error in `append` must **not** abort the harvest item — wrap the call so a failed archive logs a warning and the frame still persists *without* an `archiveUid` (degraded provenance beats a failed import). Never silent: log names source + item id.
- `reconstructSource`: returns `undefined` (not throw) for frames with no/invalid `archiveUid` or missing rows.
- Append-only triggers: any code path that attempts UPDATE/DELETE on `raw_archive` throws at the DB layer — this is intended; callers must never mutate.

## 6. Testing strategy (TDD)

Unit (hive-mind-core, co-located `tests/mind/raw-archive.test.ts`):
1. `append` inserts a row; returns `created:true` + a stable uid = sha256(content).
2. `append` is idempotent — second identical content → `created:false`, same uid, one row.
3. append-only triggers — direct `UPDATE`/`DELETE` on `raw_archive` throws.
4. injection content is **stored** (zero-loss) with `injection_flagged=1` + flags populated.
5. full content survives — a >10K-char content stores untruncated (vs the frame's 10K cap).
6. `reconstructSource(frameId)` round-trips: append → create frame with `metadata.archiveUid` → reconstruct returns the row; returns `undefined` for an unlinked frame.
7. `list` / `count` paging + source filter.
8. migration: a pre-existing DB (no `raw_archive`) gains the table + triggers idempotently on boot.

Integration (server, `tests/local/harvest-*.test.ts` sibling): one harvest item produces (a) a raw_archive row, (b) a summary frame whose `metadata.archiveUid` resolves to that row, (c) re-import is idempotent (no duplicate archive row).

## 7. Risk & SOTA safety

- **No retrieval-path change.** `raw_archive` is not in any search/scoring query; frames are unchanged in shape. The 87.66% LoCoMo number cannot move. No LoCoMo re-run required.
- **Hot-table safety.** Zero schema change to `memory_frames`; the link uses the existing `metadata` column. The `idx_frames_content_hash` boot-order regression (2026-06-12) does not apply — `raw_archive` is a standalone table with no dependency on a guarded ADD COLUMN.
- **Storage growth** is the accepted tradeoff (same posture the founder already ratified for raw-turns, 2026-06-11). Archive stores full content once per unique item.

## 8. OSS sync note (§7.5)

`raw_archive` is **generic provenance substrate** (like `ai_interactions`, which is EU-AI-Act-framed yet **not** in the OSS-excluded list) — *not* Waggle-proprietary governance like `install_audit`. So it is **OSS-bound**: it should forward-port to `marolinik/hive-mind` in the next curated regeneration. Built in the monorepo first per §7.5; the OSS mirror is regenerated separately by the maintainer. No mirror edit in this arc. Flag for the next `oss-drift-check.sh` pass.

## 9. Out of scope / follow-ups (tracked, not built)

- MCP harvest entry points (`memory-mcp`, `hive-mind-mcp-server`) — same one-line `append` wiring.
- 4-pass `HarvestPipeline` distilled frames (where item attribution is lost in synthesis) — those persist via a different path; linking them needs pipeline itemId preservation. Not on the server route (which doesn't run the pipeline for its frames).
- Memory-Center "view original source" UI button over `reconstructSource`.
- Retention / GDPR-erasure tombstone flow for the archive.