This commit is contained in:
Oleg Maslov
201
docs/superpowers/specs/2026-07-02-sticky-erasure-design.md
Normal file
201
docs/superpowers/specs/2026-07-02-sticky-erasure-design.md
Normal file
@@ -0,0 +1,201 @@
|
||||
# Sticky Erasure — GDPR Art.17 suppression that survives re-import
|
||||
|
||||
**Date:** 2026-07-02 · **Feature:** #7 Art.17 tail · **Status:** design approved, implementation pending
|
||||
**Author:** brainstormed w/ founder (Marko), grounded by `sticky-erasure-recon` workflow (6 agents, full write-path map).
|
||||
|
||||
---
|
||||
|
||||
## Problem
|
||||
|
||||
The #7 Art.17 arc (closed 2026-07-02 S1) rotates `archive_uid` to an opaque random id on erase to
|
||||
close a re-identification vector. That rotation has a **documented side-effect** (`raw-archive.ts:194-200`):
|
||||
rotating the uid frees `RawArchive.append()`'s `content→uid` dedup key, so **re-importing an
|
||||
already-erased source re-materializes it**. The searchable summary, verbatim raw-turns, and KG
|
||||
already came back on re-harvest before the rotation too (pre-existing) — so erasure is not "sticky":
|
||||
a user who exercises their right-to-erasure on a ChatGPT thread gets it back the next time that
|
||||
thread is re-exported/re-synced.
|
||||
|
||||
The fix named in-code is an **erased-subject suppression list** consulted at every write seam. A
|
||||
content-keyed tombstone is explicitly **rejected** — storing `sha256(erased-PII)` would reintroduce
|
||||
the very content-derived re-id vector the rotation removed.
|
||||
|
||||
## Decisions (founder-ratified 2026-07-02)
|
||||
|
||||
| # | Decision | Choice |
|
||||
|---|----------|--------|
|
||||
| Granularity | key on | **Per-source `(source, source_ref)`** — the same tuple `eraseBySourceRef` uses; the only durable identity that survives erasure. NOT per-content-hash (re-id vector), NOT per-subject-string (no schema handle, over-erase). |
|
||||
| Durability | permanence | **Sticky + explicit re-consent** — permanent by default; a deliberate "Allow re-import again" UI action deletes the suppression row. |
|
||||
| Scope | reach | **Group-A write paths only**; connector + `ingest_source` documented as a known limitation (they persist no durable subject key today — already un-erasable by subject, orthogonal pre-existing gap). |
|
||||
| Backfill | past erasures | **Backfill** `erased_subjects` from retained `raw_archive` erased rows so historical erasures are sticky too. |
|
||||
| Tombstone | content-hash | **OFF** (consistent with the ratified anti-re-id rotation). |
|
||||
| Capture point | where recorded | **Inside `MindErasure`** — one point feeds both the route and the MCP `erase_memory` tool (the tool emits no audit event; a ledger fed from audit events would silently miss all MCP erasures). |
|
||||
| Mind scope | isolation | **Per-mind** (per the mind-isolation pin `feedback_mind_isolation_no_cross_mind_mixing`). |
|
||||
| Read-failure | on `isSuppressed` error | **Fail-closed on the item** (skip re-materialization). A failing local-SQLite read implies a broken DB where the follow-on INSERT fails anyway; Art.17 wins on the ambiguous item. |
|
||||
|
||||
## Architecture
|
||||
|
||||
Suppression must live **one level up from `FrameStore.createIFrame`** — that universal chokepoint
|
||||
sees only `(gopId, content)` (content-hash), which is exactly the rejected key. `(source, source_ref)`
|
||||
is only in scope at the harvest boundaries. So the check hooks at those boundaries; the capture hooks
|
||||
at the erase boundary.
|
||||
|
||||
### 1. Schema — `erased_subjects` (new table)
|
||||
|
||||
Home: `packages/hive-mind-core/src/mind/schema.ts` (`SCHEMA_SQL`, appended after the `raw_archive`
|
||||
block so it ships on first-init) + an idempotent guarded block in `db.ts` `runMigrations()` (same
|
||||
pattern as the `raw_archive` / `ai_interactions` migrations, no CHECK-list drift).
|
||||
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS erased_subjects (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
source TEXT NOT NULL,
|
||||
source_ref TEXT NOT NULL,
|
||||
erased_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ','now')),
|
||||
reason TEXT,
|
||||
UNIQUE(source, source_ref)
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_erased_subjects_lookup ON erased_subjects(source, source_ref);
|
||||
```
|
||||
|
||||
**No `content`, no `content_sha256`** — generic substrate only. This is the deliberate *opposite* of
|
||||
`install_audit` (proprietary-interleaved, hand-stripped on OSS export): `erased_subjects` forward-ports
|
||||
to `marolinik/hive-mind` verbatim. No immutability triggers (rows are deletable — that is the re-consent path).
|
||||
|
||||
**Backfill** (one-time, idempotent, in the same migration block, after table create):
|
||||
```sql
|
||||
INSERT OR IGNORE INTO erased_subjects (source, source_ref, erased_at, reason)
|
||||
SELECT source, source_ref, erased_at, erased_reason
|
||||
FROM raw_archive WHERE erased_at IS NOT NULL;
|
||||
```
|
||||
Limitation: past erasures of archive-less summaries (subject-mode on a legacy/append-failed frame with
|
||||
no `raw_archive` row) leave no skeleton to backfill from — accepted (they had no durable archive anchor).
|
||||
|
||||
### 2. `SuppressionStore` (new `mind/suppression.ts`)
|
||||
|
||||
Stateless wrapper over the mind db (like other `mind/` stores), exported from `@waggle/core` and
|
||||
`@waggle/hive-mind-core`:
|
||||
|
||||
```ts
|
||||
class SuppressionStore {
|
||||
constructor(db: Database) // or the project's DB wrapper, matching RawArchive's ctor
|
||||
isSuppressed(source, sourceRef): boolean // indexed SELECT EXISTS; FAIL-CLOSED (error → true)
|
||||
record(source, sourceRef, reason?): void // INSERT OR IGNORE (idempotent)
|
||||
unsuppress(source, sourceRef): boolean // DELETE; returns whether a row was removed
|
||||
list(): { source, sourceRef, erasedAt, reason }[]
|
||||
}
|
||||
```
|
||||
|
||||
`isSuppressed` swallows read errors and returns `true` (fail-closed on the item) with a loud
|
||||
`logger.error` — a genuine read failure means the DB is broken and the subsequent write fails too.
|
||||
|
||||
### 3. Capture — inside `MindErasure` (`mind/erasure.ts`)
|
||||
|
||||
- `eraseBySourceRef(source, sourceRef, reason)` → `suppression.record(source, sourceRef, reason)`
|
||||
(subject mode — the canonical key arrives verbatim).
|
||||
- `eraseFrameComplete(frameId)` → for **each resolved** `(source, source_ref)` subject (it already
|
||||
resolves them via `reconstructSource` + the content-prefix/`metadata.sourceId` fallback) →
|
||||
`suppression.record(...)`. Subject-less frames (connector/ingest single frames) resolve no durable
|
||||
subject → nothing recorded (out of scope, documented).
|
||||
|
||||
Because both the `/api/memory/erase` route (frame + subject mode) and the MCP `erase_memory` tool call
|
||||
these primitives, both surfaces feed the ledger with no drift. Recording happens **inside the erase
|
||||
transaction** so a rolled-back erase does not leave a stale suppression row.
|
||||
|
||||
### 4. Consumption guards (fail-closed)
|
||||
|
||||
| Seam | Guard | Covers |
|
||||
|------|-------|--------|
|
||||
| 3 harvest loops — `server/routes/harvest.ts`, `hive-mind-mcp-server/tools/harvest.ts`, `memory-mcp/tools/harvest.ts` | `if (isSuppressed(item.source, item.id)) { skippedSuppressed++; continue; }` at loop top, via **one shared helper** in hive-mind-core (kills twin-drift) | The whole fan-out per item: summary `createIFrame` + `setMetadata` + `RawArchive.append` + `writeRawTurnFrames` + KG/vector cognify + wiki |
|
||||
| `server/local/harvest-autosync-frame.ts` `writeAutoSyncSummaryFrame` | guard before `createIFrame` | 30-min in-process auto-sync **and** daily cron `harvest_sync` (single shared writer) |
|
||||
| `hive-mind-core/mind/raw-archive.ts` `RawArchive.append` | skip INSERT + return `{ created:false }` if suppressed | Defense-in-depth for any future non-loop caller of the documented-gap primitive |
|
||||
|
||||
The 3 loops call `RawArchive.append` and `writeRawTurnFrames` only from inside the loop, so the loop
|
||||
`continue` already covers those vectors; the `append` guard is a durable backstop, not the primary gate.
|
||||
The shared helper signature: `shouldSuppressHarvestItem(suppression, source, sourceRef): boolean` (or a
|
||||
thin method on `SuppressionStore`) so the 3 call sites are `continue`-only, logic shared.
|
||||
|
||||
`skippedSuppressed` is surfaced in each import's result summary so a suppressed re-import is visible
|
||||
(not a silent drop) — per the "no silent caps" discipline.
|
||||
|
||||
### 5. Re-consent surface
|
||||
|
||||
- `SuppressionStore.unsuppress` + `list` exposed via `@waggle/core`.
|
||||
- Server routes (`server/src/local/routes/memory-center.ts`, next to the erase endpoint):
|
||||
- `GET /api/memory/suppression` → `list()`
|
||||
- `POST /api/memory/suppression/allow` `{ source, sourceRef }` → `unsuppress()`
|
||||
- Memory Center UI (`apps/web/.../memory/MemoryCenterTab.tsx` or the trust/erase area): a "Suppressed
|
||||
sources" list, each row with an **"Allow re-import again"** button (confirm → POST → row disappears).
|
||||
Minimal; co-located with the existing Erase affordance.
|
||||
|
||||
### 6. Out of scope (documented limitation)
|
||||
|
||||
`ingest_source` (doc/URL/PDF/text) and the PRO connector cron persist **no durable `(source, source_ref)`**
|
||||
— `connectorDataToItems` discards the item id, `ingest_source` stamps no `metadata.sourceId` and creates
|
||||
**unlinked** KG entities. They are already un-reachable by subject-mode erasure today (pre-existing,
|
||||
independent of this feature). Making them sticky requires threading the connector item id + stamping
|
||||
`sourceId` + routing `ingest_source` through `importEntitiesForFrame` — a separate engineering item.
|
||||
Noted in `raw-archive.ts` / the erase docstrings.
|
||||
|
||||
### 6b. Known limitations (post-review, deferred — adversarial review 2026-07-02, 21 raised / 14 confirmed)
|
||||
|
||||
All confirmed HIGH + most MEDIUM/LOW findings were **fixed in-arc**. Three are deliberately deferred:
|
||||
|
||||
- **claude-code `decision-of` derived subject (MEDIUM, notable).** `extractDecisions` emits a SEPARATE
|
||||
item per parent whose id = `stableHarvestId('claude-code','decision-of',parentId)` and whose content
|
||||
is the parent's decision-pattern lines. Erasing/suppressing the PARENT subject does not reach this
|
||||
derived subject (distinct key), so the derived frame (quoting the parent's decision text) survives
|
||||
erasure AND re-materializes on re-import/auto-sync. Partly pre-existing (pre-arc it re-materialized
|
||||
under a random key too). Proper fix couples erasure to the derivation (follow `metadata.extractedFrom`
|
||||
in the sweep + record the derived key) — non-trivial, claude-code-specific; deferred to a follow-up.
|
||||
- **`isSuppressed` fail-closed labeling (LOW, cosmetic).** If ONLY `erased_subjects` is unreadable while
|
||||
other tables are healthy (single-table corruption / manual DROP within one process lifetime), every
|
||||
item is skipped and the harvest response mislabels the whole-corpus silent drop as "N erased subjects
|
||||
suppressed (GDPR Art.17)". Fail-closed is the correct Art.17 posture; only the count/message conflates
|
||||
a read-error with a real suppression. Uncommon trigger; deferred (surface a separate error counter).
|
||||
- **Workspace-scope suppression is inert (LOW, cosmetic).** A `mind=workspace` erase records a suppression
|
||||
row in the workspace mind, but harvest only ever writes/reads the personal mind, so the row can never
|
||||
suppress anything and the workspace re-consent panel's promise is vacuously true. Consistent with the
|
||||
ratified per-mind model; the UI/API framing over both minds is the only misleading part. Deferred.
|
||||
|
||||
### 7. OSS forward-port (P2, monorepo-first per §7.5)
|
||||
|
||||
`suppression.ts` + the `schema.ts`/`db.ts` additions are **generic substrate** (no governance/trust
|
||||
fields) → forward-port to `marolinik/hive-mind` `packages/core` verbatim via the curated forward-port
|
||||
(co-located tests, import rewrites). Done after the monorepo commits land + any blocking OSS PR, matching
|
||||
the prior arc's cadence. The `schema.ts`/`db.ts` diffs still get the manual proprietary-review pass.
|
||||
|
||||
## Testing (TDD)
|
||||
|
||||
Unit (`hive-mind-core/tests/mind/suppression.test.ts`):
|
||||
- `record` idempotent; `isSuppressed` true/false; `unsuppress` removes + returns bool; `list` shape.
|
||||
- `isSuppressed` **fail-closed**: a forced read error returns `true`.
|
||||
|
||||
Migration/backfill (`hive-mind-core/tests/mind/schema` or db migration test):
|
||||
- Table created on fresh init; migration idempotent on re-run.
|
||||
- Backfill populates `erased_subjects` from pre-existing `raw_archive` erased rows.
|
||||
|
||||
Capture (`hive-mind-core/tests/mind/erasure.test.ts`):
|
||||
- `eraseBySourceRef` records the pair; `eraseFrameComplete` records each resolved subject; rolled-back
|
||||
erase leaves **no** suppression row.
|
||||
|
||||
Integration — **the headline test**:
|
||||
- Harvest a source → erase it (subject or frame mode) → **re-import the same source** → assert the
|
||||
summary, raw-turns, `raw_archive` row, and KG entities do **NOT** re-materialize, and `skippedSuppressed>0`.
|
||||
- `unsuppress` → re-import the same source → assert it **does** re-materialize.
|
||||
- `RawArchive.append` on a suppressed subject → `{ created:false }`, no row.
|
||||
- `writeAutoSyncSummaryFrame` on a suppressed subject → no frame.
|
||||
|
||||
Gates: `npx tsc --noEmit` on {shared, hive-mind-core, server, memory-mcp, apps/web}; touched-area
|
||||
Vitest green; then a multi-lens adversarial review workflow (matching the arc's cadence) with every
|
||||
confirmed finding fixed/documented in-arc.
|
||||
|
||||
## Phasing
|
||||
|
||||
1. **Substrate** — schema + migration + backfill + `SuppressionStore` + capture in `MindErasure`. (hive-mind-core)
|
||||
2. **Guards** — shared helper + 3 harvest loops + `writeAutoSyncSummaryFrame` + `RawArchive.append` backstop.
|
||||
3. **Re-consent** — routes + Memory Center UI.
|
||||
4. **Adversarial review** — multi-lens workflow; fix confirmed findings.
|
||||
5. **OSS forward-port** — P2, after monorepo lands (per §7.5).
|
||||
|
||||
Commit per phase; push held until founder asks (main is shared with concurrent sessions — fetch-checked
|
||||
fast-forward at push time, as in the prior arc).
|
||||
Reference in New Issue
Block a user