This commit is contained in:
Oleg Maslov
430
docs/backend-map/sections/03b-api-memory.md
Normal file
430
docs/backend-map/sections/03b-api-memory.md
Normal file
@@ -0,0 +1,430 @@
|
||||
# 03b — Memory / Knowledge / Wiki / Harvest / Import / Identity API
|
||||
|
||||
## Purpose
|
||||
|
||||
This section is the frontend rebuild contract for Waggle OS's **memory subsystem** HTTP API: recalling and saving memory frames, browsing the knowledge graph, compiling and reading the personal wiki, harvesting external AI exports (ChatGPT / Claude / Gemini / Claude Code) into memory, importing conversation exports, reading/writing the agent identity record, tracking document versions, and the GDPR "right to erasure" flow. Every endpoint below is grounded in actual route files under `packages/server/src/local/routes/`. All routes are served by the **local sidecar** (Fastify), registered in `packages/server/src/local/index.ts`.
|
||||
|
||||
> **Scope note.** The recall/save/search/CRUD endpoints under `/api/memory/frames` and `/api/memory/search` live in `routes/memory.ts` (a sibling of the assigned files). They are the "recall/save memory" surface the rebuild needs, so they are documented here. The knowledge-graph read (`/api/memory/graph`) is in `routes/knowledge.ts`. Where a route lives in a specific file, the file is named in the per-section heading.
|
||||
|
||||
---
|
||||
|
||||
## Mental model
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
UI[Frontend / Lovable rebuild]
|
||||
|
||||
subgraph Memory["Memory frames (routes/memory.ts)"]
|
||||
SEARCH["/api/memory/search"]
|
||||
FRAMES["/api/memory/frames (GET/POST/PUT/DELETE/PATCH)"]
|
||||
STATS["/api/memory/stats"]
|
||||
end
|
||||
|
||||
subgraph KG["Knowledge graph (routes/knowledge.ts)"]
|
||||
GRAPH["/api/memory/graph"]
|
||||
end
|
||||
|
||||
subgraph Wiki["Wiki compiler (routes/wiki.ts)"]
|
||||
WPAGES["/api/wiki/pages*"]
|
||||
WCOMPILE["/api/wiki/compile"]
|
||||
WEXPORT["/api/wiki/export/{obsidian,notion}"]
|
||||
end
|
||||
|
||||
subgraph Harvest["Harvest (routes/harvest.ts)"]
|
||||
HPREV["/api/harvest/preview"]
|
||||
HCOMMIT["/api/harvest/commit"]
|
||||
HSRC["/api/harvest/sources"]
|
||||
HRUNS["/api/harvest/runs*"]
|
||||
HPROG["/api/harvest/progress (SSE)"]
|
||||
end
|
||||
|
||||
subgraph Import["Legacy import (routes/import.ts)"]
|
||||
IPREV["/api/import/preview"]
|
||||
ICOMMIT["/api/import/commit"]
|
||||
end
|
||||
|
||||
subgraph Identity["Identity (routes/identity.ts + mind.ts)"]
|
||||
IDGET["/api/identity (GET/POST)"]
|
||||
MIND["/api/mind/{identity,awareness,skills}"]
|
||||
end
|
||||
|
||||
DOCS["/api/workspaces/:id/documents (routes/documents.ts)"]
|
||||
ERASE["/api/data/erase (routes/data-erase.ts)"]
|
||||
|
||||
UI --> Memory & KG & Wiki & Harvest & Import & Identity & DOCS & ERASE
|
||||
|
||||
PERSONAL[(personal.mind\nSQLite)]
|
||||
WS[(workspace minds\nSQLite)]
|
||||
Memory --> PERSONAL & WS
|
||||
KG --> PERSONAL & WS
|
||||
Wiki --> PERSONAL
|
||||
Harvest --> PERSONAL
|
||||
Import --> PERSONAL
|
||||
Identity --> PERSONAL & WS
|
||||
```
|
||||
|
||||
**Two memory scopes everywhere.** Almost every read/write accepts an optional workspace selector. The canonical query/body param is **`workspace`**, but **`workspaceId` is also accepted** as an alias (do not rely on only one). Omitting it (or passing `personal`) targets the **personal mind**. Workspace minds are separate SQLite DBs resolved via `server.agentState.getWorkspaceMindDb(id)`.
|
||||
|
||||
**Frame field naming.** SQLite stores `snake_case` (`created_at`, `frame_type`, `gop_id`, `access_count`). The memory routes run results through `normalizeFrame()` which emits a **camelCase UI shape** (see below). The knowledge-graph and wiki routes return **raw DB rows** (snake_case), not normalized.
|
||||
|
||||
---
|
||||
|
||||
## 1. Memory frames — recall & save (`routes/memory.ts`)
|
||||
|
||||
Base entity is a **memory frame**. The UI-normalized frame shape returned by search/list/edit endpoints:
|
||||
|
||||
| Field (UI) | Type | Source DB column | Notes |
|
||||
|---|---|---|---|
|
||||
| `id` | number | `id` | |
|
||||
| `content` | string | `content` | |
|
||||
| `source` | string | `source` | provenance: `user_stated` \| `tool_verified` \| `agent_inferred` \| `import` \| `system` |
|
||||
| `source_mind` | string | (computed) | `personal` \| `workspace` — which mind it came from |
|
||||
| `mind` | string | (computed) | legacy alias of `source_mind` |
|
||||
| `frameType` | string | `frame_type` | `'I'` (independent) or `'P'` (predicate); defaults `'I'` |
|
||||
| `importance` | string | `importance` | `critical` \| `important` \| `normal` \| `temporary` \| `deprecated` |
|
||||
| `timestamp` | string (ISO) | `created_at` | |
|
||||
| `score` | number? | `score` | present on search results |
|
||||
| `gop` | string | `gop_id` | session/group id |
|
||||
| `accessCount` | number | `access_count` | |
|
||||
| `authorId` / `authorName` | string? | `author_id` / `author_name` | only on team-synced frames |
|
||||
| `workspaceName` | string? | `_workspace_name` | only on global cross-workspace search |
|
||||
|
||||
### Endpoints
|
||||
|
||||
| Method | Path | Purpose |
|
||||
|---|---|---|
|
||||
| GET | `/api/memory/search` | Full-text search frames across personal + workspace minds |
|
||||
| GET | `/api/memory/frames` | List recent frames (no query) for Memory tab initial load |
|
||||
| POST | `/api/memory/frames` | Direct memory **write** (save a frame); optional entity extraction |
|
||||
| PUT | `/api/memory/frames/:id` | Edit a frame's content and/or importance |
|
||||
| PATCH | `/api/memory/frames/:id/access` | Atomically increment `access_count` |
|
||||
| DELETE | `/api/memory/frames/:id` | Delete a frame by id |
|
||||
| GET | `/api/memory/stats` | Frame/entity/relation counts (personal + optional workspace + totals) |
|
||||
|
||||
**`GET /api/memory/search`** — query params: `q` (required; 400 if missing), `scope` (`personal` \| `workspace` \| `all` \| `global`, default `all`), `limit` (default 20), `workspace`/`workspaceId`, `since`/`until` (ISO date strings for temporal filtering). Response: `{ results: Frame[], count: number }`. `scope=global` searches personal + every workspace mind and adds `workspaceName` to each result.
|
||||
|
||||
**`GET /api/memory/frames`** — params: `workspace`/`workspaceId`, `limit` (default 50), `since`/`until`. Returns `{ results: Frame[], count: number }` sorted newest-first.
|
||||
|
||||
**`POST /api/memory/frames`** — body `{ content (required), workspace?, importance?, source? }`, query `?extract=true|false` (default true; runs entity extraction + co-occurrence relations after save). `content` is XSS-sanitized server-side. Validates `importance` (default `normal`) and `source` (default `import`; 400 on invalid). Dedupes identical content. Response on success: `{ saved: true, frameId, mind, importance, source, extraction?: { entitiesExtracted, relationsCreated } }`. On duplicate: `{ saved: false, duplicate: true, frameId, mind, message }`. Emits a `memory_write` audit event.
|
||||
|
||||
**`PUT /api/memory/frames/:id`** — body `{ content (required), importance? }`, query `workspace?`. 400 on missing content / invalid importance, 404 if frame not found. Returns the normalized frame plus `updated: true`.
|
||||
|
||||
**`PATCH /api/memory/frames/:id/access`** — query `workspace?`. Returns `{ frameId, accessed: true, accessCount, mind }`; 404 if not found.
|
||||
|
||||
**`DELETE /api/memory/frames/:id`** — query `workspace?`. Returns `{ deleted: true, frameId }`; 404 if not found. Emits `memory_delete` audit event.
|
||||
|
||||
**`GET /api/memory/stats`** — query `workspace?`. Returns:
|
||||
```json
|
||||
{
|
||||
"personal": { "frameCount": 0, "entityCount": 0, "relationCount": 0 },
|
||||
"workspace": { "frameCount": 0, "entityCount": 0, "relationCount": 0 } /* or null */,
|
||||
"total": { "frameCount": 0, "entityCount": 0, "relationCount": 0 }
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Knowledge graph read (`routes/knowledge.ts`)
|
||||
|
||||
| Method | Path | Purpose |
|
||||
|---|---|---|
|
||||
| GET | `/api/memory/graph` | Read entities + relations from a mind (or merged across all minds) |
|
||||
|
||||
**Query params:** `workspace`/`workspaceId`, `scope` (`all` \| `personal` \| `current`).
|
||||
|
||||
- `scope=personal` (also the default when no workspace + no scope): returns the personal mind's graph.
|
||||
- `scope=all`: merges the **personal mind plus every workspace mind**, tagging each row with a `_source` field (`'personal'` or the workspace name) and re-offsetting ids/relation endpoints to avoid collisions.
|
||||
- With a `workspace` id (and not `scope=all`): returns that workspace's graph; **404** `{ error: 'Workspace not found' }` if the mind can't be resolved. The workspace id is validated via `assertSafeSegment`.
|
||||
|
||||
**Response shape:** `{ entities: KGRow[], relations: KGRow[] }`. Rows are **raw DB rows** from the live (non-expired) graph:
|
||||
|
||||
```sql
|
||||
SELECT * FROM knowledge_entities WHERE valid_to IS NULL ORDER BY name
|
||||
SELECT * FROM knowledge_relations WHERE valid_to IS NULL ORDER BY id
|
||||
```
|
||||
|
||||
Known `KGRow` fields (other columns pass through untyped): entities have `id`, `name`, `type`; relations have `id`, `source_id`, `target_id`, `type`. With `scope=all`, each row also carries `_source`.
|
||||
|
||||
> **No write/CRUD endpoints for entities/relations exist in these route files.** Entity/relation creation happens implicitly: (a) inside `POST /api/memory/frames?extract=true` (entity upsert + `co_occurs_with` relations), and (b) during the harvest **cognify** step (see §4). For direct graph mutation the rebuild would use the MCP tools (`save_entity`, `create_relation`) not an HTTP route. `KnowledgeGraph` is imported but no `POST /api/memory/graph` is registered.
|
||||
|
||||
---
|
||||
|
||||
## 3. Wiki compiler (`routes/wiki.ts`)
|
||||
|
||||
All wiki routes operate on the **personal mind only** (`server.multiMind.personal`). Pages are persisted in the `wiki_pages` SQLite table; metadata/state is managed by `CompilationState` from `@waggle/wiki-compiler`.
|
||||
|
||||
| Method | Path | Purpose |
|
||||
|---|---|---|
|
||||
| GET | `/api/wiki/pages` | List all compiled page metadata |
|
||||
| GET | `/api/wiki/pages/:slug` | Get a single page's metadata (404 if missing) |
|
||||
| GET | `/api/wiki/pages/:slug/content` | Get full markdown content of a page |
|
||||
| POST | `/api/wiki/compile` | Trigger wiki compilation (incremental or full) |
|
||||
| GET | `/api/wiki/health` | Compilation health report (gaps / data quality) |
|
||||
| GET | `/api/wiki/watermark` | Current compilation watermark/state |
|
||||
| POST | `/api/wiki/export/obsidian` | Write all pages to a local dir (Obsidian-vault layout) |
|
||||
| POST | `/api/wiki/export/notion` | Push pages as child pages under a Notion root |
|
||||
|
||||
**`GET /api/wiki/pages`** → array of page metadata objects (`state.getAllPages()`).
|
||||
**`GET /api/wiki/pages/:slug`** → single metadata object, or 404 `{ error: 'Page not found' }`.
|
||||
**`GET /api/wiki/pages/:slug/content`** → `{ slug, markdown }`, or 404. Reads `SELECT markdown FROM wiki_pages WHERE slug = ?`.
|
||||
|
||||
**`POST /api/wiki/compile`** — body `{ mode?: 'incremental' | 'full', concepts?: string[] }` (default `incremental`). Response merges the compiler result with `{ llmProvider, llmModel }`.
|
||||
**Critical degraded state:** if there is **no real embedding provider** (`server.embeddingProvider` absent or active provider is `'mock'`), returns **503** `{ error, skippedReason: 'no_real_embedder' }`. The UI must surface an "add a Voyage/OpenAI key (or Ollama embedding model)" prompt rather than treating this as a normal failure. The same 503 guard applies to `GET /api/wiki/health`.
|
||||
|
||||
**`GET /api/wiki/watermark`** → current `CompilationState` watermark.
|
||||
|
||||
**`POST /api/wiki/export/obsidian`** — body `{ outDir }`. Validation: `outDir` required (400) and **must be an absolute path** (400). 409 if there are no compiled pages yet ("Run compile first"). 500 on write failure. Output layout: `{outDir}/{pageType}/{slug}.md` plus a top-level `_index.md`.
|
||||
|
||||
**`POST /api/wiki/export/notion`** — body `{ rootPageUrl }` (a notion.so URL or raw page id). 400 if missing or unparseable. Requires a Vault secret named **`notion-wiki-token`** — **503** with setup instructions if absent. 409 if no compiled pages. Returns the Notion writer stats. Delta-tracking (Notion page ids per slug, content hashes) is delegated to `CompilationState`.
|
||||
|
||||
---
|
||||
|
||||
## 4. Harvest — external AI export ingestion (`routes/harvest.ts`)
|
||||
|
||||
Harvest ingests AI tool exports into the **personal mind**, then runs **cognify** (entity/relation extraction) and an incremental **wiki recompile**. Supported `source` values (`ImportSourceType`): `chatgpt`, `claude`, `claude-desktop`, `claude-code`, `gemini`, `google-ai-studio`, and anything else falls through to a `UniversalAdapter`.
|
||||
|
||||
| Method | Path | Purpose |
|
||||
|---|---|---|
|
||||
| POST | `/api/harvest/preview` | Parse an export and show what would be imported (no save) |
|
||||
| POST | `/api/harvest/commit` | Run full pipeline: save frames → cognify → wiki recompile |
|
||||
| GET | `/api/harvest/sources` | List registered harvest sources |
|
||||
| POST | `/api/harvest/sources` | Register or update a source |
|
||||
| DELETE | `/api/harvest/sources/:source` | Remove a registered source |
|
||||
| PATCH | `/api/harvest/sources/:source` | Toggle auto-sync / interval for a source |
|
||||
| GET | `/api/harvest/progress` | **SSE** stream of import progress events |
|
||||
| GET | `/api/harvest/runs` | List recent harvest runs (debug/history) |
|
||||
| GET | `/api/harvest/runs/latest-interrupted` | Latest resumable run (drives "Resume?" banner) |
|
||||
| POST | `/api/harvest/runs/:id/abandon` | Discard an interrupted run + its cached input |
|
||||
| POST | `/api/harvest/extract-identity` | LLM-extract identity facts from recent harvest frames |
|
||||
| POST | `/api/harvest/scan-claude-code` | Scan local `~/.claude` dir for Claude Code history |
|
||||
|
||||
**`POST /api/harvest/preview`** — body `{ data, source }` (400 if either missing). Response:
|
||||
```json
|
||||
{
|
||||
"source": "claude",
|
||||
"itemCount": 42,
|
||||
"types": { "conversation": 40, "decision": 2 },
|
||||
"preview": [ { "id": "...", "title": "...", "type": "...", "source": "..." } ] /* first 10 */
|
||||
}
|
||||
```
|
||||
|
||||
**`POST /api/harvest/commit`** — body `{ data?, source?, resumeFromRun? }`. Two entry modes:
|
||||
- **Fresh run:** `data` + `source` required (400 otherwise). To trigger a **local filesystem scan** (Claude Code), send `data: { scanLocal: true }` with `source: 'claude-code'` — the adapter scans `~/.claude`. 400 if the source has no filesystem adapter or no default dir.
|
||||
- **Resume:** `resumeFromRun: <runId>` replays a prior interrupted run's cached input. 404 if run not found; 409 if already `completed`/`abandoned`; 410 if its cached input is gone/corrupt.
|
||||
|
||||
Behavior: caps each item's content at **10,000 chars**, preserves the original `item.timestamp` (strict ISO-8601 validation; falls back to ingest wall-clock with a logged warning otherwise), skips work if the content hash matches the last sync (`skipped: true`), then runs cognify and an incremental wiki compile (both fail-soft and both skip with a reason if there's no real embedder). Success response:
|
||||
```json
|
||||
{
|
||||
"source": "claude",
|
||||
"itemCount": 42,
|
||||
"saved": 42,
|
||||
"cognified": 42,
|
||||
"cognifySkippedReason": null, /* or "no_real_embedder" */
|
||||
"entitiesExtracted": 18,
|
||||
"relationsCreated": 7,
|
||||
"wikiCompiled": { "pagesCreated": 3, "pagesUpdated": 1, "pagesUnchanged": 12 },
|
||||
"wikiSkippedReason": null, /* or "no_real_embedder" */
|
||||
"runId": 17,
|
||||
"message": "Imported 42 items from claude, cognified 42 frames, wiki 4 pages updated"
|
||||
}
|
||||
```
|
||||
Other commit responses: nothing found → `{ saved: 0, message }`; unchanged since last sync → `{ saved: 0, skipped: true, message }`; personal mind unavailable → **503**.
|
||||
|
||||
**`GET /api/harvest/sources`** → `{ sources: HarvestSource[] }`. A source row carries (from `HarvestSourceStore`) `source`, display name, `sourcePath`, sync tracking incl. `lastContentHash`, and auto-sync config.
|
||||
**`POST /api/harvest/sources`** — body `{ source, displayName (both required → 400), sourcePath?, autoSync?, syncIntervalHours? }`. → `{ source: HarvestSource }`.
|
||||
**`DELETE /api/harvest/sources/:source`** → `{ ok: true }`.
|
||||
**`PATCH /api/harvest/sources/:source`** — body `{ autoSync?, syncIntervalHours? }` → `{ source: HarvestSource }`.
|
||||
|
||||
**`GET /api/harvest/progress`** — Server-Sent Events. Each event is `data: <json>` where json is `{ phase, current, total, source }`. Phases observed: `saving`, `cognifying`, `wiki-compile`. The UI subscribes on mount.
|
||||
|
||||
**`GET /api/harvest/runs`** — query `limit` (default 50, max 500) → `{ runs: HarvestRun[] }`.
|
||||
**`GET /api/harvest/runs/latest-interrupted`** → `{ run: HarvestRun | null }` (null if no resumable run or its cache file no longer exists). Run states: `running`, `failed`, `completed`, `abandoned`.
|
||||
**`POST /api/harvest/runs/:id/abandon`** — 400 invalid id, 404 not found; returns the (now abandoned) run.
|
||||
|
||||
**`POST /api/harvest/extract-identity`** — scans the most recent 50 `harvest` frames, sandboxes them, and calls the internal Haiku proxy (`claude-haiku-4-5`) to extract `{ name, role, company, industry, bio }` suggestions. Server-side gate: only fields with `confidence >= 0.5` survive (`MIN_SUGGESTION_CONFIDENCE`). Persists to `profile.identitySuggestions`. Response: `{ suggestions: IdentitySuggestion[] }` (with `note: 'no_anthropic_key'` if no Anthropic key in Vault). 503 if personal mind unavailable. `IdentitySuggestion = { field, value, confidence, sourceHint, extractedAt }`.
|
||||
|
||||
**`POST /api/harvest/scan-claude-code`** — scans `~/.claude`. Response `{ found, path, itemCount, types?, preview? }` (`preview` = first 20 items with `id`, `title`, `type`, `metadata`).
|
||||
|
||||
---
|
||||
|
||||
## 5. Legacy import (`routes/import.ts`)
|
||||
|
||||
A simpler, older import path for **ChatGPT and Claude only** (`ImportSource = 'chatgpt' | 'claude'`). Distinct from harvest: no run-store, no cognify, no wiki recompile, no SSE.
|
||||
|
||||
| Method | Path | Purpose |
|
||||
|---|---|---|
|
||||
| POST | `/api/import/preview` | Parse a ChatGPT/Claude export, show extraction (no save) |
|
||||
| POST | `/api/import/commit` | Parse + extract + save knowledge items to personal memory |
|
||||
|
||||
Both take body `{ data, source }`. 400 if missing, or if `source` is not exactly `"chatgpt"` or `"claude"`. Both return the `ImportResult` shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"source": "chatgpt",
|
||||
"conversationsFound": 10,
|
||||
"conversationsParsed": 10,
|
||||
"knowledgeExtracted": [
|
||||
{ "type": "decision|fact|preference|topic",
|
||||
"content": "...",
|
||||
"source": "chatgpt|claude",
|
||||
"conversationTitle": "...",
|
||||
"importance": "important|normal" }
|
||||
],
|
||||
"errors": []
|
||||
}
|
||||
```
|
||||
|
||||
`/api/import/commit` additionally writes each extracted item as an `import`-source frame and adds `{ saved: <n>, message }`. If nothing was extracted: `{ ...result, saved: 0, message: 'No knowledge items found to import' }`. 503 if personal mind unavailable; 500 on save failure.
|
||||
|
||||
---
|
||||
|
||||
## 6. Identity (`routes/identity.ts`) + Mind context (`routes/mind.ts`)
|
||||
|
||||
### Identity record (`routes/identity.ts`)
|
||||
|
||||
Single-row-per-mind identity table (id = 1), so create + update collapse into upsert. Defaults to the personal mind; `?workspace=<id>` selects a workspace-scoped identity (each mind owns its own identity row).
|
||||
|
||||
| Method | Path | Purpose |
|
||||
|---|---|---|
|
||||
| GET | `/api/identity` | Read the identity record (placeholder shape if unconfigured) |
|
||||
| POST | `/api/identity` | Create or update the identity record (upsert) |
|
||||
|
||||
**`GET /api/identity?workspace=<id>`** always returns 200 with the same `IdentityResponseShape`:
|
||||
```json
|
||||
{
|
||||
"configured": true,
|
||||
"name": null, "role": null, "department": null,
|
||||
"personality": null, "capabilities": null, "system_prompt": null,
|
||||
"created_at": null, "updated_at": null,
|
||||
"_note": "identity not configured for this mind" /* present only on placeholder */
|
||||
}
|
||||
```
|
||||
When unconfigured/uninitialized, `configured: false` and all fields `null` with a `_note` explaining why. The UI sees **one consistent shape** regardless of state.
|
||||
|
||||
**`POST /api/identity`** — body `{ workspace?, name?, role?, department?, personality?, capabilities?, system_prompt? }` (all optional, default to empty string). Returns the saved `IdentityResponseShape` (`configured: true` + populated `created_at`/`updated_at`). 503 if multi-mind not initialized; 500 on write failure.
|
||||
|
||||
### Mind context (`routes/mind.ts`)
|
||||
|
||||
Read-only context the CLI also accesses directly via the Orchestrator.
|
||||
|
||||
| Method | Path | Purpose |
|
||||
|---|---|---|
|
||||
| GET | `/api/mind/identity` | Agent identity **context string** → `{ identity }` |
|
||||
| GET | `/api/mind/awareness` | Current awareness state context → `{ awareness }` |
|
||||
| GET | `/api/mind/skills` | Loaded skills list → `{ skills: [{ name, length }], count }` |
|
||||
|
||||
> Note: `/api/mind/identity` returns the orchestrator's rendered identity **context** (`identity.toContext()`), not the structured record — that structured record is `/api/identity`.
|
||||
|
||||
---
|
||||
|
||||
## 7. Document version registry (`routes/documents.ts`)
|
||||
|
||||
Tracks document **versions** per workspace. Stored as JSON on disk at `~/.waggle/workspaces/{id}/documents.json` — **not** in SQLite. Part of Wave 7. All path segments validated via `assertSafeSegment`.
|
||||
|
||||
| Method | Path | Purpose |
|
||||
|---|---|---|
|
||||
| GET | `/api/workspaces/:id/documents` | List tracked documents (name + version count + latest) |
|
||||
| POST | `/api/workspaces/:id/documents` | Register a new document version (auto-increments version) |
|
||||
| GET | `/api/workspaces/:id/documents/:name/versions` | List all versions of one document |
|
||||
|
||||
**`GET .../documents`** → `{ documents: [{ name, versionCount, latestVersion: DocumentVersion | null }] }`.
|
||||
**`POST .../documents`** — body `{ name (required), path (required), sizeBytes? }`. 400 if name/path missing. Auto-assigns `version` = lastVersion+1 (starts at 1). Returns **201** `{ document: <name>, version: DocumentVersion }`.
|
||||
**`GET .../:name/versions`** → `{ name, versions: DocumentVersion[] }`; 404 if document not found.
|
||||
|
||||
`DocumentVersion = { version: number, path: string, createdAt: string (ISO), sizeBytes: number }`.
|
||||
|
||||
---
|
||||
|
||||
## 8. GDPR data erasure (`routes/data-erase.ts`)
|
||||
|
||||
Right-to-erasure (GDPR Art. 17). **The route does NOT delete anything** — it validates, snapshots, and writes a marker file. The destructive wipe runs at next service startup, before any DB is opened.
|
||||
|
||||
| Method | Path | Purpose |
|
||||
|---|---|---|
|
||||
| POST | `/api/data/erase` | Schedule full erasure of this install's data dir at next startup |
|
||||
|
||||
**Double confirmation gate (both required):**
|
||||
1. HTTP header **`X-Confirm-Erase: yes`** (exact, case-sensitive).
|
||||
2. Body field **`{ "confirmation": "I UNDERSTAND THIS IS PERMANENT" }`** (exact phrase).
|
||||
|
||||
Failure responses:
|
||||
- Missing/wrong confirmation → **400** `{ error: 'ERASE_NOT_CONFIRMED', message, requirements: { header, bodyField } }` (the `requirements` object literally tells the UI the exact header value and body phrase needed).
|
||||
- Data dir doesn't look like a Waggle dir → **400** `{ error: 'ERASE_REFUSED_UNSAFE_PATH', message }`.
|
||||
- Marker write fails → **500** `{ error: 'ERASE_MARKER_WRITE_FAILED', message }`.
|
||||
|
||||
Success → **200**:
|
||||
```json
|
||||
{
|
||||
"requestedAt": "2026-06-06T...Z",
|
||||
"markerPath": "...",
|
||||
"dataDirSnapshot": { "fileCount": 0, "totalBytes": 0, "topLevelEntries": [] },
|
||||
"instruction": "Quit Waggle and relaunch — erasure completes during startup. ..."
|
||||
}
|
||||
```
|
||||
Emits a `data_erase_requested` audit event **before** writing the marker (audit survives even if the marker write fails).
|
||||
|
||||
---
|
||||
|
||||
## Frontend rebuild cheat-sheet
|
||||
|
||||
- **Workspace param:** send `workspace=<id>` (alias `workspaceId` also works). Omit / `personal` → personal mind.
|
||||
- **Frame shape differs by route:** memory routes return **camelCase normalized** frames; knowledge-graph + wiki routes return **raw snake_case** rows.
|
||||
- **No real embedder = 503 with `skippedReason: 'no_real_embedder'`** on `/api/wiki/compile` and `/api/wiki/health`, and `*SkippedReason` flags inside the harvest commit response. Build UI affordances to prompt for an embedding key.
|
||||
- **Harvest vs Import:** harvest (`/api/harvest/*`) is the rich, resumable, multi-source pipeline with SSE + cognify + wiki recompile; import (`/api/import/*`) is the legacy ChatGPT/Claude-only path. Prefer harvest.
|
||||
- **Two identity surfaces:** `/api/identity` = structured editable record (GET/POST upsert, always one shape); `/api/mind/identity` = read-only rendered context string. Don't confuse them.
|
||||
- **Erasure needs both** the `X-Confirm-Erase: yes` header **and** the exact body phrase — and the 400 response hands you the exact requirements.
|
||||
- **Personal-mind-not-available → 503** is a common guard across harvest/import/identity write routes; handle it as a transient backend-not-ready state.
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant UI
|
||||
participant API as Sidecar
|
||||
participant DB as personal.mind
|
||||
UI->>API: POST /api/harvest/preview {data,source}
|
||||
API-->>UI: {itemCount, types, preview}
|
||||
UI->>API: GET /api/harvest/progress (SSE open)
|
||||
UI->>API: POST /api/harvest/commit {data,source}
|
||||
API->>DB: save frames (cap 10k chars)
|
||||
API-->>UI: SSE {phase:"saving",...}
|
||||
API->>DB: cognify (entities+relations)
|
||||
API-->>UI: SSE {phase:"cognifying",...}
|
||||
API->>DB: incremental wiki compile
|
||||
API-->>UI: SSE {phase:"wiki-compile",...}
|
||||
API-->>UI: {saved, cognified, wikiCompiled, runId}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## File Ingestion (`ingest.ts`) — `POST /api/ingest`
|
||||
|
||||
> Added to close audit gap #1. This is the endpoint the UI's file-upload / drag-drop surface calls to turn user files into LLM-ready text and (optionally) memory frames. Distinct from `/api/harvest/*` (conversation exports) and `/api/import` (memory import).
|
||||
|
||||
| Method | Path | Request body | Response | Streaming? |
|
||||
|---|---|---|---|---|
|
||||
| `POST` | `/api/ingest` | `{ files: { name: string, content: string }[], workspaceId?: string }` — `content` is **base64**. Route `bodyLimit` = **15 MB** (allows base64 overhead) | `{ files: IngestFileResult[] }` | No |
|
||||
|
||||
**`IngestFileResult`** = `{ name: string; type: FileType; summary: string; content?: string }`
|
||||
where `type` ∈ `image | document | spreadsheet | csv | text | archive | unsupported`.
|
||||
- For **images**, `content` is a `data:<mime>;base64,...` data URI (passed straight to a vision model).
|
||||
- For everything else, `content` is the **extracted plain text** (omitted for `unsupported` / failed extraction).
|
||||
|
||||
**Validation & error codes** (per-file, fail-fast):
|
||||
| Condition | Code |
|
||||
|---|---|
|
||||
| `files` missing / not an array / empty | `400 { error: "files array is required" }` |
|
||||
| file entry missing `name` or non-string `content` | `400 { error: "Invalid file entry: <name>" }` |
|
||||
| `content` not well-formed base64 (`/^[A-Za-z0-9+/]*={0,2}$/`) | `400 { error: "Invalid base64 content for file: <name>" }` |
|
||||
| decoded size > **10 MB** (`MAX_FILE_SIZE`, est. `content.length * 0.75`) | `413 { error: "File <name> exceeds 10 MB limit" }` |
|
||||
| `workspaceId` fails `assertSafeSegment` (path-traversal guard) | throws (4xx) before any fs touch |
|
||||
|
||||
**Supported extraction** (extension → handler):
|
||||
- **Images** png/jpg/jpeg/gif/webp/svg/bmp/ico/tiff → data URI.
|
||||
- **Documents** pdf (`pdf-parse`), docx (`mammoth`), pptx (`adm-zip`, reads `ppt/slides/slideN.xml`). Missing optional dep or scanned/encrypted file → graceful `summary` with no `content`.
|
||||
- **Spreadsheets** xlsx/xls (`exceljs`, each sheet → CSV-ish text).
|
||||
- **csv** (RFC-4180 line parse, reports columns/rows).
|
||||
- **Text/code** ~50 extensions (md/txt/json/yaml/ts/js/py/rs/go/sql/dockerfile/...).
|
||||
- **Archives** zip → lists up to 50 entry names (no extraction).
|
||||
- Unknown extension → `type: "unsupported"`, skipped from side effects.
|
||||
|
||||
**Side effects** (only when `workspaceId` is set and ≠ `"default"`, both non-blocking — ingest still 200s if they fail):
|
||||
1. Appends each non-unsupported result to the workspace **file registry** `workspaces/<id>/files.jsonl` (`{ name, type, summary, sizeBytes, ingestedAt }` — this is what `GET /api/files` reads, see 04-feature-map).
|
||||
2. Saves a **memory frame** per file via `orchestrator.autoSaveFromExchange("User uploaded file: <name>", "File ingested: ... + 500-char preview")` so uploads persist across sessions.
|
||||
Reference in New Issue
Block a user