refactor: drop the one-time copy of legacy .mind data

The first start copied Waggle's tables and the memories of the commercial
edition's .mind files into waggle.db and mem0. The product is still in
development and nobody has data to bring over, so both copies, their tests
and the legacy creation-time field are gone. ADR-0007 records the change.
This commit is contained in:
shumeiko.it
2026-10-28 12:35:41 +01:00
parent fb3697ca6c
commit a8250cb375
12 changed files with 15 additions and 618 deletions

View File

@@ -1,7 +1,7 @@
# ADR-0007: The memory contract and how it maps onto mem0
- Status: Accepted
- Date: 2026-10-18
- Date: 2026-10-18, amended 2026-10-28 (no copy of legacy data)
## Context
@@ -89,11 +89,10 @@ The stores of these tables take a `WaggleDB`, not a generic SQLite handle, so ha
memory database is a type error. Telemetry (`telemetry.db`) and the audit log (`audit.db`)
already had their own files and stay there.
On the first start the server copies these tables from the legacy `personal.mind`, and the
profile and awareness items of every `workspace.mind`, into `waggle.db`
(`importLegacyMindData`). The legacy files are only read, because the data directory may be
shared with the commercial edition; each scope is copied once (recorded in `waggle_meta`) and a
table only while its target is empty.
Nothing is copied from the `.mind` files of earlier versions. Until 2026-10-28 the first start
copied these tables and the legacy memories (`importLegacyMindData`, `legacy-memory-import.ts`);
the owner removed both because the product is still in development and nobody has data to
bring over. A data directory left by the commercial edition is ignored.
The logger, injection scanner, external-memory ingress guard, `WorkspaceManager` and the
embedders were copied from `@waggle/hive-mind-core` into `@waggle/core`, which exports its own
@@ -117,11 +116,6 @@ copies is part of the licensing review.
model through the built-in OpenAI-compatible proxy (or LiteLLM when the user runs it)
and reads the user's current model on every call. `memoryExtraction` in `config.json`
turns it off (default on). Failures are logged and never block a turn.
- **Legacy memories** are copied once into the store in the background
(`legacy-memory-import.ts`). The `.mind` files are opened read-only. Content prefixes
become kinds, `[mind-rawturn …]` frames and deprecated frames are skipped, the original
creation time is kept in `metadata.legacy_created_at` because mem0 sets its own. Progress
is kept per scope in `waggle_meta`, so an interrupted copy resumes.
- **Writers.** Chat write-back, compaction summaries, sub-agent and agent-run results
(`run-memory.ts`), loops, connector fetch, indexed workspace files and the failed-turn
capture all call `remember`. Contract field names are reserved in metadata; agent-run
@@ -136,8 +130,8 @@ copies is part of the licensing review.
a seventh of the latency. The reranker did not improve mem0's ranking and costs about
130 ms per search, so it stays off. The benchmark keeps running against mem0 alone after
the old engine is removed.
- **Waggle data moved to waggle.db** as well: the install audit trail (`install_audit`,
copied from `personal.mind` with the other personal tables) and the file index
- **Waggle data moved to waggle.db** as well: the install audit trail (`install_audit`)
and the file index
(`file_index`, keyed by scope and path, pointing at memory ids; files are re-indexed on
their next write rather than migrated).
- **Removed with the old engine:** the weaver package (consolidation, decay, transcript

View File

@@ -93,8 +93,7 @@ app\src-tauri\target\x86_64-pc-windows-msvc\release\bundle\nsis\Waggle_0.2.0_x64
`%USERPROFILE%\.waggle\logs\service.log`.
4. Chat needs a model: an API key in Settings → API Keys, or a running Ollama.
Waggle keeps its data in `%USERPROFILE%\.waggle`. If the commercial edition was used on this
machine, the first start copies its data once and leaves the old files unchanged.
Waggle keeps its data in `%USERPROFILE%\.waggle`.
## Troubleshooting

View File

@@ -52,7 +52,6 @@ export {
type TracePayload, type TraceToolCall, type TraceReasoningStep,
type StartTraceInput, type FinalizeTraceInput, type TraceQueryFilter,
} from './execution-traces.js';
export { importLegacyMindData, type LegacyMindSource, type LegacyImportReport } from './waggle-db-import.js';
// ── Waggle-specific orchestration ──
export { WaggleConfig, type ProviderEntry } from './config.js';

View File

@@ -1,165 +0,0 @@
/**
* One-time copy of Waggle's own tables from the legacy memory databases
* (personal.mind, workspace.mind) into waggle.db.
*
* The legacy files are only read: the data directory may be shared with the
* commercial edition, which keeps using them. Each source is copied once,
* recorded in waggle_meta, and a table is copied only while its target is
* empty, so a re-run never duplicates rows.
*/
import fs from 'node:fs';
import Database from 'better-sqlite3';
import type { Database as DatabaseType } from 'better-sqlite3';
import { AwarenessLayer } from './awareness.js';
import { CronStore } from './cron-store.js';
import { ExecutionTraceStore } from './execution-traces.js';
import { IdentityLayer } from './identity.js';
import { SkillHashStore } from './skill-hashes.js';
import type { DataScope, WaggleDB } from './waggle-db.js';
/** Tables that only ever lived in personal.mind, copied with their ids. */
const PERSONAL_TABLES = [
'cron_schedules',
'cron_execution_history',
'cron_run_leases',
'notifications',
'pending_actions',
'skill_hashes',
'feedback_entries',
'execution_traces',
'execution_trace_spend',
'execution_trace_spend_reservations',
'install_audit',
] as const;
const PROFILE_COLUMNS = ['name', 'role', 'department', 'personality', 'capabilities', 'system_prompt', 'created_at', 'updated_at'];
const AWARENESS_COLUMNS = ['category', 'content', 'priority', 'metadata', 'created_at', 'expires_at'];
export interface LegacyMindSource {
/** Path to a legacy .mind file. */
path: string;
scope: DataScope;
}
export interface LegacyImportReport {
scope: DataScope;
status: 'imported' | 'already-imported' | 'missing' | 'failed';
copied: Record<string, number>;
error?: string;
}
function importKey(scope: DataScope): string {
return `import:${scope}`;
}
function tableExists(db: DatabaseType, table: string): boolean {
return db.prepare("SELECT 1 FROM sqlite_master WHERE type = 'table' AND name = ?").get(table) !== undefined;
}
function columnsOf(db: DatabaseType, table: string): string[] {
return (db.prepare(`PRAGMA table_info(${table})`).all() as Array<{ name: string }>).map(c => c.name);
}
function copyRows(
source: DatabaseType,
target: DatabaseType,
sourceTable: string,
targetTable: string,
columns: string[],
extra: Record<string, unknown> = {},
): number {
const rows = source.prepare(`SELECT ${columns.join(', ')} FROM ${sourceTable}`).all() as Array<Record<string, unknown>>;
if (rows.length === 0) return 0;
const allColumns = [...columns, ...Object.keys(extra)];
const insert = target.prepare(
`INSERT INTO ${targetTable} (${allColumns.join(', ')}) VALUES (${allColumns.map(() => '?').join(', ')})`,
);
for (const row of rows) {
insert.run(...columns.map(c => row[c]), ...Object.values(extra));
}
return rows.length;
}
function copyPersonalTables(source: DatabaseType, target: DatabaseType, copied: Record<string, number>): void {
for (const table of PERSONAL_TABLES) {
if (!tableExists(source, table)) continue;
if (!tableExists(target, table)) {
// Lazily created tables (feedback_entries) take the legacy definition.
const ddl = source.prepare("SELECT sql FROM sqlite_master WHERE type = 'table' AND name = ?").get(table) as { sql: string };
target.exec(ddl.sql);
}
const empty = target.prepare(`SELECT 1 FROM ${table} LIMIT 1`).get() === undefined;
if (!empty) continue;
const targetColumns = new Set(columnsOf(target, table));
const shared = columnsOf(source, table).filter(c => targetColumns.has(c));
copied[table] = copyRows(source, target, table, table, shared);
}
}
function copyScopedTables(source: DatabaseType, target: DatabaseType, scope: DataScope, copied: Record<string, number>): void {
if (tableExists(source, 'identity')) {
const hasProfile = target.prepare('SELECT 1 FROM profiles WHERE scope = ?').get(scope) !== undefined;
if (!hasProfile) {
const sourceColumns = new Set(columnsOf(source, 'identity'));
const shared = PROFILE_COLUMNS.filter(c => sourceColumns.has(c));
copied.profiles = copyRows(source, target, 'identity', 'profiles', shared, { scope });
}
}
if (tableExists(source, 'awareness')) {
const hasItems = target.prepare('SELECT 1 FROM awareness_items WHERE scope = ? LIMIT 1').get(scope) !== undefined;
if (!hasItems) {
const sourceColumns = new Set(columnsOf(source, 'awareness'));
const shared = AWARENESS_COLUMNS.filter(c => sourceColumns.has(c));
copied.awareness_items = copyRows(source, target, 'awareness', 'awareness_items', shared, { scope });
}
}
}
/**
* Copies Waggle data from each legacy source into `waggleDb`, once per scope.
* The personal source also brings the app-wide tables; workspace sources bring
* their profile and awareness items.
*
* Tables owned outside core (install_audit, owned by the server) must be created
* by their owner before this call; otherwise they take the legacy definition.
*/
export function importLegacyMindData(waggleDb: WaggleDB, sources: LegacyMindSource[]): LegacyImportReport[] {
// Create every core target table with its current schema first.
new CronStore(waggleDb);
new SkillHashStore(waggleDb);
new ExecutionTraceStore(waggleDb);
new IdentityLayer(waggleDb);
new AwarenessLayer(waggleDb);
const target = waggleDb.getDatabase();
const reports: LegacyImportReport[] = [];
for (const { path: sourcePath, scope } of sources) {
const copied: Record<string, number> = {};
if (waggleDb.getMeta(importKey(scope))) {
reports.push({ scope, status: 'already-imported', copied });
continue;
}
const stat = fs.statSync(sourcePath, { throwIfNoEntry: false });
if (!stat || stat.size === 0) {
reports.push({ scope, status: 'missing', copied });
continue;
}
let source: DatabaseType | null = null;
try {
source = new Database(sourcePath, { fileMustExist: true });
const db = source;
target.transaction(() => {
if (scope === 'personal') copyPersonalTables(db, target, copied);
copyScopedTables(db, target, scope, copied);
})();
waggleDb.setMeta(importKey(scope), new Date().toISOString());
reports.push({ scope, status: 'imported', copied });
} catch (err) {
reports.push({ scope, status: 'failed', copied: {}, error: err instanceof Error ? err.message : String(err) });
} finally {
source?.close();
}
}
return reports;
}

View File

@@ -1,14 +1,10 @@
import fs from 'node:fs';
import os from 'node:os';
import path from 'node:path';
import Database from 'better-sqlite3';
import { afterEach, beforeEach, describe, expect, it } from 'vitest';
import { AwarenessLayer } from '../src/awareness.js';
import { CronStore } from '../src/cron-store.js';
import { ExecutionTraceStore } from '../src/execution-traces.js';
import { IdentityLayer } from '../src/identity.js';
import { WaggleDB, workspaceDataScope } from '../src/waggle-db.js';
import { importLegacyMindData } from '../src/waggle-db-import.js';
let tmpDir: string;
@@ -65,65 +61,3 @@ describe('profile and awareness scopes', () => {
db.close();
});
});
describe('importLegacyMindData', () => {
function legacyMind(file: string, setup: (db: Database.Database) => void): string {
const p = path.join(tmpDir, file);
const db = new Database(p);
setup(db);
db.close();
return p;
}
it('copies app tables, the profile and awareness once, and leaves the source unchanged', () => {
const personalPath = legacyMind('personal.mind', (db) => {
db.exec(`
CREATE TABLE identity (id INTEGER PRIMARY KEY CHECK (id = 1), name TEXT NOT NULL, role TEXT NOT NULL DEFAULT '',
department TEXT NOT NULL DEFAULT '', personality TEXT NOT NULL DEFAULT '', capabilities TEXT NOT NULL DEFAULT '',
system_prompt TEXT NOT NULL DEFAULT '', created_at TEXT NOT NULL DEFAULT (datetime('now')),
updated_at TEXT NOT NULL DEFAULT (datetime('now')));
INSERT INTO identity (id, name, role) VALUES (1, 'Ana', 'Founder');
CREATE TABLE awareness (id INTEGER PRIMARY KEY AUTOINCREMENT, category TEXT NOT NULL, content TEXT NOT NULL,
priority INTEGER NOT NULL DEFAULT 0, created_at TEXT NOT NULL DEFAULT (datetime('now')), expires_at TEXT);
INSERT INTO awareness (category, content) VALUES ('task', 'Draft the plan');
CREATE TABLE feedback_entries (id INTEGER PRIMARY KEY AUTOINCREMENT, session_id TEXT NOT NULL,
message_index INTEGER NOT NULL, rating TEXT NOT NULL, reason TEXT, detail TEXT NOT NULL DEFAULT '',
created_at TEXT NOT NULL DEFAULT (datetime('now')));
INSERT INTO feedback_entries (session_id, message_index, rating) VALUES ('s1', 2, 'up');
`);
// App tables created by the real stores, as the legacy personal.mind had them.
const legacy = { getDatabase: () => db } as unknown as WaggleDB;
new CronStore(legacy).create({ name: 'Morning briefing', cronExpr: '0 8 * * *', jobType: 'proactive' });
new ExecutionTraceStore(legacy).start({ sessionId: 's1', input: 'hi' });
});
const wsPath = legacyMind('workspace.mind', (db) => {
db.exec(`CREATE TABLE awareness (id INTEGER PRIMARY KEY AUTOINCREMENT, category TEXT NOT NULL, content TEXT NOT NULL,
priority INTEGER NOT NULL DEFAULT 0, metadata TEXT NOT NULL DEFAULT '{}', created_at TEXT NOT NULL DEFAULT (datetime('now')),
expires_at TEXT);
INSERT INTO awareness (category, content) VALUES ('pending', 'Review the contract');`);
});
const before = fs.readFileSync(personalPath);
const db = new WaggleDB(path.join(tmpDir, 'waggle.db'));
const sources = [
{ path: personalPath, scope: 'personal' as const },
{ path: wsPath, scope: workspaceDataScope('alpha') },
{ path: path.join(tmpDir, 'missing.mind'), scope: workspaceDataScope('ghost') },
];
const reports = importLegacyMindData(db, sources);
expect(reports.map(r => r.status)).toEqual(['imported', 'imported', 'missing']);
expect(new IdentityLayer(db).get()).toMatchObject({ name: 'Ana', role: 'Founder', scope: 'personal' });
expect(new AwarenessLayer(db).getAll().map(i => i.content)).toEqual(['Draft the plan']);
expect(new AwarenessLayer(db, workspaceDataScope('alpha')).getAll().map(i => i.content)).toEqual(['Review the contract']);
expect(new CronStore(db).list().map(s => s.name)).toEqual(['Morning briefing']);
expect(db.getDatabase().prepare('SELECT COUNT(*) AS n FROM feedback_entries').get()).toEqual({ n: 1 });
expect(db.getDatabase().prepare('SELECT COUNT(*) AS n FROM execution_traces').get()).toEqual({ n: 1 });
expect(fs.readFileSync(personalPath).equals(before)).toBe(true);
const again = importLegacyMindData(db, sources);
expect(again.map(r => r.status)).toEqual(['already-imported', 'already-imported', 'missing']);
expect(new CronStore(db).list()).toHaveLength(1);
db.close();
});
});

View File

@@ -10,7 +10,7 @@ import Fastify, { type FastifyInstance, type FastifyRequest } from 'fastify';
import cors from '@fastify/cors';
import fastifyStatic from '@fastify/static';
import websocket from '@fastify/websocket';
import { WaggleDB, importLegacyMindData, workspaceDataScope, type LegacyMindSource, WorkspaceManager, WaggleConfig, createEmbeddingProvider, type EmbeddingProviderConfig, type EmbeddingProviderInstance, CronStore, AwarenessLayer, VaultStore, SkillHashStore, TelemetryStore, TELEMETRY_EVENTS, ExecutionTraceStore } from '@waggle/core';
import { WaggleDB, workspaceDataScope, WorkspaceManager, WaggleConfig, createEmbeddingProvider, type EmbeddingProviderConfig, type EmbeddingProviderInstance, CronStore, AwarenessLayer, VaultStore, SkillHashStore, TelemetryStore, TELEMETRY_EVENTS, ExecutionTraceStore } from '@waggle/core';
import { corsOriginAllowed } from './cors-config.js';
import { InstallAuditStore } from './install-audit.js';
import { selfBaseUrl, selfRequestHeaders } from './self-request.js';
@@ -68,7 +68,6 @@ import type { RiskLevel } from '@waggle/shared';
import { runConnectorFetch } from './connector-harvest.js';
import { openMemoryStore } from './memory-runtime.js';
import { MemoryExtractionQueue } from './memory-extraction.js';
import { importLegacyMemories, type LegacyMemorySource } from './legacy-memory-import.js';
import { PERSONAL_SCOPE, workspaceScope, type MemoryStore } from '@waggle/memory-contract';
import { runLoopTick } from './loop-executor.js';
import { enqueueHeldAction } from './held-action-executor.js';
@@ -486,36 +485,14 @@ export async function buildLocalServer(config: Partial<LocalConfig> = {}) {
const defaultWorkspaceId = defaultWorkspaceConfig.id;
server.decorate('workspaceManager', wsManager);
// The legacy memory file of earlier versions; only the one-time copies of
// legacy data below read it, and nothing creates it any more.
const personalPath = path.join(fullConfig.dataDir, 'personal.mind');
// Waggle's own data (automations, notifications, cost tracking, profile and
// awareness) lives in waggle.db, apart from memory (ADR-0007). The first
// start copies it from the legacy .mind files, which stay untouched.
// awareness) lives in waggle.db, apart from memory (ADR-0007).
const waggleDb = WaggleDB.openInDataDir(fullConfig.dataDir);
server.decorate('waggleDb', waggleDb);
const legacySources: LegacyMindSource[] = [{ path: personalPath, scope: 'personal' }];
for (const ws of wsManager.list()) {
try {
legacySources.push({ path: wsManager.getMindPath(ws.id), scope: workspaceDataScope(ws.id) });
} catch {
// A workspace whose directory fails the path checks has nothing to copy.
}
}
// Install audit store — persistent trail for capability install events. Its
// table must exist before the legacy copy below, which fills it.
// Install audit store — persistent trail for capability install events.
const auditStore = new InstallAuditStore(waggleDb);
server.decorate('auditStore', auditStore);
for (const report of importLegacyMindData(waggleDb, legacySources)) {
if (report.status === 'imported') {
log.info(`[waggle-db] copied legacy data for ${report.scope}: ${JSON.stringify(report.copied)}`);
} else if (report.status === 'failed') {
log.warn(`[waggle-db] could not copy legacy data for ${report.scope}: ${report.error}`);
}
}
// Cron store — SQLite-backed schedule persistence for Solo
const cronStore = new CronStore(waggleDb);
server.decorate('cronStore', cronStore);
@@ -2479,10 +2456,8 @@ export async function buildLocalServer(config: Partial<LocalConfig> = {}) {
// Stop MCP servers
await mcpRuntime.stopAll().catch(() => {});
// Stop collecting chat turns for fact extraction and the legacy memory copy.
// Stop collecting chat turns for fact extraction.
memoryExtraction.stop();
legacyMemoryImport.cancel();
await legacyMemoryImport.done;
// Close all workspace sessions (new concurrent model)
sessionManager.closeAll();
@@ -2500,31 +2475,5 @@ export async function buildLocalServer(config: Partial<LocalConfig> = {}) {
}
});
// One-time copy of memories from the legacy .mind files into the memory
// store, in the background; the legacy files are only read.
const legacyMemoryImport = (() => {
let cancelled = false;
const sources: LegacyMemorySource[] = [{ path: personalPath, scope: PERSONAL_SCOPE }];
for (const ws of wsManager.list()) {
try {
sources.push({ path: wsManager.getMindPath(ws.id), scope: workspaceScope(ws.id) });
} catch {
// A workspace whose directory fails the path checks has nothing to copy.
}
}
const done = new Promise<void>(resolve => setImmediate(resolve))
.then(() => importLegacyMemories(memory, waggleDb, sources, (report) => {
if (report.status === 'imported') {
log.info(`[memory] copied ${report.copied} legacy memories for ${report.scope} (${report.skipped} skipped)`);
} else if (report.status === 'failed') {
log.warn(`[memory] could not copy legacy memories for ${report.scope}: ${report.error}`);
}
}, () => cancelled))
.then(() => undefined, (err: unknown) => {
log.warn(`[memory] legacy memory copy stopped: ${err instanceof Error ? err.message : String(err)}`);
});
return { done, cancel: () => { cancelled = true; } };
})();
return server;
}

View File

@@ -1,173 +0,0 @@
/**
* One-time copy of memories from the legacy `.mind` files into the memory
* store (P1.5). The legacy files are only read: the data directory may be
* shared with the commercial edition. Each scope is copied once; progress is
* kept in waggle.db, so an interrupted copy resumes where it stopped, and the
* store does not write identical text twice in one scope.
*/
import fs from 'node:fs';
import Database from 'better-sqlite3';
import {
MEMORY_IMPORTANCE,
MEMORY_SOURCES,
scopeKey,
type MemoryImportance,
type MemoryKind,
type MemoryScope,
type MemorySource,
type MemoryStore,
} from '@waggle/memory-contract';
import type { WaggleDB } from '@waggle/core';
export interface LegacyMemorySource {
/** A legacy `.mind` file. */
path: string;
scope: MemoryScope;
}
export interface LegacyMemoryImportReport {
scope: string;
status: 'imported' | 'already-imported' | 'missing' | 'failed';
copied: number;
skipped: number;
error?: string;
}
interface LegacyFrameRow {
id: number;
content: string;
importance: string;
source: string | null;
created_at: string | null;
}
/** Content prefixes the legacy engine used as types, mapped to contract kinds. */
const PREFIX_KINDS: ReadonlyArray<[RegExp, MemoryKind, boolean]> = [
// [pattern, kind, strip the first line (a header)]
[/^\[mind-profile [^\]]*\]/, 'profile', true],
[/^\[mind-fact\]/, 'fact', true],
[/^\[mind-event\]/, 'event', true],
[/^(User preference:|Style note:)/, 'preference', false],
[/^Decision:/, 'decision', false],
[/^(\[Session summary|Research findings:)/, 'summary', false],
[/^(\[Loop:|\[Sub-agent:|User asked:|Work completed:|Recommendation:)/, 'note', false],
];
/** Raw dialogue turns were an engine-specific recall lane; they are not memories. */
const SKIPPED_PREFIX = /^\[mind-rawturn /;
const metaKey = (scope: MemoryScope): string => `memory-import:${scopeKey(scope)}`;
const BATCH = 200;
export async function importLegacyMemories(
memory: MemoryStore,
waggleDb: WaggleDB,
sources: readonly LegacyMemorySource[],
onProgress?: (report: LegacyMemoryImportReport) => void,
isCancelled: () => boolean = () => false,
): Promise<LegacyMemoryImportReport[]> {
const reports: LegacyMemoryImportReport[] = [];
for (const source of sources) {
if (isCancelled()) break;
const report = await importOne(memory, waggleDb, source, isCancelled);
reports.push(report);
onProgress?.(report);
}
return reports;
}
async function importOne(
memory: MemoryStore,
waggleDb: WaggleDB,
source: LegacyMemorySource,
isCancelled: () => boolean,
): Promise<LegacyMemoryImportReport> {
const scope = scopeKey(source.scope);
const report: LegacyMemoryImportReport = { scope, status: 'imported', copied: 0, skipped: 0 };
const state = waggleDb.getMeta(metaKey(source.scope));
if (state === 'done') return { ...report, status: 'already-imported' };
if (!fs.existsSync(source.path) || fs.statSync(source.path).size === 0) {
waggleDb.setMeta(metaKey(source.scope), 'done');
return { ...report, status: 'missing' };
}
let db: Database.Database | null = null;
try {
db = new Database(source.path, { readonly: true, fileMustExist: true });
const hasFrames = db.prepare(
"SELECT 1 FROM sqlite_master WHERE type = 'table' AND name = 'memory_frames'",
).get();
if (!hasFrames) {
waggleDb.setMeta(metaKey(source.scope), 'done');
return { ...report, status: 'missing' };
}
let lastId = state ? Number(state) || 0 : 0;
const page = db.prepare(
`SELECT id, content, importance, source, created_at FROM memory_frames
WHERE id > ? ORDER BY id LIMIT ?`,
);
for (;;) {
const rows = page.all(lastId, BATCH) as LegacyFrameRow[];
if (rows.length === 0) break;
for (const row of rows) {
if (isCancelled()) {
waggleDb.setMeta(metaKey(source.scope), String(lastId));
return report;
}
const mapped = mapFrame(row);
if (mapped) {
await memory.remember({
scope: source.scope,
text: mapped.text,
kind: mapped.kind,
importance: mapped.importance,
source: mapped.source,
origin: 'legacy-import',
...(row.created_at && { metadata: { legacy_created_at: row.created_at } }),
});
report.copied++;
} else {
report.skipped++;
}
lastId = row.id;
}
waggleDb.setMeta(metaKey(source.scope), String(lastId));
}
waggleDb.setMeta(metaKey(source.scope), 'done');
return report;
} catch (err) {
return { ...report, status: 'failed', error: err instanceof Error ? err.message : String(err) };
} finally {
db?.close();
}
}
/** Converts a legacy frame, or returns null for one that is not carried over. */
export function mapFrame(row: Pick<LegacyFrameRow, 'content' | 'importance' | 'source'>): {
text: string;
kind: MemoryKind;
importance: MemoryImportance;
source: MemorySource;
} | null {
const content = row.content?.trim() ?? '';
if (!content || row.importance === 'deprecated' || SKIPPED_PREFIX.test(content)) return null;
let kind: MemoryKind = 'fact';
let text = content;
for (const [pattern, mappedKind, stripHeader] of PREFIX_KINDS) {
if (!pattern.test(content)) continue;
kind = mappedKind;
if (stripHeader) {
const newline = content.indexOf('\n');
text = newline >= 0 ? content.slice(newline + 1).trim() : content.replace(pattern, '').trim();
}
break;
}
if (!text) return null;
const importance = (MEMORY_IMPORTANCE as readonly string[]).includes(row.importance)
? row.importance as MemoryImportance
: 'normal';
const source = row.source && (MEMORY_SOURCES as readonly string[]).includes(row.source)
? row.source as MemorySource
: 'import';
return { text, kind, importance, source };
}

View File

@@ -7,7 +7,7 @@
* importance it had.
* - `confirmedAt` (ISO 8601): the user confirmed a memory extracted in the background.
* - `title`, `tags`, `evidence`, `relatedMemoryIds`, `mergedIntoId`.
* Chat write-back stores `trace_id`; the legacy import stores `legacy_created_at`.
* Chat write-back stores `trace_id`.
*/
import type { FastifyInstance } from 'fastify';
import {
@@ -76,18 +76,3 @@ export function memoryLifecycle(record: MemoryRecord): MemoryLifecycle {
if (record.metadata.archived === true) return 'archived';
return awaitsReview(record) ? 'unreviewed' : 'active';
}
/**
* When the memory was first written. Memories copied from the legacy engine
* keep their original time in metadata; the store's own time is the copy time.
*/
export function memoryCreatedAt(record: MemoryRecord): string {
const legacy = record.metadata.legacy_created_at;
if (typeof legacy === 'string') {
// SQLite's datetime('now') is UTC without a zone: "YYYY-MM-DD HH:MM:SS".
const iso = /^\d{4}-\d\d-\d\d \d\d:\d\d:\d\d$/.test(legacy) ? `${legacy.replace(' ', 'T')}Z` : legacy;
const time = Date.parse(iso);
if (Number.isFinite(time)) return new Date(time).toISOString();
}
return record.createdAt;
}

View File

@@ -20,7 +20,6 @@ import {
asImportance,
asKind,
awaitsReview,
memoryCreatedAt,
memoryLifecycle,
workspaceIdOf,
writeScopeForWorkspace,
@@ -100,7 +99,7 @@ export function recordToMemory(record: MemoryRecord): Memory {
tags: stringArray(meta.tags),
relatedMemoryIds: stringArray(meta.relatedMemoryIds),
status: memoryLifecycle(record),
createdAt: memoryCreatedAt(record),
createdAt: record.createdAt,
...(record.updatedAt && { updatedAt: record.updatedAt }),
};
}

View File

@@ -22,7 +22,6 @@ import {
allScopes,
asImportance,
asKind,
memoryCreatedAt,
memoryLifecycle,
scopeForWorkspace,
workspaceIdOf,
@@ -193,7 +192,7 @@ export function toFrameResponse(record: MemoryRecord | MemoryHit, workspaceName?
mind: record.scope.kind,
workspaceId: workspaceIdOf(record) ?? null,
status: memoryLifecycle(record),
timestamp: memoryCreatedAt(record),
timestamp: record.createdAt,
...('score' in record ? { score: record.score } : {}),
...(workspaceName ? { workspaceName } : {}),
};

View File

@@ -1,112 +0,0 @@
import fs from 'node:fs';
import os from 'node:os';
import path from 'node:path';
import Database from 'better-sqlite3';
import { afterEach, describe, expect, it } from 'vitest';
import { WaggleDB } from '@waggle/core';
import { createInMemoryMemoryStore, PERSONAL_SCOPE, workspaceScope } from '@waggle/memory-contract';
import { importLegacyMemories, mapFrame } from '../../src/local/legacy-memory-import.js';
const dirs: string[] = [];
function tempDir(): string {
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'waggle-legacy-import-'));
dirs.push(dir);
return dir;
}
afterEach(() => {
for (const dir of dirs.splice(0)) fs.rmSync(dir, { recursive: true, force: true });
});
/** Writes a minimal legacy .mind file with the given frames. */
function legacyMind(file: string, rows: Array<{ content: string; importance?: string; source?: string }>): void {
const db = new Database(file);
db.exec(`CREATE TABLE memory_frames (
id INTEGER PRIMARY KEY AUTOINCREMENT, content TEXT NOT NULL,
importance TEXT NOT NULL DEFAULT 'normal', source TEXT, created_at TEXT
)`);
const insert = db.prepare('INSERT INTO memory_frames (content, importance, source, created_at) VALUES (?, ?, ?, ?)');
for (const row of rows) insert.run(row.content, row.importance ?? 'normal', row.source ?? 'user_stated', '2026-05-01 10:00:00');
db.close();
}
describe('mapFrame', () => {
it('turns content prefixes into kinds and strips lane headers', () => {
expect(mapFrame({ content: '[mind-fact]\nLives in Novi Sad', importance: 'normal', source: 'system' }))
.toEqual({ text: 'Lives in Novi Sad', kind: 'fact', importance: 'normal', source: 'system' });
expect(mapFrame({ content: '[mind-profile Mila]\nDesigner', importance: 'normal', source: null })?.kind).toBe('profile');
expect(mapFrame({ content: '[mind-event]\n[2026-05-01] Moved', importance: 'normal', source: null })?.kind).toBe('event');
expect(mapFrame({ content: 'Decision: use Postgres', importance: 'important', source: 'user_stated' })?.kind).toBe('decision');
expect(mapFrame({ content: 'User preference: short answers', importance: 'normal', source: 'user_stated' }))
.toMatchObject({ text: 'User preference: short answers', kind: 'preference' });
expect(mapFrame({ content: 'Plain fact', importance: 'normal', source: 'weird' })?.source).toBe('import');
});
it('drops deprecated frames, raw dialogue turns and empty text', () => {
expect(mapFrame({ content: 'Old', importance: 'deprecated', source: null })).toBeNull();
expect(mapFrame({ content: '[mind-rawturn conv=1 turn=2 speaker=user]\nhello', importance: 'normal', source: null })).toBeNull();
expect(mapFrame({ content: ' ', importance: 'normal', source: null })).toBeNull();
});
});
describe('importLegacyMemories', () => {
it('copies each scope once and leaves the legacy files unchanged', async () => {
const dir = tempDir();
const personal = path.join(dir, 'personal.mind');
const workspace = path.join(dir, 'workspace.mind');
legacyMind(personal, [
{ content: 'User preference: answers in Russian' },
{ content: 'Obsolete', importance: 'deprecated' },
]);
legacyMind(workspace, [{ content: 'Decision: ship on Friday', importance: 'important' }]);
const before = [fs.readFileSync(personal), fs.readFileSync(workspace)];
const memory = createInMemoryMemoryStore();
const waggleDb = new WaggleDB(path.join(dir, 'waggle.db'));
const sources = [
{ path: personal, scope: PERSONAL_SCOPE },
{ path: workspace, scope: workspaceScope('ws1') },
{ path: path.join(dir, 'missing.mind'), scope: workspaceScope('ws2') },
];
const reports = await importLegacyMemories(memory, waggleDb, sources);
expect(reports.map((r) => [r.scope, r.status, r.copied, r.skipped])).toEqual([
['personal', 'imported', 1, 1],
['workspace:ws1', 'imported', 1, 0],
['workspace:ws2', 'missing', 0, 0],
]);
const personalMemories = await memory.list({ scopes: [PERSONAL_SCOPE] });
expect(personalMemories).toMatchObject([
{ text: 'User preference: answers in Russian', kind: 'preference', origin: 'legacy-import' },
]);
expect(personalMemories[0]!.metadata).toEqual({ legacy_created_at: '2026-05-01 10:00:00' });
expect(await memory.list({ scopes: [workspaceScope('ws1')] })).toMatchObject([
{ text: 'Decision: ship on Friday', kind: 'decision', importance: 'important' },
]);
const again = await importLegacyMemories(memory, waggleDb, sources);
expect(again.map((r) => r.status)).toEqual(['already-imported', 'already-imported', 'already-imported']);
expect(await memory.count({ scopes: [PERSONAL_SCOPE, workspaceScope('ws1')] })).toBe(2);
expect([fs.readFileSync(personal), fs.readFileSync(workspace)]).toEqual(before);
waggleDb.close();
});
it('resumes after a cancelled copy', async () => {
const dir = tempDir();
const personal = path.join(dir, 'personal.mind');
legacyMind(personal, [{ content: 'First fact' }, { content: 'Second fact' }, { content: 'Third fact' }]);
const memory = createInMemoryMemoryStore();
const waggleDb = new WaggleDB(path.join(dir, 'waggle.db'));
let calls = 0;
const cancelled = await importLegacyMemories(
memory, waggleDb, [{ path: personal, scope: PERSONAL_SCOPE }], undefined, () => ++calls > 2,
);
expect(cancelled[0]!.copied).toBe(1);
const resumed = await importLegacyMemories(memory, waggleDb, [{ path: personal, scope: PERSONAL_SCOPE }]);
expect(resumed[0]).toMatchObject({ status: 'imported', copied: 2 });
expect((await memory.list({ scopes: [PERSONAL_SCOPE] })).map((r) => r.text).sort())
.toEqual(['First fact', 'Second fact', 'Third fact']);
waggleDb.close();
});
});

View File

@@ -329,15 +329,4 @@ describe('Memory Center routes', () => {
expect((await server.inject({ method: 'POST', url: '/api/memory/suppression/allow', payload: {} })).statusCode).toBe(404);
expect((await server.inject({ method: 'GET', url: `/api/memory/${mem.id}/source` })).statusCode).toBe(404);
});
it('shows the original creation time of a memory copied from the legacy engine', async () => {
const [change] = await memory.remember({
scope: PERSONAL_SCOPE,
text: 'Legacy memory.',
origin: 'legacy-import',
metadata: { legacy_created_at: '2025-03-01 10:20:30' },
});
const res = await server.inject({ method: 'GET', url: `/api/memory/${change!.record.id}` });
expect(res.json().createdAt).toBe('2025-03-01T10:20:30.000Z');
});
});