oleg.m/waggle-os
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

moving
Some checks failed
Installer Smoke / installer-smoke (push) Has been cancelled
Details

Browse Source
This commit is contained in:
Oleg Maslov
2026-09-02 10:10:29 +02:00
commit 0c3e2ead3b
3841 changed files with 970576 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

200
packages/hive-mind-wiki-compiler/LICENSE Normal file
View File

@@ -0,0 +1,200 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to the Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by the Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding any notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. Please also get an appropriate
"Contributors" file going in your project.
Copyright 2026 Egzakta Group d.o.o.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

36
packages/hive-mind-wiki-compiler/NOTICE Normal file
View File

@@ -0,0 +1,36 @@
hive-mind
Copyright 2026 Egzakta Group d.o.o.
This product includes software developed by Egzakta Group d.o.o.
(https://egzakta.com).
Portions of this software were extracted from Waggle OS
(https://waggle-os.ai) — specifically the memory substrate layers
(frames, hybrid search, knowledge graph, identity, awareness, sessions),
the harvest pipeline, and the wiki compiler.
The following Waggle OS subsystems are intentionally NOT part of this
distribution and remain proprietary to Egzakta Group:
- EU AI Act compliance reporting and audit trail (compliance/*)
- Agent runtime, personas, and behavioral specifications (packages/agent/*)
- Self-evolution engine: GEPA iterative optimization, EvolveSchema,
execution trace storage, evolution run storage, improvement signals
- Encrypted secret vault (mind/vault.ts)
- Tier and billing system (Stripe integration, feature gating)
- Tauri desktop shell and Waggle web UI
- Multi-agent coordination (WaggleDance, subagent orchestration)
See EXTRACTION.md in the repository root for the full source mapping.
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

48
packages/hive-mind-wiki-compiler/README.md Normal file
View File

@@ -0,0 +1,48 @@
# @hive-mind/wiki-compiler
[![npm](https://img.shields.io/npm/v/@hive-mind/wiki-compiler.svg)](https://www.npmjs.com/package/@hive-mind/wiki-compiler)
[![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](./LICENSE)
Compile memory frames and knowledge-graph entities from [`@hive-mind/core`](https://www.npmjs.com/package/@hive-mind/core) into interlinked wiki pages using any LLM provider.
## What it produces
- **Entity pages** — one per significant entity (person, project, technology)
- **Concept pages** — synthesized for abstract concepts and recurring themes
- **Synthesis pages** — cross-cutting pages connecting entities and concepts
- **Health reports** — data-quality analysis, gap detection
Compilation is **incremental**: only new or modified frames trigger recompilation.
## Install
```bash
npm install @hive-mind/wiki-compiler @hive-mind/core
```
Bring your own LLM provider — Anthropic SDK is an optional peer:
```bash
npm install @anthropic-ai/sdk
```
## Quick use
```ts
import { MindDB } from '@hive-mind/core';
import { WikiCompiler } from '@hive-mind/wiki-compiler';
const db = new MindDB('~/.hive-mind/my-project.mind');
const compiler = new WikiCompiler(db, { provider: 'anthropic' });
await compiler.compile({ incremental: true });
const report = await compiler.health();
```
## License
Apache 2.0. See [LICENSE](./LICENSE) and [NOTICE](./NOTICE).
## Part of hive-mind
Full docs and architecture at the [monorepo root](https://github.com/marolinik/hive-mind).

59
packages/hive-mind-wiki-compiler/package.json Normal file
View File

@@ -0,0 +1,59 @@
{
"name": "@waggle/hive-mind-wiki-compiler",
"version": "0.1.0",
"description": "Compile memory frames and knowledge graph entities into interlinked wiki pages via LLM synthesis",
"license": "Apache-2.0",
"type": "module",
"main": "dist/index.js",
"exports": {
".": {
"types": "./dist/index.d.ts",
"import": "./dist/index.js"
}
},
"scripts": {
"build": "tsc --build",
"clean": "tsc --build --clean",
"test": "node ../../node_modules/vitest/vitest.mjs run --root ../.. --config vitest.config.ts packages/hive-mind-wiki-compiler/src"
},
"dependencies": {
"@waggle/hive-mind-core": "*"
},
"peerDependencies": {
"@anthropic-ai/sdk": "^0.24.0"
},
"peerDependenciesMeta": {
"@anthropic-ai/sdk": {
"optional": true
}
},
"devDependencies": {
"@anthropic-ai/sdk": "^0.24.0"
},
"publishConfig": {
"access": "public"
},
"repository": {
"type": "git",
"url": "git+https://github.com/marolinik/waggle-os.git",
"directory": "packages/hive-mind-wiki-compiler"
},
"homepage": "https://github.com/marolinik/waggle-os/tree/main/packages/hive-mind-wiki-compiler#readme",
"bugs": {
"url": "https://github.com/marolinik/waggle-os/issues"
},
"author": "Egzakta Group d.o.o. <hello@egzakta.com> (https://egzakta.com)",
"keywords": [
"wiki",
"memory",
"knowledge-graph",
"llm",
"ai",
"synthesis",
"hive-mind"
],
"engines": {
"node": ">=20"
},
"types": "dist/index.d.ts"
}

212
packages/hive-mind-wiki-compiler/src/compiler.test.ts Normal file
View File

@@ -0,0 +1,212 @@
import { afterEach, beforeEach, describe, expect, it } from 'vitest';
import { existsSync, readFileSync, rmSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import {
MindDB,
FrameStore,
HybridSearch,
KnowledgeGraph,
createEmbeddingProvider,
type EmbeddingProviderInstance,
} from '@waggle/hive-mind-core';
import { WikiCompiler } from './compiler.js';
import { CompilationState } from './state.js';
import type { LLMSynthesizeFn } from './types.js';
/**
* A tiny deterministic LLM stub. The prompt already embeds all frame
* content; the stub just echoes an identifying slice back so the test
* can assert that the page was actually composed through the
* synthesize() callback rather than bypassed.
*/
const stubSynthesize: LLMSynthesizeFn = async (prompt) => {
const topic = prompt.match(/about "([^"]+)"/) ?? prompt.match(/discuss "([^"]+)"/);
return `## Summary\nStub synthesis of ${topic?.[1] ?? 'unknown topic'} based on the provided frames.\n\n## Key Facts\n- synthesized by stub`;
};
describe('WikiCompiler', () => {
let dbPath: string;
let db: MindDB;
let frames: FrameStore;
let kg: KnowledgeGraph;
let embedder: EmbeddingProviderInstance;
let search: HybridSearch;
let state: CompilationState;
let compiler: WikiCompiler;
beforeEach(async () => {
dbPath = join(tmpdir(), `hmind-wiki-compiler-${Date.now()}-${Math.random()}.mind`);
db = new MindDB(dbPath);
// Bootstrap session row so frames can reference a gop_id.
db.getDatabase().prepare(
"INSERT INTO sessions (gop_id, status, started_at) VALUES ('g-test', 'active', datetime('now'))",
).run();
frames = new FrameStore(db);
kg = new KnowledgeGraph(db);
embedder = await createEmbeddingProvider({ provider: 'mock' });
search = new HybridSearch(db, embedder);
state = new CompilationState(db);
compiler = new WikiCompiler(kg, frames, search, state, {
synthesize: stubSynthesize,
minFramesPerPage: 2,
maxFramesPerCall: 10,
});
});
afterEach(() => {
db.close();
if (existsSync(dbPath)) rmSync(dbPath);
for (const suffix of ['-shm', '-wal']) {
if (existsSync(dbPath + suffix)) rmSync(dbPath + suffix);
}
});
it('compileEntityPage returns null below minFramesPerPage', async () => {
const alice = kg.createEntity('person', 'Alice', { role: 'engineer' });
// Only one frame mentioning Alice — below minFramesPerPage=2
await frames.createIFrame('g-test', 'Alice joined the team this week');
const page = await compiler.compileEntityPage(alice);
expect(page).toBeNull();
});
it('compileEntityPage composes a markdown page with frontmatter, body, and relations', async () => {
const alice = kg.createEntity('person', 'Alice', { role: 'engineer' });
const acme = kg.createEntity('organization', 'Acme Corp', {});
kg.createRelation(alice.id, acme.id, 'works_at', 0.9);
await frames.createIFrame('g-test', 'Alice joined the team this week', 'important', 'user_stated');
await frames.createIFrame('g-test', 'Alice leads the search work', 'normal', 'user_stated');
await frames.createIFrame('g-test', 'Alice and Bob paired on the migration', 'normal', 'user_stated');
const page = await compiler.compileEntityPage(alice);
expect(page).not.toBeNull();
expect(page!.slug).toBe('alice');
expect(page!.frontmatter.type).toBe('entity');
expect(page!.frontmatter.entity_type).toBe('person');
expect(page!.frontmatter.sources).toBeGreaterThanOrEqual(2);
expect(page!.frontmatter.related_entities).toContain('Acme Corp');
expect(page!.markdown).toContain('# Alice');
expect(page!.markdown).toContain('Stub synthesis of Alice');
expect(page!.markdown).toContain('type: entity');
expect(page!.contentHash).toMatch(/^[a-f0-9]{16}$/);
});
it('compileConceptPage includes related entities discovered via KG search', async () => {
kg.createEntity('concept', 'Observability', {});
kg.createEntity('technology', 'Prometheus', {});
kg.createEntity('technology', 'Grafana', {});
await frames.createIFrame('g-test', 'Observability is about logs and traces', 'normal', 'user_stated');
await frames.createIFrame('g-test', 'Observability: Prometheus scrapes metrics', 'normal', 'user_stated');
await frames.createIFrame('g-test', 'Observability dashboards in Grafana help triage', 'normal', 'user_stated');
const page = await compiler.compileConceptPage('Observability');
expect(page).not.toBeNull();
expect(page!.frontmatter.type).toBe('concept');
expect(page!.slug).toBe('observability');
expect(page!.markdown).toContain('# Observability');
});
it('compileSynthesisPage returns null when only one source discusses the topic', async () => {
await frames.createIFrame('g-test', 'Rust migration plan v1', 'normal', 'import');
await frames.createIFrame('g-test', 'Rust migration plan v2', 'normal', 'import');
const page = await compiler.compileSynthesisPage('Rust migration');
expect(page).toBeNull();
});
it('compileSynthesisPage produces a page when multiple sources discuss the topic', async () => {
await frames.createIFrame('g-test', 'Rust migration feasibility studied (imported)', 'normal', 'import');
await frames.createIFrame('g-test', 'Rust migration POC begun per user', 'normal', 'user_stated');
await frames.createIFrame('g-test', 'Rust migration: agent suggested crates', 'normal', 'agent_inferred');
const page = await compiler.compileSynthesisPage('Rust migration');
expect(page).not.toBeNull();
expect(page!.frontmatter.type).toBe('synthesis');
expect(page!.slug).toBe('synthesis-rust-migration');
expect(page!.frontmatter.related_entities.length).toBeGreaterThanOrEqual(2);
});
it('compileIndex produces a nested markdown catalog of all stored page types', () => {
state.upsertPage('alice', 'entity', 'Alice', 'h-1', [1], 2, '# Alice');
state.upsertPage('bob', 'entity', 'Bob', 'h-2', [2], 2, '# Bob');
state.upsertPage('obs', 'concept', 'Observability', 'h-3', [3], 4, '# Obs');
state.upsertPage('synthesis-rust', 'synthesis', 'Synthesis: Rust migration', 'h-4', [4], 3, '# S');
const idx = compiler.compileIndex();
expect(idx.slug).toBe('index');
expect(idx.markdown).toContain('# Wiki Index');
expect(idx.markdown).toContain('## Entities');
expect(idx.markdown).toContain('[[Alice]]');
expect(idx.markdown).toContain('[[Bob]]');
expect(idx.markdown).toContain('## Concepts');
expect(idx.markdown).toContain('[[Observability]]');
expect(idx.markdown).toContain('## Cross-Source Synthesis');
expect(idx.markdown).toContain('[[Synthesis: Rust migration]]');
});
it('compileHealth flags missing pages for entities without coverage', async () => {
const alice = kg.createEntity('person', 'Alice', {});
const acme = kg.createEntity('organization', 'Acme Corp', {});
kg.createRelation(alice.id, acme.id, 'works_at', 0.9);
const health = compiler.compileHealth();
expect(health.totalEntities).toBe(2);
const missingAlice = health.issues.find(
(i) => i.type === 'missing_page' && i.entity === 'Alice',
);
expect(missingAlice).toBeDefined();
expect(missingAlice!.severity === 'medium' || missingAlice!.severity === 'high').toBe(true);
expect(health.dataQualityScore).toBeLessThanOrEqual(100);
});
it('compile() full run exercises entity + concept + synthesis + index + watermark', async () => {
// Two distinct entities with enough frames across multiple sources
const alice = kg.createEntity('person', 'Alice', { role: 'engineer' });
kg.createEntity('person', 'Bob', { role: 'designer' });
const acme = kg.createEntity('organization', 'Acme Corp', {});
kg.createRelation(alice.id, acme.id, 'works_at', 0.9);
for (let i = 0; i < 4; i++) {
await frames.createIFrame('g-test', `Alice pushed a PR about search #${i}`, 'normal', 'user_stated');
}
for (let i = 0; i < 3; i++) {
await frames.createIFrame('g-test', `Bob reviewed the design doc #${i}`, 'normal', 'import');
}
await frames.createIFrame('g-test', 'Alice and Bob paired on the refactor (agent observed)', 'normal', 'agent_inferred');
const result = await compiler.compile({ concepts: ['search'] });
expect(result.pagesCreated + result.pagesUpdated).toBeGreaterThanOrEqual(1);
expect(result.durationMs).toBeGreaterThanOrEqual(0);
expect(result.watermark.lastFrameId).toBeGreaterThan(0);
// State should have at least an index page now
const indexPage = state.getPage('index');
expect(indexPage?.name).toBe('Wiki Index');
// Watermark persisted
const w = state.getWatermark();
expect(w.lastFrameId).toBeGreaterThan(0);
});
it('exportToMarkdown returns slug→markdown bundles including placeholder for empty pages', () => {
state.upsertPage('alice', 'entity', 'Alice', 'h-1', [1], 2, '# Alice\n\nbody');
state.upsertPage('bob', 'entity', 'Bob', 'h-2', [2], 1, ''); // empty markdown
const bundle = compiler.exportToMarkdown();
expect(bundle.get('alice')).toContain('# Alice');
expect(bundle.get('bob')).toContain('no content compiled yet');
});
it('exportToDirectory writes one .md file per page into the target dir', async () => {
state.upsertPage('alice', 'entity', 'Alice', 'h-1', [1], 2, '# Alice\n\nbody');
state.upsertPage('bob', 'entity', 'Bob', 'h-2', [2], 2, '# Bob\n\nbody');
const outDir = join(tmpdir(), `hmind-wiki-export-${Date.now()}`);
const { filesWritten } = await compiler.exportToDirectory(outDir);
expect(filesWritten).toBe(2);
expect(readFileSync(join(outDir, 'alice.md'), 'utf-8')).toContain('# Alice');
expect(readFileSync(join(outDir, 'bob.md'), 'utf-8')).toContain('# Bob');
rmSync(outDir, { recursive: true, force: true });
});
});

555
packages/hive-mind-wiki-compiler/src/compiler.ts Normal file
View File

@@ -0,0 +1,555 @@
/**
* Wiki Compiler — compiles memory frames + KG into interlinked wiki pages.
*
* Core compilation functions:
* - compileEntityPage() — entity → frames + relations → LLM → markdown
* - compileConceptPage() — topic → search → LLM → markdown
* - compileSynthesisPage() — cross-source pattern detection
* - compileIndex() — navigable catalog
* - compileHealth() — contradictions, gaps, data quality
*/
import type {
KnowledgeGraph,
FrameStore,
HybridSearch,
Entity,
} from '@waggle/hive-mind-core';
import type {
WikiPage,
WikiPageType,
CompilerConfig,
CompilationResult,
HealthReport,
HealthIssue,
} from './types.js';
import { CompilationState, contentHash } from './state.js';
import { entityPagePrompt, conceptPagePrompt, synthesisPagePrompt } from './prompts.js';
function slugify(name: string): string {
return name
.toLowerCase()
.replace(/[^a-z0-9]+/g, '-')
.replace(/^-|-$/g, '')
.slice(0, 80);
}
function buildFrontmatter(
type: WikiPageType,
name: string,
frameIds: number[],
relatedEntities: string[],
confidence: number,
entityType?: string,
): string {
const lines = [
'---',
`type: ${type}`,
...(entityType ? [`entity_type: ${entityType}`] : []),
`name: "${name.replace(/"/g, '\\"')}"`,
`confidence: ${confidence.toFixed(2)}`,
`sources: ${frameIds.length}`,
`last_compiled: ${new Date().toISOString()}`,
`frame_ids: [${frameIds.join(', ')}]`,
`related_entities: [${relatedEntities.map(e => `"${e}"`).join(', ')}]`,
'---',
];
return lines.join('\n');
}
export class WikiCompiler {
private kg: KnowledgeGraph;
private frames: FrameStore;
private search: HybridSearch;
private state: CompilationState;
private config: Required<CompilerConfig>;
constructor(
kg: KnowledgeGraph,
frames: FrameStore,
search: HybridSearch,
state: CompilationState,
config: CompilerConfig,
) {
this.kg = kg;
this.frames = frames;
this.search = search;
this.state = state;
this.config = {
synthesize: config.synthesize,
outputDir: config.outputDir ?? 'wiki',
minFramesPerPage: config.minFramesPerPage ?? 2,
maxFramesPerCall: config.maxFramesPerCall ?? 30,
minConfidence: config.minConfidence ?? 0.3,
};
}
// ── Entity Page ───────────────────────────────────────────────
async compileEntityPage(entity: Entity): Promise<WikiPage | null> {
// Gather frames mentioning this entity
const searchResults = await this.search.search(entity.name, {
limit: this.config.maxFramesPerCall,
});
const frameData = searchResults.map(r => ({
id: r.frame.id,
content: r.frame.content,
created_at: r.frame.created_at,
}));
if (frameData.length < this.config.minFramesPerPage) {
return null; // Not enough data for a page
}
// Gather relations
const outRels = this.kg.getRelationsFrom(entity.id);
const inRels = this.kg.getRelationsTo(entity.id);
const relations: { target: string; relationType: string; confidence: number }[] = [];
const relatedEntities: string[] = [];
for (const rel of outRels) {
const target = this.kg.getEntity(rel.target_id);
if (target && target.valid_to === null) {
relations.push({ target: target.name, relationType: rel.relation_type, confidence: rel.confidence });
relatedEntities.push(target.name);
}
}
for (const rel of inRels) {
const source = this.kg.getEntity(rel.source_id);
if (source && source.valid_to === null) {
relations.push({ target: source.name, relationType: `${rel.relation_type} (inbound)`, confidence: rel.confidence });
if (!relatedEntities.includes(source.name)) {
relatedEntities.push(source.name);
}
}
}
// Synthesize via LLM
const prompt = entityPagePrompt(entity.name, entity.entity_type, frameData, relations);
const body = await this.config.synthesize(prompt);
const frameIds = frameData.map(f => f.id);
const confidence = frameData.length > 5 ? 0.9 : frameData.length > 2 ? 0.7 : 0.5;
const slug = slugify(entity.name);
const frontmatter = buildFrontmatter('entity', entity.name, frameIds, relatedEntities, confidence, entity.entity_type);
const markdown = `${frontmatter}\n\n# ${entity.name}\n\n${body}`;
return {
slug,
frontmatter: {
type: 'entity',
entity_type: entity.entity_type,
name: entity.name,
confidence,
sources: frameIds.length,
last_compiled: new Date().toISOString(),
frame_ids: frameIds,
related_entities: relatedEntities,
},
markdown,
contentHash: contentHash(markdown),
};
}
// ── Concept Page ──────────────────────────────────────────────
async compileConceptPage(conceptName: string): Promise<WikiPage | null> {
const searchResults = await this.search.search(conceptName, {
limit: this.config.maxFramesPerCall,
});
const frameData = searchResults.map(r => ({
id: r.frame.id,
content: r.frame.content,
created_at: r.frame.created_at,
}));
if (frameData.length < this.config.minFramesPerPage) {
return null;
}
// Find related entities via KG
const entityResults = this.kg.searchEntities(conceptName, 10);
const relatedEntities = entityResults.map(e => e.name);
const prompt = conceptPagePrompt(conceptName, frameData, relatedEntities);
const body = await this.config.synthesize(prompt);
const frameIds = frameData.map(f => f.id);
const confidence = frameData.length > 5 ? 0.85 : 0.6;
const slug = slugify(conceptName);
const frontmatter = buildFrontmatter('concept', conceptName, frameIds, relatedEntities, confidence);
const markdown = `${frontmatter}\n\n# ${conceptName}\n\n${body}`;
return {
slug,
frontmatter: {
type: 'concept',
name: conceptName,
confidence,
sources: frameIds.length,
last_compiled: new Date().toISOString(),
frame_ids: frameIds,
related_entities: relatedEntities,
},
markdown,
contentHash: contentHash(markdown),
};
}
// ── Synthesis Page ────────────────────────────────────────────
async compileSynthesisPage(topic: string): Promise<WikiPage | null> {
const searchResults = await this.search.search(topic, {
limit: this.config.maxFramesPerCall * 2, // more context for synthesis
});
// Group frames by source to detect cross-source patterns
const bySource = new Map<string, typeof searchResults>();
for (const r of searchResults) {
const source = r.frame.source ?? 'unknown';
let group = bySource.get(source);
if (!group) {
group = [];
bySource.set(source, group);
}
group.push(r);
}
// Need frames from at least 2 sources for synthesis
if (bySource.size < 2) {
return null;
}
const crossSourceFrames = searchResults.slice(0, this.config.maxFramesPerCall).map(r => ({
id: r.frame.id,
content: r.frame.content,
source: r.frame.source ?? 'unknown',
created_at: r.frame.created_at,
}));
const prompt = synthesisPagePrompt(topic, crossSourceFrames);
const body = await this.config.synthesize(prompt);
const frameIds = crossSourceFrames.map(f => f.id);
const confidence = bySource.size > 3 ? 0.85 : 0.65;
const slug = `synthesis-${slugify(topic)}`;
const sources = Array.from(bySource.keys());
const frontmatter = buildFrontmatter('synthesis', `Synthesis: ${topic}`, frameIds, sources, confidence);
const markdown = `${frontmatter}\n\n# Synthesis: ${topic}\n\n${body}`;
return {
slug,
frontmatter: {
type: 'synthesis',
name: `Synthesis: ${topic}`,
confidence,
sources: frameIds.length,
last_compiled: new Date().toISOString(),
frame_ids: frameIds,
related_entities: sources,
},
markdown,
contentHash: contentHash(markdown),
};
}
// ── Index Page ────────────────────────────────────────────────
compileIndex(): WikiPage {
const allPages = this.state.getAllPages();
const entityPages = allPages.filter(p => p.pageType === 'entity');
const conceptPages = allPages.filter(p => p.pageType === 'concept');
const synthesisPages = allPages.filter(p => p.pageType === 'synthesis');
const lines: string[] = [
'# Wiki Index',
'',
`*${allPages.length} pages compiled — last updated ${new Date().toISOString().slice(0, 10)}*`,
'',
];
if (entityPages.length > 0) {
lines.push('## Entities', '');
for (const p of entityPages) {
lines.push(`- [[${p.name}]] — ${p.sourceCount} sources (${p.compiledAt.slice(0, 10)})`);
}
lines.push('');
}
if (conceptPages.length > 0) {
lines.push('## Concepts', '');
for (const p of conceptPages) {
lines.push(`- [[${p.name}]] — ${p.sourceCount} sources (${p.compiledAt.slice(0, 10)})`);
}
lines.push('');
}
if (synthesisPages.length > 0) {
lines.push('## Cross-Source Synthesis', '');
for (const p of synthesisPages) {
lines.push(`- [[${p.name}]] — ${p.sourceCount} sources (${p.compiledAt.slice(0, 10)})`);
}
lines.push('');
}
const markdown = lines.join('\n');
return {
slug: 'index',
frontmatter: {
type: 'index',
name: 'Wiki Index',
confidence: 1.0,
sources: allPages.length,
last_compiled: new Date().toISOString(),
frame_ids: [],
related_entities: [],
},
markdown,
contentHash: contentHash(markdown),
};
}
// ── Health Report ─────────────────────────────────────────────
compileHealth(): HealthReport {
const issues: HealthIssue[] = [];
const allPages = this.state.getAllPages();
const entityCount = this.kg.getEntityCount();
const frameStats = this.frames.getStats();
// Check for orphan entities (KG entities with no wiki page)
const entities = this.kg.getEntities(1000);
const pageNames = new Set(allPages.map(p => p.name.toLowerCase()));
for (const entity of entities) {
if (!pageNames.has(entity.name.toLowerCase())) {
// Check if entity has enough frames to justify a page
const rels = this.kg.getRelationsFrom(entity.id);
if (rels.length > 0 || entity.entity_type === 'person' || entity.entity_type === 'project') {
issues.push({
type: 'missing_page',
severity: rels.length > 2 ? 'high' : 'medium',
description: `Entity "${entity.name}" (${entity.entity_type}) has no wiki page`,
entity: entity.name,
suggestion: `Run: compileEntityPage("${entity.name}")`,
});
}
}
}
// Check for weak pages (few sources)
for (const page of allPages) {
if (page.sourceCount < 2 && page.pageType !== 'index') {
issues.push({
type: 'weak_confidence',
severity: 'low',
description: `Page "${page.name}" has only ${page.sourceCount} source(s)`,
entity: page.name,
suggestion: 'Gather more data about this topic',
});
}
}
// Check for orphan entities in KG (no relations at all)
for (const entity of entities) {
const outRels = this.kg.getRelationsFrom(entity.id);
const inRels = this.kg.getRelationsTo(entity.id);
if (outRels.length === 0 && inRels.length === 0) {
issues.push({
type: 'orphan_entity',
severity: 'low',
description: `Entity "${entity.name}" (${entity.entity_type}) has no relations`,
entity: entity.name,
suggestion: 'Consider adding relations or retiring this entity',
});
}
}
// Data quality score
const hasEntities = entityCount > 0 ? 20 : 0;
const hasFrames = frameStats.total > 10 ? 20 : frameStats.total > 0 ? 10 : 0;
const hasPages = allPages.length > 5 ? 20 : allPages.length > 0 ? 10 : 0;
const lowIssues = issues.filter(i => i.severity === 'high').length;
const issueDeduction = Math.min(40, lowIssues * 10);
const dataQualityScore = Math.max(0, hasEntities + hasFrames + hasPages + 40 - issueDeduction);
return {
totalEntities: entityCount,
totalFrames: frameStats.total,
totalPages: allPages.length,
issues,
dataQualityScore,
compiledAt: new Date().toISOString(),
};
}
// ── Full Compilation ──────────────────────────────────────────
async compile(options?: { incremental?: boolean; concepts?: string[] }): Promise<CompilationResult> {
const startTime = Date.now();
const incremental = options?.incremental ?? true;
const watermark = this.state.getWatermark();
let pagesCreated = 0;
let pagesUpdated = 0;
let pagesUnchanged = 0;
const entityPageNames: string[] = [];
const conceptPageNames: string[] = [];
const synthesisPageNames: string[] = [];
// 1. Compile entity pages for all significant entities
const entities = this.kg.getEntities(200);
for (const entity of entities) {
// Skip entities we've already compiled unless new frames exist
if (incremental && watermark.lastFrameId > 0) {
const existingPage = this.state.getPage(slugify(entity.name));
if (existingPage) {
// Check if new frames mention this entity
const newFrames = this.state.getFramesSince(watermark.lastFrameId, 100);
const mentionsEntity = newFrames.some(f =>
f.content.toLowerCase().includes(entity.name.toLowerCase()),
);
if (!mentionsEntity) {
pagesUnchanged++;
continue;
}
}
}
const page = await this.compileEntityPage(entity);
if (page) {
const result = this.state.upsertPage(
page.slug, 'entity', entity.name,
page.contentHash, page.frontmatter.frame_ids, page.frontmatter.sources,
page.markdown,
);
if (result.action === 'created') pagesCreated++;
else if (result.action === 'updated') pagesUpdated++;
else pagesUnchanged++;
entityPageNames.push(entity.name);
}
}
// 2. Compile concept pages (user-specified or auto-detected)
const concepts = options?.concepts ?? this.detectConcepts(entities);
for (const concept of concepts) {
const page = await this.compileConceptPage(concept);
if (page) {
const result = this.state.upsertPage(
page.slug, 'concept', concept,
page.contentHash, page.frontmatter.frame_ids, page.frontmatter.sources,
page.markdown,
);
if (result.action === 'created') pagesCreated++;
else if (result.action === 'updated') pagesUpdated++;
else pagesUnchanged++;
conceptPageNames.push(concept);
}
}
// 3. Compile synthesis pages for topics with cross-source data
for (const concept of concepts) {
const page = await this.compileSynthesisPage(concept);
if (page) {
const result = this.state.upsertPage(
page.slug, 'synthesis', `Synthesis: ${concept}`,
page.contentHash, page.frontmatter.frame_ids, page.frontmatter.sources,
page.markdown,
);
if (result.action === 'created') pagesCreated++;
else if (result.action === 'updated') pagesUpdated++;
else pagesUnchanged++;
synthesisPageNames.push(concept);
}
}
// 4. Compile index
const indexPage = this.compileIndex();
this.state.upsertPage('index', 'index', 'Wiki Index', indexPage.contentHash, [], 0, indexPage.markdown);
// 5. Update watermark
const maxFrameId = this.state.getMaxFrameId();
const totalCompiled = pagesCreated + pagesUpdated;
this.state.updateWatermark(maxFrameId, totalCompiled);
// 6. Health check
const health = this.compileHealth();
return {
pagesCreated,
pagesUpdated,
pagesUnchanged,
entityPages: entityPageNames,
conceptPages: conceptPageNames,
synthesisPages: synthesisPageNames,
healthIssues: health.issues.length,
watermark: { lastFrameId: maxFrameId, lastCompiledAt: new Date().toISOString(), pagesCompiled: totalCompiled },
durationMs: Date.now() - startTime,
};
}
// ── Helpers ───────────────────────────────────────────────────
/** Auto-detect concepts from entity types and common topics. */
private detectConcepts(entities: Entity[]): string[] {
const typeCounts = new Map<string, number>();
for (const e of entities) {
typeCounts.set(e.entity_type, (typeCounts.get(e.entity_type) ?? 0) + 1);
}
const concepts: string[] = [];
// Add entity types that have multiple entries as concepts
for (const [type, count] of typeCounts) {
if (count >= 3 && type !== 'concept') {
concepts.push(type);
}
}
// Add 'concept' type entities directly as concepts
const conceptEntities = entities.filter(e => e.entity_type === 'concept');
for (const e of conceptEntities) {
if (!concepts.includes(e.name)) {
concepts.push(e.name);
}
}
return concepts.slice(0, 20); // Cap at 20 concepts
}
/**
* Export all wiki pages as a flat markdown bundle.
* Returns a map of slug → markdown content, suitable for writing to disk.
*/
exportToMarkdown(): Map<string, string> {
const pages = this.state.getAllPages();
const result = new Map<string, string>();
for (const page of pages) {
result.set(page.slug, page.markdown || `# ${page.name}\n\n(no content compiled yet)`);
}
return result;
}
/**
* Export all wiki pages to a directory as individual .md files.
* Creates the directory if it doesn't exist.
*/
async exportToDirectory(dir: string): Promise<{ filesWritten: number }> {
const { mkdirSync, writeFileSync } = await import('node:fs');
mkdirSync(dir, { recursive: true });
const pages = this.exportToMarkdown();
let count = 0;
for (const [slug, markdown] of pages) {
writeFileSync(`${dir}/${slug}.md`, markdown, 'utf-8');
count++;
}
return { filesWritten: count };
}
}

38
packages/hive-mind-wiki-compiler/src/index.ts Normal file
View File

@@ -0,0 +1,38 @@
/**
* @waggle/hive-mind-wiki-compiler — LLM-powered knowledge synthesis.
*
* Takes a MindDB instance (from @waggle/hive-mind-core) and compiles memory
* frames plus knowledge-graph entities into interlinked wiki pages.
*
* Five page types:
* entity — per-entity synthesis (name, facts, timeline, relations,
* open questions, contradictions)
* concept — per-topic synthesis with related-entity back-links
* synthesis — cross-source pattern detection (agreements,
* contradictions, emergent insights)
* index — navigable catalog of all compiled pages
* health — data-quality report (gaps, orphans, weak confidence)
*
* See CompilerConfig for the LLM callback contract, and
* resolveSynthesizer() for the built-in provider chain (Anthropic
* Haiku → Ollama → echo fallback).
*/
export { WikiCompiler } from './compiler.js';
export { CompilationState, contentHash } from './state.js';
export { resolveSynthesizer } from './synthesizer.js';
export type { ResolvedSynthesizer, SynthesizerConfig } from './synthesizer.js';
export { entityPagePrompt, conceptPagePrompt, synthesisPagePrompt } from './prompts.js';
export type {
WikiPage,
WikiPageType,
WikiPageFrontmatter,
CompilationWatermark,
PageRecord,
CompilerConfig,
LLMSynthesizeFn,
CompilationResult,
HealthReport,
HealthIssue,
HealthIssueType,
} from './types.js';

94
packages/hive-mind-wiki-compiler/src/prompts.ts Normal file
View File

@@ -0,0 +1,94 @@
/**
* LLM prompts for wiki page compilation.
*/
export function entityPagePrompt(
entityName: string,
entityType: string,
frames: { id: number; content: string; created_at: string }[],
relations: { target: string; relationType: string; confidence: number }[],
): string {
const frameList = frames
.map(f => `[Frame #${f.id}, ${f.created_at}]: ${f.content}`)
.join('\n\n');
const relationList = relations.length > 0
? relations.map(r => `- ${r.target} (${r.relationType}, confidence: ${r.confidence})`).join('\n')
: 'No relations found.';
return `You are a wiki compiler. Synthesize the following memory frames about "${entityName}" (${entityType}) into a wiki page.
## Source Frames (${frames.length} total)
${frameList}
## Known Relations
${relationList}
## Instructions
1. Write a **Summary** section (2-3 sentences synthesizing all knowledge)
2. Write a **Key Facts** section with bullet points citing frame IDs (e.g., "from frame #42")
3. Write a **Timeline** section if temporal events are present (table: Date | Event | Source)
4. Write a **Relations** section listing connected entities with [[wiki links]]
5. Write an **Open Questions** section noting gaps or unresolved contradictions
6. If frames contradict each other, note it in a **Contradictions** section with confidence assessment
Be concise. Cite frame IDs for every claim. Use markdown formatting.
Output ONLY the page body (no frontmatter — that's added automatically).`;
}
export function conceptPagePrompt(
conceptName: string,
frames: { id: number; content: string; created_at: string }[],
relatedEntities: string[],
): string {
const frameList = frames
.map(f => `[Frame #${f.id}, ${f.created_at}]: ${f.content}`)
.join('\n\n');
const entityList = relatedEntities.length > 0
? relatedEntities.map(e => `- [[${e}]]`).join('\n')
: 'None identified.';
return `You are a wiki compiler. Synthesize the following memory frames about the concept "${conceptName}" into a wiki page.
## Source Frames (${frames.length} total)
${frameList}
## Related Entities
${entityList}
## Instructions
1. Write a **TL;DR** (2 sentences max)
2. Write a **What We Know** section synthesizing all frames
3. Write a **Sources & Evolution** section showing how understanding evolved over time
4. Write a **Related Topics** section with [[wiki links]]
5. Note any gaps or areas needing more data
Be concise. Cite frame IDs. Use markdown formatting.
Output ONLY the page body (no frontmatter).`;
}
export function synthesisPagePrompt(
topic: string,
crossSourceFrames: { id: number; content: string; source: string; created_at: string }[],
): string {
const frameList = crossSourceFrames
.map(f => `[Frame #${f.id}, source: ${f.source}, ${f.created_at}]: ${f.content}`)
.join('\n\n');
return `You are a wiki compiler performing cross-source synthesis. Multiple independent sources discuss "${topic}". Find patterns, agreements, and contradictions.
## Frames from Multiple Sources (${crossSourceFrames.length} total)
${frameList}
## Instructions
1. Write a **Cross-Source Summary** — what do multiple sources agree on?
2. Write a **Patterns** section — recurring themes across sources
3. Write a **Contradictions** section — where sources disagree (with frame IDs)
4. Write an **Insights** section — what can we conclude that no single source stated?
5. Write a **Confidence Assessment** — how reliable is this synthesis?
This is the most valuable page type. Focus on insights that emerge from combining sources.
Be concise. Cite frame IDs. Use markdown formatting.
Output ONLY the page body (no frontmatter).`;
}

132
packages/hive-mind-wiki-compiler/src/state.test.ts Normal file
View File

@@ -0,0 +1,132 @@
import { afterEach, beforeEach, describe, expect, it } from 'vitest';
import { existsSync, rmSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { MindDB } from '@waggle/hive-mind-core';
import { CompilationState, contentHash } from './state.js';
describe('CompilationState', () => {
let dbPath: string;
let db: MindDB;
let state: CompilationState;
beforeEach(() => {
dbPath = join(tmpdir(), `hmind-wiki-state-${Date.now()}-${Math.random()}.mind`);
db = new MindDB(dbPath);
state = new CompilationState(db);
});
afterEach(() => {
db.close();
if (existsSync(dbPath)) rmSync(dbPath);
for (const suffix of ['-shm', '-wal']) {
if (existsSync(dbPath + suffix)) rmSync(dbPath + suffix);
}
});
it('contentHash() is stable and returns a 16-char sha256 prefix', () => {
const a = contentHash('hello world');
const b = contentHash('hello world');
const c = contentHash('different');
expect(a).toBe(b);
expect(a).not.toBe(c);
expect(a).toHaveLength(16);
expect(a).toMatch(/^[a-f0-9]{16}$/);
});
it('getWatermark() returns zeros before first compile', () => {
const w = state.getWatermark();
expect(w.lastFrameId).toBe(0);
expect(w.pagesCompiled).toBe(0);
expect(w.lastCompiledAt).toBe('');
});
it('updateWatermark() persists and upserts on subsequent calls', () => {
state.updateWatermark(42, 5);
let w = state.getWatermark();
expect(w.lastFrameId).toBe(42);
expect(w.pagesCompiled).toBe(5);
expect(w.lastCompiledAt).toMatch(/\d{4}-\d{2}-\d{2}/);
state.updateWatermark(100, 12);
w = state.getWatermark();
expect(w.lastFrameId).toBe(100);
expect(w.pagesCompiled).toBe(12);
});
it('upsertPage() returns created / updated / unchanged by content hash', () => {
const first = state.upsertPage('slug-a', 'entity', 'Alice', 'hash-1', [1, 2], 2, '# Alice v1');
expect(first.action).toBe('created');
// Same content hash → unchanged
const same = state.upsertPage('slug-a', 'entity', 'Alice', 'hash-1', [1, 2], 2, '# Alice v1');
expect(same.action).toBe('unchanged');
// Different content hash → updated
const updated = state.upsertPage('slug-a', 'entity', 'Alice', 'hash-2', [1, 2, 3], 3, '# Alice v2');
expect(updated.action).toBe('updated');
});
it('getPage() / getAllPages() / getPagesByType() round-trip correctly', () => {
state.upsertPage('a', 'entity', 'Alice', 'h-a', [1], 1, '# A');
state.upsertPage('b', 'entity', 'Bob', 'h-b', [2], 1, '# B');
state.upsertPage('c', 'concept', 'Roadmap', 'h-c', [3], 1, '# C');
const a = state.getPage('a');
expect(a?.name).toBe('Alice');
expect(a?.contentHash).toBe('h-a');
expect(a?.markdown).toBe('# A');
const all = state.getAllPages();
expect(all).toHaveLength(3);
// Ordered by name
expect(all.map((p) => p.name)).toEqual(['Alice', 'Bob', 'Roadmap']);
const entities = state.getPagesByType('entity');
expect(entities.map((p) => p.name)).toEqual(['Alice', 'Bob']);
const concepts = state.getPagesByType('concept');
expect(concepts.map((p) => p.name)).toEqual(['Roadmap']);
});
it('deletePage() removes by slug and returns whether a row changed', () => {
state.upsertPage('a', 'entity', 'Alice', 'h', [], 0, '# A');
expect(state.deletePage('a')).toBe(true);
expect(state.deletePage('a')).toBe(false);
expect(state.getPage('a')).toBeUndefined();
});
it('getMaxFrameId() queries the memory_frames table (0 when empty)', () => {
expect(state.getMaxFrameId()).toBe(0);
// Insert a session + two frames directly against the db
const raw = db.getDatabase();
raw.prepare(
"INSERT INTO sessions (gop_id, status, started_at) VALUES ('g-1', 'active', datetime('now'))",
).run();
raw.prepare(
"INSERT INTO memory_frames (frame_type, gop_id, t, content) VALUES ('I', 'g-1', 0, 'first')",
).run();
raw.prepare(
"INSERT INTO memory_frames (frame_type, gop_id, t, content) VALUES ('I', 'g-1', 1, 'second')",
).run();
expect(state.getMaxFrameId()).toBe(2);
});
it('getFramesSince() returns only frames with id > watermark', () => {
const raw = db.getDatabase();
raw.prepare(
"INSERT INTO sessions (gop_id, status, started_at) VALUES ('g-1', 'active', datetime('now'))",
).run();
for (let i = 0; i < 5; i++) {
raw.prepare(
"INSERT INTO memory_frames (frame_type, gop_id, t, content) VALUES ('I', 'g-1', ?, ?)",
).run(i, `frame ${i}`);
}
const afterTwo = state.getFramesSince(2);
expect(afterTwo.map((f) => f.id)).toEqual([3, 4, 5]);
expect(afterTwo[0].content).toBe('frame 2');
});
});

160
packages/hive-mind-wiki-compiler/src/state.ts Normal file
View File

@@ -0,0 +1,160 @@
/**
* Compilation State Tracker — SQLite-backed watermarks and page hashes.
*
* Tracks what has been compiled and when, enabling incremental compilation.
* Uses the same MindDB instance as the memory system.
*/
import { createHash } from 'node:crypto';
import type { MindDB } from '@waggle/hive-mind-core';
import type { WikiPageType, CompilationWatermark, PageRecord } from './types.js';
const PAGES_TABLE = `
CREATE TABLE IF NOT EXISTS wiki_pages (
slug TEXT PRIMARY KEY,
page_type TEXT NOT NULL,
name TEXT NOT NULL,
content_hash TEXT NOT NULL,
markdown TEXT NOT NULL DEFAULT '',
frame_ids TEXT NOT NULL DEFAULT '[]',
compiled_at TEXT NOT NULL DEFAULT (datetime('now')),
source_count INTEGER NOT NULL DEFAULT 0
)`;
const WATERMARK_TABLE = `
CREATE TABLE IF NOT EXISTS wiki_watermark (
id INTEGER PRIMARY KEY CHECK (id = 1),
last_frame_id INTEGER NOT NULL DEFAULT 0,
last_compiled_at TEXT NOT NULL DEFAULT (datetime('now')),
pages_compiled INTEGER NOT NULL DEFAULT 0
)`;
export function contentHash(content: string): string {
return createHash('sha256').update(content).digest('hex').slice(0, 16);
}
export class CompilationState {
private db: MindDB;
constructor(db: MindDB) {
this.db = db;
this.ensureSchema();
}
private ensureSchema(): void {
const raw = this.db.getDatabase();
raw.prepare(PAGES_TABLE).run();
raw.prepare(WATERMARK_TABLE).run();
// Migration: add markdown column if missing (for databases created before v1.1)
try {
raw.prepare("SELECT markdown FROM wiki_pages LIMIT 0").get();
} catch {
raw.prepare("ALTER TABLE wiki_pages ADD COLUMN markdown TEXT NOT NULL DEFAULT ''").run();
}
}
// ── Watermark ─────────────────────────────────────────────────
getWatermark(): CompilationWatermark {
const raw = this.db.getDatabase();
const row = raw.prepare('SELECT * FROM wiki_watermark WHERE id = 1').get() as {
last_frame_id: number;
last_compiled_at: string;
pages_compiled: number;
} | undefined;
if (!row) {
return { lastFrameId: 0, lastCompiledAt: '', pagesCompiled: 0 };
}
return {
lastFrameId: row.last_frame_id,
lastCompiledAt: row.last_compiled_at,
pagesCompiled: row.pages_compiled,
};
}
updateWatermark(lastFrameId: number, pagesCompiled: number): void {
const raw = this.db.getDatabase();
raw.prepare(`
INSERT INTO wiki_watermark (id, last_frame_id, last_compiled_at, pages_compiled)
VALUES (1, ?, datetime('now'), ?)
ON CONFLICT(id) DO UPDATE SET
last_frame_id = excluded.last_frame_id,
last_compiled_at = excluded.last_compiled_at,
pages_compiled = excluded.pages_compiled
`).run(lastFrameId, pagesCompiled);
}
// ── Page Records ──────────────────────────────────────────────
getPage(slug: string): PageRecord | undefined {
return this.db.getDatabase().prepare(
'SELECT slug, page_type as pageType, name, content_hash as contentHash, markdown, frame_ids as frameIds, compiled_at as compiledAt, source_count as sourceCount FROM wiki_pages WHERE slug = ?',
).get(slug) as PageRecord | undefined;
}
getAllPages(): PageRecord[] {
return this.db.getDatabase().prepare(
'SELECT slug, page_type as pageType, name, content_hash as contentHash, markdown, frame_ids as frameIds, compiled_at as compiledAt, source_count as sourceCount FROM wiki_pages ORDER BY name',
).all() as PageRecord[];
}
getPagesByType(pageType: WikiPageType): PageRecord[] {
return this.db.getDatabase().prepare(
'SELECT slug, page_type as pageType, name, content_hash as contentHash, markdown, frame_ids as frameIds, compiled_at as compiledAt, source_count as sourceCount FROM wiki_pages WHERE page_type = ? ORDER BY name',
).all(pageType) as PageRecord[];
}
upsertPage(
slug: string,
pageType: WikiPageType,
name: string,
hash: string,
frameIds: number[],
sourceCount: number,
markdown = '',
): { action: 'created' | 'updated' | 'unchanged' } {
const raw = this.db.getDatabase();
const existing = this.getPage(slug);
if (existing && existing.contentHash === hash) {
return { action: 'unchanged' };
}
raw.prepare(`
INSERT INTO wiki_pages (slug, page_type, name, content_hash, markdown, frame_ids, compiled_at, source_count)
VALUES (?, ?, ?, ?, ?, ?, datetime('now'), ?)
ON CONFLICT(slug) DO UPDATE SET
content_hash = excluded.content_hash,
markdown = excluded.markdown,
frame_ids = excluded.frame_ids,
compiled_at = excluded.compiled_at,
source_count = excluded.source_count
`).run(slug, pageType, name, hash, markdown, JSON.stringify(frameIds), sourceCount);
return { action: existing ? 'updated' : 'created' };
}
deletePage(slug: string): boolean {
const result = this.db.getDatabase().prepare(
'DELETE FROM wiki_pages WHERE slug = ?',
).run(slug);
return result.changes > 0;
}
/** Get the highest frame ID in the database. */
getMaxFrameId(): number {
const row = this.db.getDatabase().prepare(
'SELECT COALESCE(MAX(id), 0) as max_id FROM memory_frames',
).get() as { max_id: number };
return row.max_id;
}
/** Get frames newer than a given ID. */
getFramesSince(frameId: number, limit = 500): { id: number; content: string; importance: string; source: string; created_at: string }[] {
return this.db.getDatabase().prepare(
'SELECT id, content, importance, source, created_at FROM memory_frames WHERE id > ? ORDER BY id ASC LIMIT ?',
).all(frameId, limit) as { id: number; content: string; importance: string; source: string; created_at: string }[];
}
}

109
packages/hive-mind-wiki-compiler/src/synthesizer.test.ts Normal file
View File

@@ -0,0 +1,109 @@
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import { resolveSynthesizer } from './synthesizer.js';
// The resolver contract only needs to know that the optional SDK can load.
// Mock the package here so Vitest's CJS/ESM interop does not decide the result
// of a provider-selection unit test; the real package import is covered by the
// installed-package/runtime smoke.
vi.mock('@anthropic-ai/sdk', () => ({
default: class MockAnthropic {},
}));
describe('resolveSynthesizer', () => {
const envBackup: Record<string, string | undefined> = {};
const envKeys = ['ANTHROPIC_API_KEY', 'OLLAMA_URL', 'OLLAMA_MODEL'];
beforeEach(() => {
for (const k of envKeys) {
envBackup[k] = process.env[k];
delete process.env[k];
}
});
afterEach(() => {
for (const k of envKeys) {
if (envBackup[k] === undefined) delete process.env[k];
else process.env[k] = envBackup[k];
}
vi.unstubAllGlobals();
});
it('falls back to echo when no env vars or explicit config are present', async () => {
const resolver = await resolveSynthesizer();
expect(resolver.provider).toBe('echo');
expect(resolver.model).toBe('echo');
expect(typeof resolver.synthesize).toBe('function');
});
it('picks anthropic when ANTHROPIC_API_KEY is present and SDK is available', async () => {
process.env.ANTHROPIC_API_KEY = 'sk-test-ignored-for-resolution-check';
const resolver = await resolveSynthesizer();
// The resolver should select anthropic without contacting the API.
expect(resolver.provider).toBe('anthropic');
expect(resolver.model).toBe('claude-haiku-4-5-20251001');
});
it('honours anthropicModel override', async () => {
process.env.ANTHROPIC_API_KEY = 'sk-test';
const resolver = await resolveSynthesizer({ anthropicModel: 'claude-opus-override' });
expect(resolver.provider).toBe('anthropic');
expect(resolver.model).toBe('claude-opus-override');
});
it('picks ollama over echo when OLLAMA_URL reports healthy', async () => {
vi.stubGlobal('fetch', vi.fn(async () => new Response('ok', { status: 200 })));
process.env.OLLAMA_URL = 'http://127.0.0.1:11434';
const resolver = await resolveSynthesizer();
expect(resolver.provider).toBe('ollama');
expect(resolver.model).toBe('llama3.2');
});
it('honours OLLAMA_MODEL override when ollama is selected', async () => {
vi.stubGlobal('fetch', vi.fn(async () => new Response('ok', { status: 200 })));
process.env.OLLAMA_URL = 'http://127.0.0.1:11434';
process.env.OLLAMA_MODEL = 'qwen3:7b';
const resolver = await resolveSynthesizer();
expect(resolver.provider).toBe('ollama');
expect(resolver.model).toBe('qwen3:7b');
});
it('falls through to echo if ollama health check fails', async () => {
vi.stubGlobal(
'fetch',
vi.fn(async () => new Response('down', { status: 503 })),
);
process.env.OLLAMA_URL = 'http://127.0.0.1:11434';
const resolver = await resolveSynthesizer();
expect(resolver.provider).toBe('echo');
});
it('echo synthesizer extracts entity name, frame count, and facts from the prompt', async () => {
const resolver = await resolveSynthesizer();
const prompt = `wiki about "Alice".
## Source Frames (3 total)
[Frame #7, 2026-04-18]: works at Acme Corp
[Frame #8, 2026-04-18]: role is engineer
[Frame #9, 2026-04-18]: based in Berlin
`;
const out = await resolver.synthesize(prompt);
expect(out).toContain('3 source frames');
expect(out).toContain('Alice');
expect(out).toContain('works at Acme Corp');
expect(out).toContain('(#7)');
expect(out).toContain('echo synthesizer');
expect(out).toMatch(/ANTHROPIC_API_KEY or OLLAMA_URL/);
});
it('echo synthesizer tolerates concept-shaped prompts', async () => {
const resolver = await resolveSynthesizer();
const prompt = `wiki about the concept "Observability"
## Source Frames (1 total)
[Frame #1, 2026-04-18]: Observability = logs + metrics + traces.
`;
const out = await resolver.synthesize(prompt);
expect(out).toContain('Observability');
expect(out).toContain('1 source frames');
});
});

157
packages/hive-mind-wiki-compiler/src/synthesizer.ts Normal file
View File

@@ -0,0 +1,157 @@
/**
* LLM Synthesizer — resolves the best available LLM for wiki page synthesis.
*
* Priority chain:
* 1. Anthropic Haiku (cheapest, fastest, best for synthesis)
* 2. Ollama (free, local)
* 3. Echo (no LLM, returns structured stub)
*
* All synthesizers implement LLMSynthesizeFn: (prompt: string) => Promise<string>
*/
import type { LLMSynthesizeFn } from './types.js';
export interface SynthesizerConfig {
/** Anthropic API key. Falls back to process.env.ANTHROPIC_API_KEY if not provided. */
anthropicApiKey?: string;
/** Anthropic model id. Default: claude-haiku-4-5-20251001. */
anthropicModel?: string;
/** Ollama base URL. Falls back to process.env.OLLAMA_URL if not provided. */
ollamaUrl?: string;
/** Ollama model name (default: llama3.2). Falls back to process.env.OLLAMA_MODEL. */
ollamaModel?: string;
/** Max tokens for synthesis output (default: 1500) */
maxTokens?: number;
}
// ── Anthropic (Haiku) ───────────────────────────────────────────
function createAnthropicSynthesizer(apiKey: string, model: string, maxTokens: number): LLMSynthesizeFn {
return async (prompt: string): Promise<string> => {
// Dynamic import — @anthropic-ai/sdk is an optional peer dep.
const { default: Anthropic } = await import('@anthropic-ai/sdk');
const client = new Anthropic({ apiKey });
const response = await client.messages.create({
model,
max_tokens: maxTokens,
messages: [{ role: 'user', content: prompt }],
});
const textBlock = response.content.find(b => b.type === 'text');
return textBlock?.text ?? '';
};
}
// ── Ollama ──────────────────────────────────────────────────────
function createOllamaSynthesizer(baseUrl: string, model: string, maxTokens: number): LLMSynthesizeFn {
return async (prompt: string): Promise<string> => {
const url = `${baseUrl.replace(/\/$/, '')}/api/generate`;
const response = await fetch(url, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
model,
prompt,
stream: false,
options: { num_predict: maxTokens },
}),
signal: AbortSignal.timeout(60_000),
});
if (!response.ok) {
throw new Error(`Ollama error: ${response.status} ${response.statusText}`);
}
const data = await response.json() as { response: string };
return data.response ?? '';
};
}
// ── Echo (fallback) ─────────────────────────────────────────────
function createEchoSynthesizer(): LLMSynthesizeFn {
return async (prompt: string): Promise<string> => {
const frameMatch = prompt.match(/\((\d+) total\)/);
const frameCount = frameMatch ? frameMatch[1] : '?';
const nameMatch = prompt.match(/about "([^"]+)"/) ?? prompt.match(/concept "([^"]+)"/);
const name = nameMatch ? nameMatch[1] : 'this topic';
// Extract actual frame content for a basic summary
const frameLines = (prompt.match(/\[Frame #\d+.*?\]: .+/g) || []);
const facts = frameLines.slice(0, 8).map(line => {
const m = line.match(/\[Frame (#\d+).*?\]: (.+)/);
return m ? `- ${m[2].slice(0, 200)} *(${m[1]})*` : null;
}).filter(Boolean);
return `## Summary\nCompiled from ${frameCount} source frames about ${name}.\n\n` +
(facts.length > 0 ? `## Key Facts\n${facts.join('\n')}\n\n` : '') +
`> *Compiled with echo synthesizer. Connect an LLM for richer synthesis.*\n` +
`> Set ANTHROPIC_API_KEY or OLLAMA_URL in your environment.`;
};
}
// ── Resolver ────────────────────────────────────────────────────
export interface ResolvedSynthesizer {
synthesize: LLMSynthesizeFn;
provider: 'anthropic' | 'ollama' | 'echo';
model: string;
}
/**
* Resolve the best available LLM synthesizer.
* Checks env vars and config, returns the first working option.
*/
export async function resolveSynthesizer(config?: SynthesizerConfig): Promise<ResolvedSynthesizer> {
const maxTokens = config?.maxTokens ?? 1500;
const anthropicModel = config?.anthropicModel ?? 'claude-haiku-4-5-20251001';
// 1. Try Anthropic
const anthropicKey = config?.anthropicApiKey ?? process.env.ANTHROPIC_API_KEY;
if (anthropicKey) {
try {
// Verify the SDK is importable
await import('@anthropic-ai/sdk');
return {
synthesize: createAnthropicSynthesizer(anthropicKey, anthropicModel, maxTokens),
provider: 'anthropic',
model: anthropicModel,
};
} catch {
// SDK not available — fall through
}
}
// 2. Try Ollama
const ollamaUrl = config?.ollamaUrl ?? process.env.OLLAMA_URL;
const ollamaModel = config?.ollamaModel ?? process.env.OLLAMA_MODEL ?? 'llama3.2';
if (ollamaUrl) {
try {
// Quick health check
const health = await fetch(`${ollamaUrl.replace(/\/$/, '')}/api/tags`, {
signal: AbortSignal.timeout(3_000),
});
if (health.ok) {
return {
synthesize: createOllamaSynthesizer(ollamaUrl, ollamaModel, maxTokens),
provider: 'ollama',
model: ollamaModel,
};
}
} catch {
// Ollama not reachable — fall through
}
}
// 3. Echo fallback
return {
synthesize: createEchoSynthesizer(),
provider: 'echo',
model: 'echo',
};
}

115
packages/hive-mind-wiki-compiler/src/types.ts Normal file
View File

@@ -0,0 +1,115 @@
/**
* Wiki Compiler Types — page definitions, compilation config, and state tracking.
*/
// ── Page Types ──────────────────────────────────────────────────
export type WikiPageType =
| 'entity'
| 'concept'
| 'synthesis'
| 'index'
| 'health';
export interface WikiPageFrontmatter {
type: WikiPageType;
name: string;
entity_type?: string;
confidence: number;
sources: number;
last_compiled: string;
frame_ids: number[];
related_entities: string[];
}
export interface WikiPage {
/** URL-safe slug, e.g. "project-alpha" */
slug: string;
/** Page frontmatter */
frontmatter: WikiPageFrontmatter;
/** Full markdown content (including frontmatter as YAML) */
markdown: string;
/** SHA-256 hash of content for change detection */
contentHash: string;
}
// ── Compilation State ───────────────────────────────────────────
export interface CompilationWatermark {
/** Highest frame ID processed in last compilation */
lastFrameId: number;
/** ISO timestamp of last compilation */
lastCompiledAt: string;
/** Number of pages generated/updated */
pagesCompiled: number;
}
export interface PageRecord {
slug: string;
pageType: WikiPageType;
name: string;
contentHash: string;
markdown: string;
frameIds: string; // JSON array
compiledAt: string;
sourceCount: number;
}
// ── Compiler Configuration ──────────────────────────────────────
export type LLMSynthesizeFn = (prompt: string) => Promise<string>;
export interface CompilerConfig {
/** Function to call the LLM for synthesis */
synthesize: LLMSynthesizeFn;
/** Output directory for wiki pages (default: wiki/) */
outputDir?: string;
/** Minimum frames to justify a page (default: 2) */
minFramesPerPage?: number;
/** Maximum frames to send as context per LLM call (default: 30) */
maxFramesPerCall?: number;
/** Minimum confidence for entity pages (default: 0.3) */
minConfidence?: number;
}
// ── Compilation Result ──────────────────────────────────────────
export interface CompilationResult {
pagesCreated: number;
pagesUpdated: number;
pagesUnchanged: number;
entityPages: string[];
conceptPages: string[];
synthesisPages: string[];
healthIssues: number;
watermark: CompilationWatermark;
durationMs: number;
}
// ── Health Report ───────────────────────────────────────────────
export type HealthIssueType =
| 'contradiction'
| 'gap'
| 'orphan_entity'
| 'weak_confidence'
| 'stale_page'
| 'missing_page';
export interface HealthIssue {
type: HealthIssueType;
severity: 'high' | 'medium' | 'low';
description: string;
entity?: string;
frameIds?: number[];
suggestion?: string;
}
export interface HealthReport {
totalEntities: number;
totalFrames: number;
totalPages: number;
issues: HealthIssue[];
dataQualityScore: number; // 0-100
compiledAt: string;
}

12
packages/hive-mind-wiki-compiler/tsconfig.json Normal file
View File

@@ -0,0 +1,12 @@
{
"extends": "../../tsconfig.base.json",
"compilerOptions": {
"composite": true,
"rootDir": "./src",
"outDir": "./dist",
"tsBuildInfoFile": "./dist/.tsbuildinfo"
},
"references": [{ "path": "../hive-mind-core" }],
"include": ["src/**/*.ts"],
"exclude": ["dist", "node_modules", "**/*.test.ts", "**/*.spec.ts"]
}