waggle-os/docs/decisions/2026-04-28-test-coverage-gap-report.md

decision_id: 2026-04-28-test-coverage-gap-report date: 2026-04-28 phase: parallel work — test coverage gap analysis (report-only) verdict: heuristic estimate. Real P1 gaps small after false-positive correction; ~6-8 modules genuinely below 80%. Bulk of "below 80%" signal is connector files (P2-bordering-P3) and a few legacy utility modules. Recommended Sprint 12 additions: small focused list, not a wholesale test-debt cleanup. constraint: report-only; no code changes; no test additions; no package.json modifications methodology_caveat: vitest @vitest/coverage-v8 not installed; package.json constraint forbids adding it. Static src→test heuristic used as fallback. KNOWN false-positive class: per-file source matched against per-file test only — multi-source-per-test conventions (e.g., prompt-shapes.test.ts covers 7 source files) flagged as 0-coverage incorrectly. Correction pass applied below.

Test Coverage Gap Report

1. Methodology

The repo's vitest.config.ts declares a coverage block with provider v8, but the corresponding package @vitest/coverage-v8 is not in node_modules. Running npx vitest run --coverage returns:

MISSING DEPENDENCY  Cannot find dependency '@vitest/coverage-v8'

Attempting npx --package=@vitest/coverage-v8 -- vitest run --coverage fails because the local vitest install can't resolve a package outside its own node_modules. The PM constraint "DO NOT modify package.json" forbids the obvious fix (npm install --no-save -D @vitest/coverage-v8 would persist a lockfile change).

Fallback methodology used here: static source-to-test mapping via filename stem matching, with test-file-LoC-to-source-file-LoC ratio as a coverage proxy.

For each .ts file in packages/agent/src/ and packages/core/src/ (excluding .d.ts), the scan:

1. Counts non-comment, non-empty LoC (srcLoc)
2. Searches tests/ (recursive) for *.test.ts files matching one of:
   • <stem>.test.ts exact match
   • <dir-parts-joined-with-dash>-<stem>.test.ts (waggle-os convention: long-task-checkpoint.test.ts for src/long-task/checkpoint.ts)
   • Loose match: filename contains stem AND path contains all dir parts
3. Sums matched test files' LoC (testLoc)
4. Computes ratio = testLoc / srcLoc
5. Estimates coverage bucket: 0 (no tests) / <50 (ratio < 0.3) / 50-80 (0.3–0.7) / >=80 (>= 0.7)
6. Classifies priority:
   • P1: critical agent-loop / substrate path (agent-loop, retrieval-agent-loop, orchestrator, personas, behavioral-spec, output-normalize, run-meta, prompt-shapes/, long-task/, mind/* core, vault, harvest pipeline, injection-scanner, cost-tracker, tool-filter, permissions)
   • P3: barrel / type-only (index.ts, types.ts, *-types.ts, constants.ts)
   • P2: everything else

Caveats (honest disclosure):

• The heuristic does NOT measure actual line coverage. A test file that imports the source but exercises only one function will still register a high testLoc/srcLoc ratio and read as "well-covered."
• The heuristic FALSE-POSITIVES on multi-source-per-test conventions: tests/prompt-shapes.test.ts (a single 1100-LoC test file covering all 7 prompt-shape source files) only matches prompt-shapes/types.ts via stem, so the other 6 prompt-shape files report as "zero tests." A correction pass below resolves the most obvious cases.
• Pure-data files (persona-data.ts, declarative arrays) are flagged as zero-test even when their data is exercised indirectly through the logic file's tests (personas.test.ts).
• index.ts barrel files are typed as P3 and excluded from priority analysis.

Net interpretation: the report is a directional signal, not a measurement. P1 gaps surfaced by this heuristic should be cross-checked manually before booking work. P2/P3 gaps are useful for backlog priority discussion.

2. Per-package summary

Package	Modules total	With tests	Without tests	Est. ≥80%	Est. <80%	Rough cov% by LoC
packages/agent	156	100	56	74	82	46%
packages/core	64	37	27	32	32	56%

The <80% count is inflated by the false-positive class noted above. Genuine gaps after correction: see §3.

Distribution of <80% modules across priority buckets (heuristic, pre-correction):

• P1: 15 (mostly false positives — see §3 correction pass)
• P2: 93 (real signal: connectors + heavy utility files)
• P3: 6 (barrel files + types — out of scope per §5)

3. Module-by-module gap list

P1 — critical path (correction pass applied)

The heuristic flagged 15 P1 modules. Cross-checking against the actual test directory shows most are covered through multi-source-per-test files. The corrected list:

Module	LoC	Heuristic verdict	Corrected status	Action
agent/persona-data.ts	911	0 tests	covered indirectly by personas.test.ts (PERSONAS array iterated via logic file). Pure data; not directly testable in isolation.	None — false positive
agent/retrieval-agent-loop.ts	828	ratio 0.55 / 50-80	Likely well-covered: retrieval-agent-loop.test.ts (455 LoC, 25 tests) + long-task-loop-integration.test.ts (847 LoC, 34 tests including Phase 4.7 assertion suite). Combined test LoC ≈ 1300 against 828 src LoC → ratio ≈1.6.	None — heuristic missed cross-file integration test
agent/behavioral-spec.ts	338	ratio 0.19 / <50	Real gap. Only behavioral-spec-overrides.test.ts (65 LoC) tests the override mechanism. The main BEHAVIORAL_SPEC constant + section structure + COMPACTION_PROMPT export aren't directly verified by a dedicated suite. Mitigated by indirect testing through orchestrator, but no lock-in test.	Sprint 12 candidate: add a structural-shape test for the spec (sections present, COMPACTION_PROMPT non-empty, etc.) — small effort, high pin-value
core/harvest/pipeline.ts	283	ratio 0.53 / 50-80	pipeline-injection.test.ts (66 LoC) + pipeline-progress.test.ts (83 LoC) cover injection scanning + progress reporting. End-to-end harvest flow not exercised.	Sprint 12 candidate: end-to-end harvest test against fixture data
agent/cost-tracker.ts	123	ratio 0.40 / <50	cost-tracker.test.ts exists but thin. CostTracker class methods have partial coverage.	Sprint 12 candidate: complete CostTracker method matrix (record / get / reset / aggregate)
agent/prompt-shapes/selector.ts	116	0 tests	covered by prompt-shapes.test.ts (1100+ LoC, 65 tests including selector dispatch tests).	None — false positive
agent/prompt-shapes/claude.ts	90	0 tests	covered by prompt-shapes.test.ts	None — false positive
agent/prompt-shapes/types.ts	83	0 tests	Type-only module; no testable runtime behavior. Indirectly verified by tsc strict on consumers.	None — out of scope per §5
agent/prompt-shapes/qwen-thinking.ts	81	0 tests	covered by prompt-shapes.test.ts	None — false positive
agent/prompt-shapes/qwen-non-thinking.ts	79	0 tests	covered by prompt-shapes.test.ts	None — false positive
agent/prompt-shapes/generic-simple.ts	73	0 tests	covered by prompt-shapes.test.ts	None — false positive
core/injection-scanner.ts	71	0 tests	Real gap. agent/src/injection-scanner.ts has a dedicated test (agent/tests/injection-scanner.test.ts); the duplicate core/src/injection-scanner.ts does NOT have a core/tests/injection-scanner.test.ts. Substantively the same scanning logic. Either dedupe (Sprint 12 cleanup candidate) or add core-side test.	Sprint 12 candidate: investigate dedup vs add test
agent/prompt-shapes/gpt.ts	62	0 tests	covered by prompt-shapes.test.ts	None — false positive
agent/prompt-shapes/index.ts	44	0 tests	Barrel. P3.	None — out of scope
agent/custom-personas.ts	40	0 tests	Borderline gap. loadCustomPersonas() reads from disk; not exercised by personas.test.ts (which uses static PERSONAS). Small file.	Sprint 12 candidate: minimal test for loadCustomPersonas with fixture

Genuine P1 gaps after correction: 4 modules (behavioral-spec, harvest/pipeline, cost-tracker, custom-personas) plus 1 dedup-or-test investigation (core/injection-scanner). All small / focused.

P2 — supporting (top 25 by LoC)

These are real gap signals. Connector files are dominant — a known cluster of test debt that has accumulated as new connectors were added without test scaffolding.

Module	LoC	Coverage est.	Notes
agent/system-tools.ts	831	50-80	2 test files (system-tools + bash-sandboxing). Big surface; partial.
agent/skill-tools.ts	777	<50	1 test file. 28 tools defined; thin coverage of CRUD paths.
agent/evolve-schema.ts	734	50-80	evolve-schema.test.ts covers main path; mutation kinds may be partial.
agent/document-tools.ts	559	<50	Document generation (docx/pdf/pptx) thin on output validation.
agent/commands/workflow-commands.ts	537	50-80	Command-registry handlers; partial.
agent/lsp-tools.ts	406	<50	LSP integration; partial.
agent/workflow-harness.ts	406	0 tests	Multi-phase harness; not tested directly. Real gap.
core/file-store.ts	398	50-80	File-store abstraction.
core/mind/embedding-provider.ts	386	<50	Embedding-provider switching (api / inprocess / litellm / ollama); likely thin.
agent/subagent-tools.ts	333	50-80	Subagent spawn/coord; partial.
agent/workflow-tools.ts	319	50-80	Workflow CRUD tools.
agent/compliance-pdf.ts	315	50-80	EU AI Act compliance PDF rendering.
agent/browser-tools.ts	306	50-80	Playwright browser tools.
agent/connectors/obsidian-connector.ts	300	0 tests	Connector cluster, no tests.
core/harvest/claude-code-adapter.ts	300	0 tests	Harvest adapter for Claude Code.
agent/cross-workspace-tools.ts	294	0 tests	Cross-workspace file ops.
agent/git-tools.ts	274	50-80	Git operations.
core/telemetry.ts	273	50-80	Telemetry pipeline.
core/cron-store.ts	270	50-80	Cron job persistence.
agent/connectors/gdrive-connector.ts	266	0 tests	Connector cluster.
agent/connectors/notion-connector.ts	262	0 tests	Connector cluster.
agent/connector-search.ts	248	50-80	Connector search (also exercised by harvest pipeline tests indirectly).
agent/connectors/confluence-connector.ts	247	0 tests	Connector cluster.
agent/connectors/trello-connector.ts	247	0 tests	Connector cluster.
agent/connectors/outlook-connector.ts	244	0 tests	Connector cluster.

Connector cluster: 30+ connector files in packages/agent/src/connectors/ are each ~100-300 LoC, mostly without dedicated tests. They share a common base class (BaseConnector) which IS tested (connector-sdk.test.ts). Argument for low-priority: connectors are mostly thin adapters around external APIs; integration testing each one requires real auth and external services. Argument for higher-priority: pre-launch product surface; P1 customer impact if a connector breaks silently.

P3 — barrel / type-only

Out of scope per §5. Listed for completeness:

Module	LoC	Reason
agent/index.ts	409	Barrel re-exports
core/compliance/types.ts	161	Type-only
core/harvest/types.ts	103	Type-only
agent/connectors/index.ts	30	Barrel
core/harvest/index.ts	14	Barrel
core/compliance/index.ts	5	Barrel

4. Recommended Sprint 12 additions

Small focused list — not a wholesale test-debt cleanup. The 23-item Sprint 12 backlog from Phase 4.4/4.5 already exists; adding ~8 high-value test items here would bring Sprint 12 to ~31 items, still tractable.

P1 — critical path test gaps (5 items, ~12-16 hours):

1. agent/behavioral-spec.ts — structural-shape test (sections present, COMPACTION_PROMPT non-empty, BEHAVIORAL_SPEC_SECTIONS export consistent). Estimate: 2 hours.
2. core/injection-scanner.ts — investigate dedup vs add test against agent/injection-scanner.ts. Estimate: 1-2 hours (most of which is decision, not coding).
3. core/harvest/pipeline.ts — end-to-end harvest test against fixture data covering at least one adapter (chatgpt or claude-code). Estimate: 4 hours.
4. agent/cost-tracker.ts — complete CostTracker method matrix. Estimate: 2 hours.
5. agent/custom-personas.ts — loadCustomPersonas test with fixture directory. Estimate: 2 hours.

P2 — biggest real gaps (top 3 by impact, ~16-20 hours):

6. agent/workflow-harness.ts — createHarnessRun / advancePhase / canRetry not tested. 406 LoC of orchestration code. Estimate: 6 hours.
7. agent/skill-tools.ts — round-trip tests for create_skill + discover_skills + auto_extract_skills. 777 LoC; current coverage thin. Estimate: 6 hours.
8. agent/document-tools.ts — output-shape validation tests for docx/pdf/pptx generation. 559 LoC. Estimate: 4 hours.

Connector cluster — discuss separately: The 6+ untested connector files (gdrive / obsidian / notion / confluence / trello / outlook + ~25 more) form a coherent gap. PM recommendation needed on whether to tackle as a single Sprint 12 work-item or spread across sprints. Each connector is ~100-300 LoC and ~3-5 hours to test against mocked HTTP. If all 30+ connectors → ~90-150 hours. Large effort; likely deferred to a dedicated "connector-test-debt" sprint rather than bundled into Sprint 12 cleanup.

5. Out-of-scope notes

The following modules are correctly NOT booked for test additions:

1. Type-only files (prompt-shapes/types.ts, compliance/types.ts, harvest/types.ts): no runtime behavior; tsc strict on consumers verifies type correctness.
2. Barrel files (index.ts at every level): re-exports only; tested transitively when consumers import them.
3. Persona data array (persona-data.ts, 911 LoC): pure declarative data; iterated through personas.test.ts via the logic file.
4. Generated/legacy connectors if any exist (none confirmed in this audit).
5. Files with multi-source-per-test coverage: prompt-shapes/*.ts (covered by prompt-shapes.test.ts), the personas.test.ts family. The static heuristic mis-flags these but they have substantive coverage through their shared test files.

6. Audit chain

Item	Value
Branch HEAD	c9bda3d (Phase 4.7, unchanged)
Coverage tool	NOT installed (@vitest/coverage-v8 absent); fallback static heuristic used
Scan script	D:\Projects\waggle-os\tmp\coverage-gap-scan.mjs
Scan output	D:\Projects\waggle-os\tmp\coverage-gap-output.json
Modules scanned	220 (156 agent + 64 core)
Modules flagged <80% (heuristic)	114 (82 agent + 32 core)
Genuine P1 gaps after correction	4-5 (behavioral-spec / harvest pipeline / cost-tracker / custom-personas / core injection-scanner dedup-or-test)
Genuine P2 gaps prioritized	top 3 (workflow-harness / skill-tools / document-tools)
Connector cluster	flagged for separate PM decision (30+ files, large effort)
Cost	$0
Code modified	0
Tests added	0

7. PM ratification asks

1. Accept the heuristic methodology disclaimer — true coverage instrumentation requires installing @vitest/coverage-v8 which violates the package.json constraint. Static src→test mapping is the next-best signal but has known false-positive class.
2. Authorize a one-time install of @vitest/coverage-v8 for a future precise-measurement pass? (Optional follow-up; if approved would yield definitive numbers but requires lockfile change.)
3. Add the 8 recommended Sprint 12 items (5 P1 + 3 P2-top) to the existing 23-item Sprint 12 backlog → total Sprint 12 = ~31 items?
4. Decide separately on the connector cluster — single Sprint 12 work-item (~90-150 hours, dedicated focus) vs distributed across multiple sprints vs deferred to post-launch?

---

End of test coverage gap report. Resuming Phase 5 standby.