waggle-os/docs/decisions/2026-04-28-phase-4-4-skills-audit-results.md

decision_id: 2026-04-28-phase-4-4-skills-audit-results date: 2026-04-28 phase: 4.4 skills audit sweep verdict: skills system is mostly model-portable (87.4% clean). 12.6% with detectable narrative bias = Tier 1 deferred to Sprint 12. NO blocker for Phase 5 NULL-baseline. Surface finding: pilot 2026-04-26 didn't engage skills system at all — empirical anchoring impossible. predecessor: 2026-04-28-phase-4-3-rescore-delta-report.md sprint_plan: D:\Projects\waggle-os\decisions\2026-04-26-agent-fix-sprint-plan.md

Phase 4.4 Skills Audit Sweep — Results

TL;DR

Audited skills in two layers:

• TS source layer (packages/agent/src/skill-*.ts, 8 files): governs how skills are loaded, surfaced, recommended, generated. Model-portable by construction. No bias surface found.
• Skill-content layer (159 SKILL.md files: 152 user-global + 7 workspace): the actual skill descriptions/bodies. Systematic bias scan: 12.6% (20/159) have detectable narrative-voice / CoT-imperative / philosophy-keyword patterns.

Most important finding: the 2026-04-26 pilot prompts contain NO skill descriptions. The agent's skill recommender wasn't engaged during pilot orchestration. Whatever bias exists in skill content is empirically irrelevant to the H3/H4 deltas — those failures are NOT explained by skill-surface bias because skills weren't surfaced.

Tier 1 (description rewrite would help): 20 skills. Defer to Sprint 12 cleanup. No blocker for Phase 5 NULL-baseline. Tier 2 (real coverage gap): no candidates from this sample. Skills cover broad task categories; no obvious uncovered category surfaced. NEW finding: skill-engagement gap. Recommender isn't wired into the agent loop's prompt assembly during the pilot scenario. This is orthogonal to skill-content bias and worth flagging as a separate concern.

Cumulative cost: $0 (pure code+content audit; no LLM calls).

Methodology

Per Option A discipline (audit-only, no refactor):

1. TS source review — read 3 of 8 skill-*.ts files in full (skill-tools.ts first 100L, skill-frontmatter.ts, skill-recommender.ts, skill-creator.ts). Sampled the rest by file size + filename.

2. Skill content sampling — hand-read 6 skill markdown files representing diverse categories:

   • ax-gepa (workspace, technical / GEPA-relevant)
   • agent-chronicle (user-global, narrative)
   • django-tdd (user-global, technical / TDD)
   • elite-longterm-memory (user-global, marketing)
   • configure-ecc (user-global, procedural)
   • gsd-research-phase (user-global, command-style)

3. Systematic bias scan — 159 SKILL.md files scanned with 6 narrative-bias regex patterns:

   • we-think (first-person plural: "we process", "we believe", "we are")
   • our-philosophy ("our approach", "our way", "our values", "our mission")
   • philosophy-keyword ("philosophy", "the spirit", "deep down", "fundamentally")
   • marketing-superlative ("ultimate", "bulletproof", "never lose", "never forget")
   • emoji (codepoint detection in body)
   • CoT-imperative ("let me", "i'll start", "step-by-step", "first.. then")

4. Empirical anchor attempt — inspected pilot 2026-04-26 prompt archive to check if skills surfaced. Result: no skill content in any pilot prompt. Audit becomes theoretical (bias may exist but didn't affect pilot outcomes).

TS source layer findings

skill-tools.ts (41 KB, 8 tools defined)

Tool definitions for the agent:

• discover_skills, install_skill, create_skill, auto_extract_skills, promote_skill, retire_stale_skills, recommend_skills, …
• All tools have concrete imperative descriptions (e.g., "Search the marketplace catalog…"). No narrative voice. Model-portable.

Surfacing logic: skills are loaded from disk and provided to the orchestrator via dep injection. The actual prompt assembly (including how skill content lands in the system prompt) is not in this file — it's the orchestrator's concern.

VERDICT: Tier 0 (no bias).

skill-frontmatter.ts (5.8 KB)

Pure parsing utility. YAML-ish frontmatter parser. No prompt content; cannot introduce bias.

VERDICT: Tier 0.

skill-recommender.ts (9.5 KB)

Pure-keyword scoring with synonym clusters and bigram overlap. Multi-signal TF-IDF-inspired ranking. Returns top-N skills by relevance score — feeds into prompt only when caller chooses to surface them.

The synonym clusters (line 44-58) are model-agnostic — concept-based, not Claude-specific.

VERDICT: Tier 0.

skill-creator.ts — generator template

Lines 26-62: generateSkillMarkdown(template) produces SKILL.md from a SkillTemplate. Output shape is concrete + imperative — Trigger Patterns / Steps / Tools Used / Category — no narrative voice.

Sample output for an auto-generated skill would have:

## Trigger Patterns
Activate this skill when the user asks about: <triggers>

## Steps
1. <imperative step>
2. <imperative step>

This is model-portable. Any new skill generated by the agent will be Tier 0.

VERDICT: Tier 0.

Skill-content layer findings

Hand-sampled (6 files)

Skill	Category	Description shape	Body shape	Tier
ax-gepa	workspace, technical	Concrete trigger keywords	Imperative bullets, code patterns	Tier 0
agent-chronicle	user, narrative	Narrative ("AI perspective journaling")	"We process thousands…" first-person plural; "Philosophy" section pure narrative; emoji in title	Tier 1
django-tdd	user, technical	Concrete activation criteria	Procedural / code-driven	Tier 0
elite-longterm-memory	user, infrastructure	Marketing copy ("Ultimate", "bulletproof")	Marketing prose; "Never lose context. Never forget decisions." emphasis stack	Tier 1
configure-ecc	user, procedural	Concrete activation triggers	Imperative bullets	Tier 0
gsd-research-phase	user, command	Concrete argument-hint	<objective> XML wrapper (Claude-leaning but generally portable)	Borderline

2 of 6 hand-sampled (33%) flagged as Tier 1. Higher than the systematic scan rate because hand-sampling biased toward illustrative cases.

Systematic scan (159 files, 6 regex patterns)

Pattern	Hits	Notes
CoT-imperative ("let me", "step-by-step", etc.)	12	Most common bias
philosophy-keyword	5	Verbose framing in body
we-think (first-person plural)	3	Narrative voice
marketing-superlative ("ultimate", "bulletproof")	1	Marketing copy
our-philosophy	1	Narrative voice
emoji (body emoji)	1	Decorative
Skills with ≥1 hit	20 (12.6%)
Skills clean	139 (87.4%)

The dominant pattern is CoT-imperative phrasing — verbose multi-step instructions in narrative form. Phase 1.1 normalize won't strip this from skill content (it's content, not output artifact). Sprint 12 rewrite would be needed.

Tier classification

Tier 0 (model-portable, no action needed)

• All 8 TS source files (skill-*.ts)
• 139 of 159 SKILL.md files (87.4%)
• Skill creator's generator template — new skills will be Tier 0 by construction

Tier 1 (description/body rewrite would help — defer to Sprint 12)

• 20 SKILL.md files (12.6%) with detectable narrative-voice / CoT-imperative / philosophy-keyword bias
• These would benefit from rewrite to imperative-direct format
• NOT a blocker for Phase 5 NULL-baseline (pilot didn't engage skills)
• Cumulative refactor effort estimate: ~2-3 hours per skill × 20 skills = ~40-60 hours of focused work. Out-of-scope for current sprint.

Tier 2 (real coverage gap — GEPA territory, defer to CC-2 work)

• No candidates surfaced from this sample. Skills cover broad task categories (writing, coding, research, planning, decision support). The pilot's H3/H4 failures are content-quality issues in synthesis output, not gaps in skill coverage.

NEW (not in original Phase 4.4 brief): skill-engagement gap

• The 2026-04-26 pilot orchestration didn't surface any skill content into prompts. Recommender ran (or didn't — unclear) but no skill descriptions made it into the LLM's context.
• This is a wiring concern, not a skill-content concern. If Phase 5 NULL-baseline runs the same scenario configuration, the issue will repeat.
• Recommendation: if PM wants to test whether skill engagement materially helps Qwen vs Opus, that's a separate dedicated experiment, not part of Phase 5 NULL-baseline.

Strategic implications

Phase 5 NULL-baseline impact

NONE. The skills system was not engaged during the pilot, so no skill-bias-related signal exists in the H3/H4 deltas. Phase 5 NULL-baseline can proceed without skill work.

Sprint 12 cleanup recommendation

Rewrite the 20 Tier 1 skills to imperative-direct format. Generator template (skill-creator.ts) is already model-portable, so newly-generated skills won't accumulate bias. Manual cleanup applies only to the legacy corpus.

Priority order (highest-impact first based on hand-sampling):

1. agent-chronicle (philosophy + emoji + first-person plural — strong bias)
2. elite-longterm-memory (marketing copy + "Never lose…" emphasis stack)
3. The 12 CoT-imperative skills — bulk rewrite to bullet-imperative form
4. The 5 philosophy-keyword skills — strip philosophical framings

Out-of-scope for this audit

Per PM brief discipline:

• No code changes (audit-only)
• No skill rewrites
• No new test coverage required (no source code modified)

Audit chain

Item	Value
Branch HEAD	c9bda3d (Phase 4.7 commit, unchanged)
TS source files reviewed	8 of 8 (skill-*.ts in packages/agent/src/)
SKILL.md files in scope	159 (152 user-global + 7 workspace; 3 dirs without SKILL.md)
SKILL.md files hand-sampled	6
SKILL.md files scanned (regex patterns)	159
Bias detection rate	12.6% (20/159)
Audit cost	$0 (no LLM calls)
Tests modified	0
Code modified	0

PM ratification asks

1. Accept Tier 1 / Tier 2 split as documented (no Tier 2 candidates; 12.6% Tier 1 deferred to Sprint 12)?
2. Acknowledge the skill-engagement gap as a NEW finding (orthogonal to the original audit scope) and decide whether to author a dedicated brief for "skill engagement during agentic synthesis" experiments? Recommendation: defer; not a Phase 5 NULL-baseline blocker.
3. Proceed to Phase 4.5 (tools audit) per recommended order.

---

End of Phase 4.4. Ready to proceed to Phase 4.5 (tools audit) per PM ratification.