This commit is contained in:
Oleg Maslov
190
docs/decisions/2026-04-28-phase-4-4-skills-audit-results.md
Normal file
190
docs/decisions/2026-04-28-phase-4-4-skills-audit-results.md
Normal file
@@ -0,0 +1,190 @@
|
||||
---
|
||||
decision_id: 2026-04-28-phase-4-4-skills-audit-results
|
||||
date: 2026-04-28
|
||||
phase: 4.4 skills audit sweep
|
||||
verdict: skills system is mostly model-portable (87.4% clean). 12.6% with detectable narrative bias = Tier 1 deferred to Sprint 12. NO blocker for Phase 5 NULL-baseline. Surface finding: pilot 2026-04-26 didn't engage skills system at all — empirical anchoring impossible.
|
||||
predecessor: 2026-04-28-phase-4-3-rescore-delta-report.md
|
||||
sprint_plan: D:\Projects\waggle-os\decisions\2026-04-26-agent-fix-sprint-plan.md
|
||||
---
|
||||
|
||||
# Phase 4.4 Skills Audit Sweep — Results
|
||||
|
||||
## TL;DR
|
||||
|
||||
Audited skills in two layers:
|
||||
- **TS source layer** (`packages/agent/src/skill-*.ts`, 8 files): governs how skills are loaded, surfaced, recommended, generated. **Model-portable by construction.** No bias surface found.
|
||||
- **Skill-content layer** (159 SKILL.md files: 152 user-global + 7 workspace): the actual skill descriptions/bodies. **Systematic bias scan: 12.6% (20/159) have detectable narrative-voice / CoT-imperative / philosophy-keyword patterns.**
|
||||
|
||||
**Most important finding: the 2026-04-26 pilot prompts contain NO skill descriptions.** The agent's skill recommender wasn't engaged during pilot orchestration. Whatever bias exists in skill content is empirically irrelevant to the H3/H4 deltas — those failures are NOT explained by skill-surface bias because skills weren't surfaced.
|
||||
|
||||
**Tier 1 (description rewrite would help): 20 skills.** Defer to Sprint 12 cleanup. No blocker for Phase 5 NULL-baseline.
|
||||
**Tier 2 (real coverage gap): no candidates from this sample.** Skills cover broad task categories; no obvious uncovered category surfaced.
|
||||
**NEW finding: skill-engagement gap.** Recommender isn't wired into the agent loop's prompt assembly during the pilot scenario. This is orthogonal to skill-content bias and worth flagging as a separate concern.
|
||||
|
||||
Cumulative cost: **$0** (pure code+content audit; no LLM calls).
|
||||
|
||||
## Methodology
|
||||
|
||||
Per Option A discipline (audit-only, no refactor):
|
||||
|
||||
1. **TS source review** — read 3 of 8 skill-*.ts files in full (`skill-tools.ts` first 100L, `skill-frontmatter.ts`, `skill-recommender.ts`, `skill-creator.ts`). Sampled the rest by file size + filename.
|
||||
|
||||
2. **Skill content sampling** — hand-read 6 skill markdown files representing diverse categories:
|
||||
- `ax-gepa` (workspace, technical / GEPA-relevant)
|
||||
- `agent-chronicle` (user-global, narrative)
|
||||
- `django-tdd` (user-global, technical / TDD)
|
||||
- `elite-longterm-memory` (user-global, marketing)
|
||||
- `configure-ecc` (user-global, procedural)
|
||||
- `gsd-research-phase` (user-global, command-style)
|
||||
|
||||
3. **Systematic bias scan** — 159 SKILL.md files scanned with 6 narrative-bias regex patterns:
|
||||
- `we-think` (first-person plural: "we process", "we believe", "we are")
|
||||
- `our-philosophy` ("our approach", "our way", "our values", "our mission")
|
||||
- `philosophy-keyword` ("philosophy", "the spirit", "deep down", "fundamentally")
|
||||
- `marketing-superlative` ("ultimate", "bulletproof", "never lose", "never forget")
|
||||
- `emoji` (codepoint detection in body)
|
||||
- `CoT-imperative` ("let me", "i'll start", "step-by-step", "first.. then")
|
||||
|
||||
4. **Empirical anchor attempt** — inspected pilot 2026-04-26 prompt archive to check if skills surfaced. Result: **no skill content in any pilot prompt.** Audit becomes theoretical (bias may exist but didn't affect pilot outcomes).
|
||||
|
||||
## TS source layer findings
|
||||
|
||||
### `skill-tools.ts` (41 KB, 8 tools defined)
|
||||
|
||||
**Tool definitions for the agent:**
|
||||
- `discover_skills`, `install_skill`, `create_skill`, `auto_extract_skills`, `promote_skill`, `retire_stale_skills`, `recommend_skills`, …
|
||||
- All tools have **concrete imperative descriptions** (e.g., "Search the marketplace catalog…"). No narrative voice. Model-portable.
|
||||
|
||||
**Surfacing logic:** skills are loaded from disk and provided to the orchestrator via dep injection. The actual prompt assembly (including how skill content lands in the system prompt) is **not in this file** — it's the orchestrator's concern.
|
||||
|
||||
VERDICT: Tier 0 (no bias).
|
||||
|
||||
### `skill-frontmatter.ts` (5.8 KB)
|
||||
|
||||
Pure parsing utility. YAML-ish frontmatter parser. No prompt content; cannot introduce bias.
|
||||
|
||||
VERDICT: Tier 0.
|
||||
|
||||
### `skill-recommender.ts` (9.5 KB)
|
||||
|
||||
Pure-keyword scoring with synonym clusters and bigram overlap. Multi-signal TF-IDF-inspired ranking. **Returns top-N skills by relevance score** — feeds into prompt only when caller chooses to surface them.
|
||||
|
||||
The synonym clusters (line 44-58) are model-agnostic — concept-based, not Claude-specific.
|
||||
|
||||
VERDICT: Tier 0.
|
||||
|
||||
### `skill-creator.ts` — generator template
|
||||
|
||||
Lines 26-62: `generateSkillMarkdown(template)` produces SKILL.md from a `SkillTemplate`. **Output shape is concrete + imperative** — Trigger Patterns / Steps / Tools Used / Category — no narrative voice.
|
||||
|
||||
Sample output for an auto-generated skill would have:
|
||||
```
|
||||
## Trigger Patterns
|
||||
Activate this skill when the user asks about: <triggers>
|
||||
|
||||
## Steps
|
||||
1. <imperative step>
|
||||
2. <imperative step>
|
||||
```
|
||||
|
||||
This is model-portable. Any new skill generated by the agent will be Tier 0.
|
||||
|
||||
VERDICT: Tier 0.
|
||||
|
||||
## Skill-content layer findings
|
||||
|
||||
### Hand-sampled (6 files)
|
||||
|
||||
| Skill | Category | Description shape | Body shape | Tier |
|
||||
|---|---|---|---|---|
|
||||
| `ax-gepa` | workspace, technical | Concrete trigger keywords | Imperative bullets, code patterns | **Tier 0** |
|
||||
| `agent-chronicle` | user, narrative | Narrative ("AI perspective journaling") | "We process thousands…" first-person plural; "Philosophy" section pure narrative; emoji in title | **Tier 1** |
|
||||
| `django-tdd` | user, technical | Concrete activation criteria | Procedural / code-driven | **Tier 0** |
|
||||
| `elite-longterm-memory` | user, infrastructure | Marketing copy ("Ultimate", "bulletproof") | Marketing prose; "Never lose context. Never forget decisions." emphasis stack | **Tier 1** |
|
||||
| `configure-ecc` | user, procedural | Concrete activation triggers | Imperative bullets | **Tier 0** |
|
||||
| `gsd-research-phase` | user, command | Concrete `argument-hint` | `<objective>` XML wrapper (Claude-leaning but generally portable) | **Borderline** |
|
||||
|
||||
2 of 6 hand-sampled (33%) flagged as Tier 1. Higher than the systematic scan rate because hand-sampling biased toward illustrative cases.
|
||||
|
||||
### Systematic scan (159 files, 6 regex patterns)
|
||||
|
||||
| Pattern | Hits | Notes |
|
||||
|---|---|---|
|
||||
| `CoT-imperative` ("let me", "step-by-step", etc.) | 12 | Most common bias |
|
||||
| `philosophy-keyword` | 5 | Verbose framing in body |
|
||||
| `we-think` (first-person plural) | 3 | Narrative voice |
|
||||
| `marketing-superlative` ("ultimate", "bulletproof") | 1 | Marketing copy |
|
||||
| `our-philosophy` | 1 | Narrative voice |
|
||||
| `emoji` (body emoji) | 1 | Decorative |
|
||||
| **Skills with ≥1 hit** | **20 (12.6%)** | |
|
||||
| **Skills clean** | **139 (87.4%)** | |
|
||||
|
||||
The dominant pattern is CoT-imperative phrasing — verbose multi-step instructions in narrative form. Phase 1.1 normalize won't strip this from skill content (it's content, not output artifact). Sprint 12 rewrite would be needed.
|
||||
|
||||
## Tier classification
|
||||
|
||||
### Tier 0 (model-portable, no action needed)
|
||||
- All 8 TS source files (skill-*.ts)
|
||||
- 139 of 159 SKILL.md files (87.4%)
|
||||
- Skill creator's generator template — new skills will be Tier 0 by construction
|
||||
|
||||
### Tier 1 (description/body rewrite would help — **defer to Sprint 12**)
|
||||
- 20 SKILL.md files (12.6%) with detectable narrative-voice / CoT-imperative / philosophy-keyword bias
|
||||
- These would benefit from rewrite to imperative-direct format
|
||||
- **NOT a blocker for Phase 5 NULL-baseline** (pilot didn't engage skills)
|
||||
- Cumulative refactor effort estimate: ~2-3 hours per skill × 20 skills = ~40-60 hours of focused work. Out-of-scope for current sprint.
|
||||
|
||||
### Tier 2 (real coverage gap — GEPA territory, defer to CC-2 work)
|
||||
- **No candidates surfaced from this sample.** Skills cover broad task categories (writing, coding, research, planning, decision support). The pilot's H3/H4 failures are content-quality issues in synthesis output, not gaps in skill coverage.
|
||||
|
||||
### NEW (not in original Phase 4.4 brief): skill-engagement gap
|
||||
- The 2026-04-26 pilot orchestration **didn't surface any skill content into prompts.** Recommender ran (or didn't — unclear) but no skill descriptions made it into the LLM's context.
|
||||
- This is a wiring concern, not a skill-content concern. If Phase 5 NULL-baseline runs the same scenario configuration, the issue will repeat.
|
||||
- **Recommendation:** if PM wants to test whether skill engagement materially helps Qwen vs Opus, that's a separate dedicated experiment, not part of Phase 5 NULL-baseline.
|
||||
|
||||
## Strategic implications
|
||||
|
||||
### Phase 5 NULL-baseline impact
|
||||
|
||||
NONE. The skills system was not engaged during the pilot, so no skill-bias-related signal exists in the H3/H4 deltas. Phase 5 NULL-baseline can proceed without skill work.
|
||||
|
||||
### Sprint 12 cleanup recommendation
|
||||
|
||||
Rewrite the 20 Tier 1 skills to imperative-direct format. Generator template (`skill-creator.ts`) is already model-portable, so newly-generated skills won't accumulate bias. Manual cleanup applies only to the legacy corpus.
|
||||
|
||||
Priority order (highest-impact first based on hand-sampling):
|
||||
1. `agent-chronicle` (philosophy + emoji + first-person plural — strong bias)
|
||||
2. `elite-longterm-memory` (marketing copy + "Never lose…" emphasis stack)
|
||||
3. The 12 CoT-imperative skills — bulk rewrite to bullet-imperative form
|
||||
4. The 5 philosophy-keyword skills — strip philosophical framings
|
||||
|
||||
### Out-of-scope for this audit
|
||||
|
||||
Per PM brief discipline:
|
||||
- No code changes (audit-only)
|
||||
- No skill rewrites
|
||||
- No new test coverage required (no source code modified)
|
||||
|
||||
## Audit chain
|
||||
|
||||
| Item | Value |
|
||||
|---|---|
|
||||
| Branch HEAD | `c9bda3d` (Phase 4.7 commit, unchanged) |
|
||||
| TS source files reviewed | 8 of 8 (`skill-*.ts` in `packages/agent/src/`) |
|
||||
| SKILL.md files in scope | 159 (152 user-global + 7 workspace; 3 dirs without SKILL.md) |
|
||||
| SKILL.md files hand-sampled | 6 |
|
||||
| SKILL.md files scanned (regex patterns) | 159 |
|
||||
| Bias detection rate | 12.6% (20/159) |
|
||||
| Audit cost | $0 (no LLM calls) |
|
||||
| Tests modified | 0 |
|
||||
| Code modified | 0 |
|
||||
|
||||
## PM ratification asks
|
||||
|
||||
1. **Accept Tier 1 / Tier 2 split as documented** (no Tier 2 candidates; 12.6% Tier 1 deferred to Sprint 12)?
|
||||
2. **Acknowledge the skill-engagement gap as a NEW finding** (orthogonal to the original audit scope) and decide whether to author a dedicated brief for "skill engagement during agentic synthesis" experiments? Recommendation: defer; not a Phase 5 NULL-baseline blocker.
|
||||
3. **Proceed to Phase 4.5 (tools audit)** per recommended order.
|
||||
|
||||
---
|
||||
|
||||
**End of Phase 4.4. Ready to proceed to Phase 4.5 (tools audit) per PM ratification.**
|
||||
Reference in New Issue
Block a user