This commit is contained in:
Oleg Maslov
204
docs/decisions/2026-04-28-phase-4-5-tools-audit-results.md
Normal file
204
docs/decisions/2026-04-28-phase-4-5-tools-audit-results.md
Normal file
@@ -0,0 +1,204 @@
|
||||
---
|
||||
decision_id: 2026-04-28-phase-4-5-tools-audit-results
|
||||
date: 2026-04-28
|
||||
phase: 4.5 tools audit sweep
|
||||
verdict: tools layer is essentially Tier 0 (99.0% bias-free). Empirical pilot signal: Qwen retrieval cells made 43% fewer tool calls than Opus — but this is a model-behavior gap, NOT a tool-description-format issue. Defer to CC-2 GEPA Tier 2 work.
|
||||
predecessor: 2026-04-28-phase-4-4-skills-audit-results.md
|
||||
sprint_plan: D:\Projects\waggle-os\decisions\2026-04-26-agent-fix-sprint-plan.md
|
||||
---
|
||||
|
||||
# Phase 4.5 Tools Audit Sweep — Results
|
||||
|
||||
## TL;DR
|
||||
|
||||
Audited 22 tool TS source files, 312 individual tool descriptions: **only 3 (1.0%) have any narrative-bias pattern**, all 3 in `skill-tools.ts` and trivial. The function-calling tools layer is essentially Tier 0 by construction.
|
||||
|
||||
**Empirical signal from pilot 2026-04-26:** unlike skills (which weren't surfaced at all), **the retrieval tool WAS engaged.** Qwen retrieval cells made systematically FEWER calls than Opus on every task:
|
||||
|
||||
| Task | Opus retrieval (B) | Qwen retrieval (D) | Gap |
|
||||
|---|---|---|---|
|
||||
| Task 1 | 2 calls / 3 steps | 1 call / 2 steps | −50% |
|
||||
| Task 2 | 2 calls / 3 steps (loop_exhausted) | 1 call / 2 steps | −50% |
|
||||
| Task 3 | 3 calls / 4 steps (loop_exhausted) | 2 calls / 4 steps | −33% |
|
||||
| **Average** | **2.33 calls** | **1.33 calls** | **−43%** |
|
||||
|
||||
This connects directly to the H4 deltas (Qwen retrieval scored lower than Opus retrieval on every task). Qwen's under-engagement with retrieval → less context → weaker synthesis → lower judge scores.
|
||||
|
||||
**But this is NOT a tool-description-format issue.** The multi-step retrieval contract (`MULTI_STEP_ACTION_CONTRACT`) is rendered identically across all 5 prompt shapes (claude, qwen-thinking, qwen-non-thinking, gpt, generic-simple) — same JSON action contract, same per-turn budget, same query guidance. Both Opus and Qwen saw the SAME tool description; they exhibited DIFFERENT behaviors.
|
||||
|
||||
**Tier classification:**
|
||||
- **Tier 0** (no action): 99.0% of tool descriptions, all 22 tool files, the multi-step retrieval contract
|
||||
- **Tier 1** (description rewrite would help): 3 minor borderline cases in skill-tools.ts, already covered by Phase 4.4
|
||||
- **Tier 2** (real behavioral gap, GEPA territory): **Qwen under-retrieves vs Opus.** Empirically anchored to pilot data. Defer to CC-2 GEPA Tier 2 work — they should target retrieval-engagement prompts during Faza 1 evolution.
|
||||
|
||||
Cumulative cost: **$0** (pure code+regex audit).
|
||||
|
||||
## Methodology
|
||||
|
||||
Per Option A discipline (audit-only, no refactor):
|
||||
|
||||
1. **TS source inventory** — 22 tool files in `packages/agent/src/`:
|
||||
- `tools.ts` (32 KB, 29 descriptions)
|
||||
- `system-tools.ts` (39 KB, 46 descriptions)
|
||||
- `git-tools.ts` (11 KB, 32 descriptions)
|
||||
- `team-tools.ts` (13 KB, 30 descriptions)
|
||||
- `skill-tools.ts` (41 KB, 28 descriptions)
|
||||
- 17 more domain-specific files
|
||||
|
||||
2. **Systematic description scan** — regex-extracted all `description: '<string>'` fields (single-line, ≥5 chars), 312 total. Applied 4 bias-pattern detectors:
|
||||
- `cot-imperative` ("let me", "step-by-step", "carefully consider", "think about")
|
||||
- `narrative-voice` ("you should", "you might", "you'll want to", "consider whether")
|
||||
- `verbose-hedging` ("generally", "typically", "usually", "might be", "could be")
|
||||
- `meta-reference` ("this tool will", "this function", "the agent should", "the model should")
|
||||
|
||||
3. **Multi-step retrieval contract review** — read `MULTI_STEP_ACTION_CONTRACT` constant in `prompt-shapes/types.ts`. Verified it's referenced identically by all 5 prompt shapes.
|
||||
|
||||
4. **Pilot empirical anchor** — extracted retrieval_calls + steps_taken from all 12 cells of pilot 2026-04-26. Compared Opus retrieval (B) vs Qwen retrieval (D) per task.
|
||||
|
||||
5. **Pilot prompt trace inspection** — read `task-1-cell-B-trace.md` to confirm what tool surface the retrieval loop actually presented to the LLM.
|
||||
|
||||
## Findings
|
||||
|
||||
### TS source layer (function-calling tools)
|
||||
|
||||
| File | Descriptions | Biased | % |
|
||||
|---|---|---|---|
|
||||
| skill-tools.ts | 28 | 3 | 10.7% |
|
||||
| All other 21 files | 284 | 0 | 0% |
|
||||
| **Total** | **312** | **3** | **1.0%** |
|
||||
|
||||
**Per-pattern breakdown:**
|
||||
- `narrative-voice`: 2 hits (both in skill-tools.ts)
|
||||
- `cot-imperative`: 1 hit (in skill-tools.ts)
|
||||
- `verbose-hedging`: 0 hits
|
||||
- `meta-reference`: 0 hits
|
||||
|
||||
**Sample biased descriptions (all 3):**
|
||||
|
||||
```
|
||||
[skill-tools.ts] (narrative-voice)
|
||||
"Create a new reusable skill from a workflow description or raw markdown content.
|
||||
Skills are loaded into your system prompt and persist across sessions. You can
|
||||
provide either raw `content` (markdown) ..."
|
||||
|
||||
[skill-tools.ts] (cot-imperative)
|
||||
"Step-by-step workflow instructions..." ← param description, not tool description
|
||||
|
||||
[skill-tools.ts] (narrative-voice)
|
||||
"Search for skills and tools you might need. Searches installed skills by
|
||||
content/name, and suggests built-in capabilities. Use when the user asks you
|
||||
to do something and you want to check if you have ..."
|
||||
```
|
||||
|
||||
These are minor. The "you might need" / "you can provide" / "step-by-step" hits are mild narrative-voice that an imperative-direct rewrite could clean up without semantic loss.
|
||||
|
||||
VERDICT: **Tier 1 (description rewrite would help) for these 3 cases — but they're already covered by Phase 4.4's recommendation since they live in skill-tools.ts.** No NEW Tier 1 work needed.
|
||||
|
||||
### Multi-step retrieval contract (the actual pilot tool surface)
|
||||
|
||||
```ts
|
||||
export const MULTI_STEP_ACTION_CONTRACT = `Output exactly ONE JSON object on its own line, no prose, no code fences:
|
||||
- To retrieve information: {"action": "retrieve", "query": "<your search query>"}
|
||||
- To finalize your answer: {"action": "finalize", "response": "<your full final answer>"}`;
|
||||
```
|
||||
|
||||
This is referenced by ALL 5 prompt shapes (claude.ts / qwen-thinking.ts / qwen-non-thinking.ts / gpt.ts / generic-simple.ts) at the same insertion point. Opus and Qwen saw byte-identical contract surface during the pilot.
|
||||
|
||||
VERDICT: **Tier 0.** Imperative-direct, no narrative voice, no hedging. Model-portable.
|
||||
|
||||
### Pilot retrieval engagement empirical signal
|
||||
|
||||
Pilot prompt-archive confirms identical surface for Opus B and Qwen D cells (read `task-1-cell-B-trace.md`):
|
||||
|
||||
```
|
||||
You have access to a private corpus of materials about this scenario via a retrieval tool.
|
||||
You CANNOT see the materials directly. You must request retrievals to get information.
|
||||
|
||||
On EACH turn, output exactly ONE JSON object on its own line, no prose, no code fences:
|
||||
- To retrieve information, output: {"action": "retrieve", "query": "<your search query>"}
|
||||
- To finalize your answer, output: {"action": "finalize", "response": "<your full final answer>"}
|
||||
|
||||
You have a maximum of 5 turns. Plan accordingly.
|
||||
Each retrieval returns up to 8 most relevant document chunks.
|
||||
Be focused: a good retrieval query is 5-15 words and targets specific information.
|
||||
```
|
||||
|
||||
Same prompt-shape rendered the contract identically for both subjects. **The tool surface is not the variable.**
|
||||
|
||||
But behavior diverged sharply:
|
||||
|
||||
| Cell | Model | retrieval_calls | steps | loop_exhausted | trio_mean |
|
||||
|---|---|---|---|---|---|
|
||||
| Task 1 / B | Opus | 2 | 3 | false | 4.94 |
|
||||
| Task 1 / D | Qwen | 1 | 2 | false | 4.39 |
|
||||
| Task 2 / B | Opus | 2 | 3 | **true** | 5.00 |
|
||||
| Task 2 / D | Qwen | 1 | 2 | false | 3.94 |
|
||||
| Task 3 / B | Opus | 3 | 4 | **true** | 4.89 |
|
||||
| Task 3 / D | Qwen | 2 | 4 | false | 4.56 |
|
||||
|
||||
Three observations:
|
||||
|
||||
1. **Qwen used the retrieval tool ~half as often as Opus** (1.33 avg vs 2.33 avg). Same surface, different behavior.
|
||||
|
||||
2. **Opus exhausted maxSteps in 2 of 3 retrieval runs** (Tasks 2 + 3, loop_exhausted=true). Opus wanted MORE retrievals than the 5-turn budget allowed; Qwen never exhausted. This is consistent with "Opus engages retrieval aggressively, Qwen finalizes early."
|
||||
|
||||
3. **Qwen retrieval (D) scored LOWER than Opus retrieval (B) on every task** (deltas: −0.55 / −1.06 / −0.33 = mean −0.65). The under-retrieval correlates with the lower scores.
|
||||
|
||||
This is the same H4 gap that Phase 4.3's rationale analysis classified as 100% Tier 2 (semantic / content). Now we have a complementary mechanistic signal: the gap manifests at least partly through under-engagement with the retrieval tool. The cause isn't WHAT the tool description says — it's HOW Qwen interprets the retrieval contract relative to its own confidence threshold for finalizing.
|
||||
|
||||
## Tier classification (final)
|
||||
|
||||
### Tier 0 (no action needed)
|
||||
- `MULTI_STEP_ACTION_CONTRACT` — model-portable, used by all 5 prompt shapes
|
||||
- 309 of 312 (99.0%) function-calling tool descriptions across 22 tool files
|
||||
- Tool surface format consistency across Opus / Qwen / GPT / generic — identical
|
||||
|
||||
### Tier 1 (description rewrite — already covered by Phase 4.4)
|
||||
- 3 minor borderline cases in `skill-tools.ts` — bundle into Phase 4.4's Sprint 12 cleanup. No NEW Tier 1 work specific to tools.
|
||||
|
||||
### Tier 2 (behavioral gap — defer to CC-2 GEPA work)
|
||||
- **Qwen under-retrieves vs Opus by ~43% on every task.**
|
||||
- This is empirically anchored to pilot 2026-04-26 data — not theoretical.
|
||||
- It's NOT a tool-description-format problem (surface is identical for both models).
|
||||
- It IS a prompting-strategy / confidence-calibration problem that GEPA evolution should target.
|
||||
- **Recommendation for CC-2 Faza 1:** during prompt evolution, include a metric or rubric component that rewards retrieval-engagement (or penalizes premature finalization) on Qwen-targeted shapes. Closing this behavioral gap should partially close the H4 score delta.
|
||||
|
||||
## Strategic implications
|
||||
|
||||
### Phase 5 NULL-baseline impact
|
||||
|
||||
NONE for tool-description bias (essentially zero detected).
|
||||
|
||||
For tool-engagement behavior: Phase 5 NULL-baseline will REPRODUCE the Qwen under-retrieval pattern (same prompt-shape, same multi-step contract). This is expected and serves as the baseline against which CC-2's GEPA-evolved Phase 5 variant will be measured. Specifically, the GEPA-evolved variant should show:
|
||||
- Qwen retrieval_calls ≥ Opus retrieval_calls per task (engagement parity)
|
||||
- Qwen H4 trio_mean delta from Opus narrowed by ≥ 0.30 points (score parity proxy)
|
||||
|
||||
If GEPA achieves both, the sovereign multiplier teza is rescued. If only the first (engagement) but not the second (score), we've decoupled tool engagement from synthesis quality — a different and harder problem.
|
||||
|
||||
### Sprint 12 cleanup
|
||||
|
||||
The 3 skill-tools.ts borderline cases overlap with Phase 4.4's Sprint 12 cleanup recommendation. No NEW tools-layer rewrite work needed.
|
||||
|
||||
## Audit chain
|
||||
|
||||
| Item | Value |
|
||||
|---|---|
|
||||
| Branch HEAD | `c9bda3d` (Phase 4.7, unchanged) |
|
||||
| Tool TS files reviewed | 22 of 22 |
|
||||
| Tool descriptions scanned | 312 |
|
||||
| Bias detection rate | 1.0% (3/312), all in skill-tools.ts |
|
||||
| Multi-step contract reviewed | yes (`prompt-shapes/types.ts:MULTI_STEP_ACTION_CONTRACT`) |
|
||||
| Pilot empirical signal | retrieval_calls Opus 2.33 / Qwen 1.33 across 3 tasks |
|
||||
| Audit cost | $0 |
|
||||
| Tests modified | 0 |
|
||||
| Code modified | 0 |
|
||||
|
||||
## PM ratification asks
|
||||
|
||||
1. **Accept Tier 0 / Tier 1 / Tier 2 split** as documented (no tools-layer Tier 1 work needed; 3 borderline cases bundle with Phase 4.4 Sprint 12 cleanup; Tier 2 = Qwen retrieval engagement gap)?
|
||||
2. **Forward the Qwen retrieval engagement signal to CC-2** for Faza 1 GEPA prompt evolution (concrete: include retrieval-engagement metric or anti-premature-finalization penalty in the metric)?
|
||||
3. **Phase 4 sweep (4.4 + 4.5) complete.** Halt for PM checkpoint per kickoff brief — Phase 5 NULL-baseline kickoff blocked on (a) CC-2 GEPA Faza 1 Checkpoint A, (b) Memory Sync Marko-side activation status, (c) updated Phase 5 brief from PM.
|
||||
|
||||
---
|
||||
|
||||
**End of Phase 4.5. Phase 4 sweep complete. Standing HALTED at PM checkpoint per Phase 4 kickoff brief.**
|
||||
Reference in New Issue
Block a user