oleg.m/waggle-os
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
master
waggle-os/docs/decisions/2026-04-28-phase-4-5-tools-audit-results.md
Oleg Maslov 0c3e2ead3b
Some checks failed
Installer Smoke / installer-smoke (push) Has been cancelled
Details
moving
2026-09-02 10:10:29 +02:00

12 KiB
Raw Permalink Blame History

decision_id: 2026-04-28-phase-4-5-tools-audit-results date: 2026-04-28 phase: 4.5 tools audit sweep verdict: tools layer is essentially Tier 0 (99.0% bias-free). Empirical pilot signal: Qwen retrieval cells made 43% fewer tool calls than Opus — but this is a model-behavior gap, NOT a tool-description-format issue. Defer to CC-2 GEPA Tier 2 work. predecessor: 2026-04-28-phase-4-4-skills-audit-results.md sprint_plan: D:\Projects\waggle-os\decisions\2026-04-26-agent-fix-sprint-plan.md

Phase 4.5 Tools Audit Sweep — Results

TL;DR

Audited 22 tool TS source files, 312 individual tool descriptions: only 3 (1.0%) have any narrative-bias pattern, all 3 in skill-tools.ts and trivial. The function-calling tools layer is essentially Tier 0 by construction.

Empirical signal from pilot 2026-04-26: unlike skills (which weren't surfaced at all), the retrieval tool WAS engaged. Qwen retrieval cells made systematically FEWER calls than Opus on every task:

Task Opus retrieval (B) Qwen retrieval (D) Gap
Task 1 2 calls / 3 steps 1 call / 2 steps −50%
Task 2 2 calls / 3 steps (loop_exhausted) 1 call / 2 steps −50%
Task 3 3 calls / 4 steps (loop_exhausted) 2 calls / 4 steps −33%
Average 2.33 calls 1.33 calls −43%

This connects directly to the H4 deltas (Qwen retrieval scored lower than Opus retrieval on every task). Qwen's under-engagement with retrieval → less context → weaker synthesis → lower judge scores.

But this is NOT a tool-description-format issue. The multi-step retrieval contract (MULTI_STEP_ACTION_CONTRACT) is rendered identically across all 5 prompt shapes (claude, qwen-thinking, qwen-non-thinking, gpt, generic-simple) — same JSON action contract, same per-turn budget, same query guidance. Both Opus and Qwen saw the SAME tool description; they exhibited DIFFERENT behaviors.

Tier classification:

  • Tier 0 (no action): 99.0% of tool descriptions, all 22 tool files, the multi-step retrieval contract
  • Tier 1 (description rewrite would help): 3 minor borderline cases in skill-tools.ts, already covered by Phase 4.4
  • Tier 2 (real behavioral gap, GEPA territory): Qwen under-retrieves vs Opus. Empirically anchored to pilot data. Defer to CC-2 GEPA Tier 2 work — they should target retrieval-engagement prompts during Faza 1 evolution.

Cumulative cost: $0 (pure code+regex audit).

Methodology

Per Option A discipline (audit-only, no refactor):

  1. TS source inventory — 22 tool files in packages/agent/src/:

    • tools.ts (32 KB, 29 descriptions)
    • system-tools.ts (39 KB, 46 descriptions)
    • git-tools.ts (11 KB, 32 descriptions)
    • team-tools.ts (13 KB, 30 descriptions)
    • skill-tools.ts (41 KB, 28 descriptions)
    • 17 more domain-specific files

  2. Systematic description scan — regex-extracted all description: '<string>' fields (single-line, ≥5 chars), 312 total. Applied 4 bias-pattern detectors:

    • cot-imperative ("let me", "step-by-step", "carefully consider", "think about")
    • narrative-voice ("you should", "you might", "you'll want to", "consider whether")
    • verbose-hedging ("generally", "typically", "usually", "might be", "could be")
    • meta-reference ("this tool will", "this function", "the agent should", "the model should")

  3. Multi-step retrieval contract review — read MULTI_STEP_ACTION_CONTRACT constant in prompt-shapes/types.ts. Verified it's referenced identically by all 5 prompt shapes.

  4. Pilot empirical anchor — extracted retrieval_calls + steps_taken from all 12 cells of pilot 2026-04-26. Compared Opus retrieval (B) vs Qwen retrieval (D) per task.

  5. Pilot prompt trace inspection — read task-1-cell-B-trace.md to confirm what tool surface the retrieval loop actually presented to the LLM.

Findings

TS source layer (function-calling tools)

File Descriptions Biased %
skill-tools.ts 28 3 10.7%
All other 21 files 284 0 0%
Total 312 3 1.0%

Per-pattern breakdown:

  • narrative-voice: 2 hits (both in skill-tools.ts)
  • cot-imperative: 1 hit (in skill-tools.ts)
  • verbose-hedging: 0 hits
  • meta-reference: 0 hits

Sample biased descriptions (all 3):

[skill-tools.ts] (narrative-voice)
  "Create a new reusable skill from a workflow description or raw markdown content.
   Skills are loaded into your system prompt and persist across sessions. You can
   provide either raw `content` (markdown) ..."

[skill-tools.ts] (cot-imperative)
  "Step-by-step workflow instructions..."   ← param description, not tool description

[skill-tools.ts] (narrative-voice)
  "Search for skills and tools you might need. Searches installed skills by
   content/name, and suggests built-in capabilities. Use when the user asks you
   to do something and you want to check if you have ..."

These are minor. The "you might need" / "you can provide" / "step-by-step" hits are mild narrative-voice that an imperative-direct rewrite could clean up without semantic loss.

VERDICT: Tier 1 (description rewrite would help) for these 3 cases — but they're already covered by Phase 4.4's recommendation since they live in skill-tools.ts. No NEW Tier 1 work needed.

Multi-step retrieval contract (the actual pilot tool surface)

export const MULTI_STEP_ACTION_CONTRACT = `Output exactly ONE JSON object on its own line, no prose, no code fences:
  - To retrieve information: {"action": "retrieve", "query": "<your search query>"}
  - To finalize your answer:  {"action": "finalize", "response": "<your full final answer>"}`;

This is referenced by ALL 5 prompt shapes (claude.ts / qwen-thinking.ts / qwen-non-thinking.ts / gpt.ts / generic-simple.ts) at the same insertion point. Opus and Qwen saw byte-identical contract surface during the pilot.

VERDICT: Tier 0. Imperative-direct, no narrative voice, no hedging. Model-portable.

Pilot retrieval engagement empirical signal

Pilot prompt-archive confirms identical surface for Opus B and Qwen D cells (read task-1-cell-B-trace.md):

You have access to a private corpus of materials about this scenario via a retrieval tool.
You CANNOT see the materials directly. You must request retrievals to get information.

On EACH turn, output exactly ONE JSON object on its own line, no prose, no code fences:
  - To retrieve information, output: {"action": "retrieve", "query": "<your search query>"}
  - To finalize your answer, output: {"action": "finalize", "response": "<your full final answer>"}

You have a maximum of 5 turns. Plan accordingly.
Each retrieval returns up to 8 most relevant document chunks.
Be focused: a good retrieval query is 5-15 words and targets specific information.

Same prompt-shape rendered the contract identically for both subjects. The tool surface is not the variable.

But behavior diverged sharply:

Cell Model retrieval_calls steps loop_exhausted trio_mean
Task 1 / B Opus 2 3 false 4.94
Task 1 / D Qwen 1 2 false 4.39
Task 2 / B Opus 2 3 true 5.00
Task 2 / D Qwen 1 2 false 3.94
Task 3 / B Opus 3 4 true 4.89
Task 3 / D Qwen 2 4 false 4.56

Three observations:

  1. Qwen used the retrieval tool ~half as often as Opus (1.33 avg vs 2.33 avg). Same surface, different behavior.

  2. Opus exhausted maxSteps in 2 of 3 retrieval runs (Tasks 2 + 3, loop_exhausted=true). Opus wanted MORE retrievals than the 5-turn budget allowed; Qwen never exhausted. This is consistent with "Opus engages retrieval aggressively, Qwen finalizes early."

  3. Qwen retrieval (D) scored LOWER than Opus retrieval (B) on every task (deltas: −0.55 / −1.06 / −0.33 = mean −0.65). The under-retrieval correlates with the lower scores.

This is the same H4 gap that Phase 4.3's rationale analysis classified as 100% Tier 2 (semantic / content). Now we have a complementary mechanistic signal: the gap manifests at least partly through under-engagement with the retrieval tool. The cause isn't WHAT the tool description says — it's HOW Qwen interprets the retrieval contract relative to its own confidence threshold for finalizing.

Tier classification (final)

Tier 0 (no action needed)

  • MULTI_STEP_ACTION_CONTRACT — model-portable, used by all 5 prompt shapes
  • 309 of 312 (99.0%) function-calling tool descriptions across 22 tool files
  • Tool surface format consistency across Opus / Qwen / GPT / generic — identical

Tier 1 (description rewrite — already covered by Phase 4.4)

  • 3 minor borderline cases in skill-tools.ts — bundle into Phase 4.4's Sprint 12 cleanup. No NEW Tier 1 work specific to tools.

Tier 2 (behavioral gap — defer to CC-2 GEPA work)

  • Qwen under-retrieves vs Opus by ~43% on every task.
  • This is empirically anchored to pilot 2026-04-26 data — not theoretical.
  • It's NOT a tool-description-format problem (surface is identical for both models).
  • It IS a prompting-strategy / confidence-calibration problem that GEPA evolution should target.
  • Recommendation for CC-2 Faza 1: during prompt evolution, include a metric or rubric component that rewards retrieval-engagement (or penalizes premature finalization) on Qwen-targeted shapes. Closing this behavioral gap should partially close the H4 score delta.

Strategic implications

Phase 5 NULL-baseline impact

NONE for tool-description bias (essentially zero detected).

For tool-engagement behavior: Phase 5 NULL-baseline will REPRODUCE the Qwen under-retrieval pattern (same prompt-shape, same multi-step contract). This is expected and serves as the baseline against which CC-2's GEPA-evolved Phase 5 variant will be measured. Specifically, the GEPA-evolved variant should show:

  • Qwen retrieval_calls ≥ Opus retrieval_calls per task (engagement parity)
  • Qwen H4 trio_mean delta from Opus narrowed by ≥ 0.30 points (score parity proxy)

If GEPA achieves both, the sovereign multiplier teza is rescued. If only the first (engagement) but not the second (score), we've decoupled tool engagement from synthesis quality — a different and harder problem.

Sprint 12 cleanup

The 3 skill-tools.ts borderline cases overlap with Phase 4.4's Sprint 12 cleanup recommendation. No NEW tools-layer rewrite work needed.

Audit chain

Item Value
Branch HEAD c9bda3d (Phase 4.7, unchanged)
Tool TS files reviewed 22 of 22
Tool descriptions scanned 312
Bias detection rate 1.0% (3/312), all in skill-tools.ts
Multi-step contract reviewed yes (prompt-shapes/types.ts:MULTI_STEP_ACTION_CONTRACT)
Pilot empirical signal retrieval_calls Opus 2.33 / Qwen 1.33 across 3 tasks
Audit cost $0
Tests modified 0
Code modified 0

PM ratification asks

  1. Accept Tier 0 / Tier 1 / Tier 2 split as documented (no tools-layer Tier 1 work needed; 3 borderline cases bundle with Phase 4.4 Sprint 12 cleanup; Tier 2 = Qwen retrieval engagement gap)?
  2. Forward the Qwen retrieval engagement signal to CC-2 for Faza 1 GEPA prompt evolution (concrete: include retrieval-engagement metric or anti-premature-finalization penalty in the metric)?
  3. Phase 4 sweep (4.4 + 4.5) complete. Halt for PM checkpoint per kickoff brief — Phase 5 NULL-baseline kickoff blocked on (a) CC-2 GEPA Faza 1 Checkpoint A, (b) Memory Sync Marko-side activation status, (c) updated Phase 5 brief from PM.

End of Phase 4.5. Phase 4 sweep complete. Standing HALTED at PM checkpoint per Phase 4 kickoff brief.

Reference in New Issue View Git Blame Copy Permalink