waggle-os/docs/briefs/2026-04-26-harness-audit-tiered-fix-plan.md

Harness Audit Tiered Fix Plan

Date: 2026-04-26 Author: PM Status: Authored awaiting Marko ratification of Tier 1 launch (pre-launch ship) Sources: research/2026-04-26-harness-audit-comparative-analysis.md + ChatGPT deep code-level analysis + PM pilot evidence (decisions/2026-04-26-pilot-verdict-FAIL.md)

Decomposition principle: ChatGPT analiza je dobra ali full-scope plan je 6-12 nedelja. Pre-launch ne čeka. Tier 1 = launch-friendly fix-evi (3-5 dana CC-1 work). Tier 2 = Sprint 12 post-launch re-pilot. Tier 3 = KVARK quarterly enterprise governance.

---

TIER 1 — Pre-launch hardening (3-5 dana CC-1 work, $0 incremental, NOT launch-blocking)

Goal

Make harness model-aware enough that Tier 2 re-pilot can isolate "harness design issue" from "model capability issue" without re-running pilot data collection.

Scope (5 deliverables)

T1.1 — Output normalization layer

File: benchmarks/harness/src/normalize.ts (new) + integration into cells.ts scoring path

Behavior:

• Strip <think>...</think> blocks (Qwen reasoning leakage)
• Strip leading "Answer:" / "Response:" / "Final answer:" labels
• Trim whitespace + remove markdown fences
• Normalize "unknown" variants ("Unknown", "UNKNOWN", "unknown.", "N/A", "None") → "unknown"
• Remove copied metadata patterns ("[memory:synth]", "# Recalled Memories")
• Optional configurable: lowercase, strip trailing punctuation, remove articles
• Store both raw and normalized output + array of normalization actions applied

Schema addition:

interface HarnessPrediction {
  rawOutput: string;
  normalizedOutput: string;
  normalizationActions: string[];
  scoreRaw: number;
  scoreNormalized: number;
  exactMatchRaw: boolean;
  exactMatchNormalized: boolean;
}

Acceptance: unit tests covering Qwen <think> strip + abstention variant normalization + metadata copy removal. No silent over-normalization (configurable per benchmark).

T1.2 — Per-model prompt profiles

File: benchmarks/harness/src/prompt-profiles.ts (new) + cells.ts refactor to consume profiles

Profiles to ship (3 minimum):

Claude/Anthropic profile:

System: You are a knowledge work assistant. Read the provided context carefully and answer the question precisely. If the context does not contain the answer, reply with "unknown".

User: [memory or context block in markdown format]
[question]

Qwen non-thinking profile:

System: You are in direct answer mode. Do not output reasoning. Do not output <think> tags. Use the context only. Return exactly one short answer. If the answer is absent, return "unknown".

User:
CONTEXT:
{context}

QUESTION:
{question}

ANSWER:

Generic-simple profile:

System: Answer the question using the provided context.

User:
{context}

Question: {question}

Profile selection: model_id → profile mapping in config file (benchmarks/harness/config/model-profiles.json)

Acceptance: per-model profile applied automatically based on model alias. Profile override via CLI flag --prompt-profile <name>. Unit tests confirm correct profile selection.

T1.3 — Failure taxonomy classifier

File: benchmarks/harness/src/failure-classify.ts (new) + integration into report generation

Categories (10):

1. correct_answer_with_extra_text — answer present in output but with surrounding prose
2. thinking_leakage — <think> tags or visible reasoning in output
3. unknown_false_negative — model said "unknown" but ground truth is in context
4. metadata_copy — output contains [memory:synth], # Recalled Memories, or other copied formatting
5. format_violation — output structure doesn't match expected (e.g., JSON when expected span)
6. punctuation_or_case_only — answer correct after punctuation/case normalization but failed raw
7. wrong_span — extracted wrong portion of context as answer
8. wrong_entity — confused entities (e.g., named one person, ground truth is another)
9. hallucination — answer not derivable from context
10. retrieval_or_harness_error — system-side failure (not model failure)

Logic: rule-based classifier + optional LLM-judge fallback for ambiguous cases. Each failed example tagged with most-applicable category.

Acceptance: unit tests covering ≥ 1 example per category. Per-cell + per-model failure distribution reported.

T1.4 — Per-cell + per-model report generation

File: benchmarks/harness/src/report.ts (new) + replaces existing minimal aggregation

Output formats:

• JSON: benchmarks/results/<run-id>/summary.json (machine-readable)
• Markdown: benchmarks/results/<run-id>/summary.md (human-readable)
• JSONL predictions: benchmarks/results/<run-id>/predictions.jsonl
• JSONL failures: benchmarks/results/<run-id>/failures.jsonl

Required metrics per (model, cell):

• accuracy / EM (raw + normalized)
• F1 (if applicable for benchmark)
• abstention rate ("unknown" outputs / total)
• thinking leakage rate (% outputs with <think> blocks pre-normalization)
• format violation rate (% outputs failing format check)
• average output length (raw + normalized)
• average latency
• failure category distribution (10-bucket histogram)
• win/loss vs raw baseline (if applicable)
• 95% confidence interval (bootstrap, if N ≥ 30)

Required metrics per run (overall):

• per-model best-cell ranking
• cross-model comparison matrix
• regression notes (Cell X improved/degraded vs baseline)

Acceptance: sample run on existing pilot JSONL (N=12) reproduces pilot summary numbers + adds normalization columns.

T1.5 — Run artifact persistence + reproducibility

File: benchmarks/harness/src/run-meta.ts (new) + emission hook in main runner

Persisted per run:

• run_id (timestamp + git SHA prefix)
• config snapshot (cells, prompts, judges, normalization settings)
• dataset hash (SHA256 of input jsonl)
• model versions + provider routing (e.g., "qwen3.6-35b-a3b@dashscope-direct, thinking=on")
• prompt profile names per model
• random seed
• git commit SHA at run time
• timestamp (UTC + local)
• normalization actions applied (per prediction)
• raw API responses (full, not just extracted answer)
• judge call traces

Acceptance: run reproduces given identical config + dataset + seed (deterministic for greedy decoding; bounded variance for sampling).

Tier 1 effort + cost

• 3-5 dana CC-1 engineering work (single contributor)
• $0 incremental API spend (refactor + unit tests; minimal smoke testing)
• No re-pilot required; existing pilot data can be re-scored with new normalization layer to validate fix as proof-of-concept

Tier 1 acceptance criteria

• All 5 deliverables (T1.1 - T1.5) shipped to main branch
• Unit tests passing
• Existing pilot JSONL (N=12) re-scored with normalization → produces report showing per-cell normalized vs raw scores + failure taxonomy distribution
• Documentation updates to README explaining new harness capabilities
• No regression to existing benchmark suite (LoCoMo Stage 3 v6 must reproduce 74% oracle ceiling with new harness)

---

TIER 2 — Sprint 12 post-launch re-pilot (2-4 nedelje)

Goal

Test whether Tier 1 harness fix + per-model evolved prompts close H3/H4 reversal observed in 2026-04-26 pilot. Re-pilot at N=20-30 first; full N=400 only if PASS.

Scope (6 deliverables)

T2.1 — Per-model GEPA optimization

Run GEPA separately per target model:

• Claude Opus 4.7 → produces Claude-tuned evolved prompt
• Qwen 3.6 35B-A3B → produces Qwen-tuned evolved prompt (with non-thinking profile baseline)

Each evolved prompt tagged with target_model_family metadata. Evolution gates check that evolved prompt is not deployed to incompatible model family without explicit override.

T2.2 — Cross-model GEPA objective (alternative path)

Optional: single GEPA optimization with multi-model scoring objective:

score = α × claude_score + β × qwen_score - γ × variance_penalty

Where variance_penalty increases if one model improves while another regresses. Useful for "portable prompt" use case. Tag prompt as cross_model_robust if passes both per-model thresholds.

T2.3 — Robustness gates in evolution-gates.ts

Add new gate: crossModelRegressionGate. Reject candidate if any target model degrades > 2pp from baseline on golden test set, unless candidate explicitly tagged as model-specific.

Add gate: formatLeakageGate. Reject candidate if thinking_leakage_rate > 5% on Qwen-class models (or other reasoning-mode-emitting families) at evaluation time.

T2.4 — Holdout / golden / adversarial eval split

Restructure eval-dataset.ts to support 4-way split:

• train: trace-mined examples GEPA mutates against
• dev: GEPA selection signal
• holdout: final approval gate (never seen during evolution)
• golden: 5-10 critical tasks every candidate must pass

Adversarial subset (sub-set of golden):

• prompt injection attempts
• ambiguous abstention cases
• format trick cases (e.g., answer present but in non-canonical form)

T2.5 — Re-pilot N=20-30 with Tier 1 harness + Tier 2 GEPA outputs

Pre-registered manifest pilot-2026-05-XX-v1 (date TBD by Sprint 12 schedule):

• Same 3 task types (synthesis, coordination, decision support)
• 4 cells per task: Opus solo, Opus + harness (Claude-evolved prompt), Qwen solo, Qwen + harness (Qwen-evolved prompt)
• Trio-strict judge ensemble (κ recalibrate on 14-instance synthesis subset, ~$0.20)
• Pre-registered hypotheses identical to 2026-04-26 pilot:
  • H2: B - A ≥ +0.30 on ≥ 2/3 tasks
  • H3: D - C ≥ +0.30 on ≥ 2/3 tasks
  • H4: D ≥ A on ≥ 2/3 tasks
• Cost cap $20-30 (similar to original pilot envelope)
• Halt rules + amendment v2 wrapper inheritance preserved

T2.6 — Full N=400 multiplier benchmark (only if T2.5 PASS)

If T2.5 PASS, authorize full N=400 multiplier benchmark using the same Tier 1 harness + Tier 2 GEPA outputs at production scale.

Pre-registered manifest pilot-2026-05-XX-N400-v1.

Cost cap $80-150 (3 models × 4 cells × 400 instances + judge ensemble).

Output: paper-grade evidence for arxiv paper update (v2 of preprint or follow-up).

Tier 2 effort + cost

• 2-4 nedelje engineering (Marko + 1 contributor + CC-1)
• ~$30 (re-pilot N=20-30) + ~$80-150 (full N=400 if authorized) = ~$110-180 total
• κ recalibration + adversarial test corpus authoring as one-time costs (minor)

Tier 2 acceptance criteria

• All 6 deliverables (T2.1 - T2.6) shipped + tested
• Re-pilot N=20-30 H2/H3/H4 verdict ratified per pre-registration
• If PASS, full N=400 manifest authorized + executed
• arxiv paper §5.4 updated with Tier 2 evidence (v2 preprint update)

---

TIER 3 — KVARK quarterly enterprise governance (multi-quarter, post-launch)

Goal

Harden self-evolve from advanced prototype to governed agent improvement platform suitable for regulated enterprise deployment. Critical for KVARK enterprise sovereign GTM motion (locked post-launch sequencing per Decision Matrix Dimension 8).

Scope (7 deliverables — high-level only; detailed brief authored at KVARK roadmap entry)

T3.1 — Security gates (prompt injection, data exfiltration, tool misuse, policy override)

T3.2 — Tool-call trajectory evaluation (not just text output)

T3.3 — Immutable policy layer (security/compliance/permission rules cannot be evolved)

T3.4 — Deployment lifecycle (proposed → reviewed → staged → canary → production → rollback)

T3.5 — Tenant isolation + EU AI Act Article 12 audit gates

T3.6 — Human-readable diff + rationale UI for evolution candidates

T3.7 — Production-grade research/staging/production mode separation

Tier 3 effort + cost

• 1-2 quarters (multi-engineer)
• Significant engineering + design + QA + compliance review
• Detailed Tier 3 brief authored at KVARK roadmap entry, not now

Tier 3 acceptance criteria

To be defined at Tier 3 brief authoring.

---

CC-1 paste-ready brief (Tier 1 only)

For Marko to paste to CC-1 if Tier 1 ratified.

[PM-AUTHORIZE-TIER-1-HARNESS-FIX]

Tier 1 harness audit fix-evi authorized. 3-5 dana scope, $0 cost, NOT launch-blocking.

Source brief: D:\Projects\PM-Waggle-OS\briefs\2026-04-26-harness-audit-tiered-fix-plan.md (§Tier 1)
Comparative analysis: D:\Projects\PM-Waggle-OS\research\2026-04-26-harness-audit-comparative-analysis.md
Pilot evidence: D:\Projects\PM-Waggle-OS\decisions\2026-04-26-pilot-verdict-FAIL.md

Goal: make harness model-aware enough that Tier 2 re-pilot (Sprint 12 post-launch) can isolate "harness design issue" from "model capability issue" without re-running pilot data collection from scratch.

5 DELIVERABLES:

T1.1 — Output normalization layer
- New file: benchmarks/harness/src/normalize.ts
- Strip <think>...</think>, leading "Answer:" labels, markdown fences, copied metadata
- Normalize "unknown" variants
- Configurable per-benchmark (no silent over-normalization)
- Store raw + normalized + actions array
- Unit tests required

T1.2 — Per-model prompt profiles
- New file: benchmarks/harness/src/prompt-profiles.ts
- Config file: benchmarks/harness/config/model-profiles.json
- 3 profiles minimum:
  * Claude/Anthropic (existing strict-extraction style preserved as-is)
  * Qwen non-thinking (simple CONTEXT/QUESTION/ANSWER format, explicit no-reasoning instruction, no markdown metadata in memory format)
  * Generic-simple (minimal scaffolding, fallback for new models)
- Auto-selected by model alias; CLI override --prompt-profile <name>
- Refactor cells.ts to consume profiles instead of hardcoded prompts

T1.3 — Failure taxonomy classifier
- New file: benchmarks/harness/src/failure-classify.ts
- 10 categories: correct_answer_with_extra_text, thinking_leakage, unknown_false_negative, metadata_copy, format_violation, punctuation_or_case_only, wrong_span, wrong_entity, hallucination, retrieval_or_harness_error
- Rule-based classifier + optional LLM-judge fallback for ambiguous
- Per-cell + per-model failure distribution in reports

T1.4 — Per-cell + per-model report generation
- New file: benchmarks/harness/src/report.ts
- Outputs: summary.json + summary.md + predictions.jsonl + failures.jsonl
- Metrics per (model, cell): EM raw + normalized, F1, abstention rate, thinking leakage rate, format violation rate, avg output length, avg latency, failure category distribution, win/loss vs baseline, bootstrap CI if N≥30
- Cross-model comparison matrix
- Reproduces existing pilot summary on N=12 data when re-scored

T1.5 — Run artifact persistence + reproducibility
- New file: benchmarks/harness/src/run-meta.ts
- Per-run persistence: run_id, config snapshot, dataset SHA256, model versions, prompt profile names, seed, git SHA, timestamp, normalization actions per prediction, raw API responses, judge traces
- Deterministic reproduction for greedy decoding

ACCEPTANCE:
- All 5 deliverables shipped to main
- Unit tests passing
- Existing pilot JSONL (benchmarks/results/pilot-2026-04-26/) re-scored with new normalization → produces report showing per-cell normalized vs raw scores + failure taxonomy
- README updated explaining new harness capabilities
- Stage 3 v6 LoCoMo benchmark reproduces 74% oracle ceiling with refactored harness (no regression)

VALIDATION TASK:
After T1.1-T1.5 ship, re-score the existing 2026-04-26 pilot JSONL (12 cells × 3 judges = 36 records) using the new normalization layer and failure classifier. Output a delta report showing:
- Per-cell raw vs normalized score
- Failure category distribution per cell
- Specifically: how many of the 8 H2/H3/H4 FAIL cells (Tasks 2+3 H2, all Tasks H3, all Tasks H4) had:
  * thinking_leakage failures (would be removed by T1.1)
  * unknown_false_negative failures (would be flagged by T1.3)
  * metadata_copy failures (would be removed by T1.1)
  * format_violation failures (would be flagged by T1.3)

This delta report is the empirical evidence for whether Tier 1 harness fix substantively addresses pilot H3/H4 reversal — input for Sprint 12 Tier 2 re-pilot decision.

Cost ceiling: $0 incremental for T1.1-T1.5 implementation + unit tests. Re-scoring existing JSONL is local computation, no API calls. New API calls only if optional LLM-judge fallback in T1.3 is invoked on edge cases (cap at $5 for that path).

DELIVERY EXPECTATION:
- Day 1: T1.1 (normalization) + T1.3 (failure taxonomy) + unit tests
- Day 2: T1.2 (prompt profiles) + cells.ts refactor
- Day 3: T1.4 (report generation) + T1.5 (run artifacts)
- Day 4: Validation task — re-score pilot JSONL, produce delta report
- Day 5: Documentation updates + smoke test on Stage 3 v6 LoCoMo to confirm no regression

Halt-and-ping triggers:
- LoCoMo Stage 3 v6 76% → 74% reproduction shows >2pp regression with new harness (config issue; halt before main merge)
- Re-scored pilot deltas show Tier 1 fix-evi do NOT substantively change H3/H4 pattern (i.e., harness-Opus-bias hipoteza is partially refuted by Tier 1 alone, requires Tier 2 GEPA per-model variant); this is informational ne halt — flag for Sprint 12 brief authoring

Standing GREEN. Proceed with T1.1-T1.5. PM ratification of any architectural decisions surfaced during implementation requested via halt-and-ping.

---

§Open questions for Marko

1. Ship Tier 1 pre-launch? PM rec: YES. Y/N
2. Tier 1 timing — parallel with launch comms work, or before? PM rec: parallel. Confirm or override.
3. Tier 2 schedule — Sprint 12 (post-launch) or accelerate? PM rec: Sprint 12. Confirm.
4. Tier 3 — file as KVARK quarterly entry? PM rec: yes, separate workstream from consumer Waggle launch. Confirm.
5. Send paste-ready Tier 1 prompt to CC-1 now, or wait for additional ratification? PM rec: send now if 1-4 ratified.