This commit is contained in:
Oleg Maslov
237
decisions/2026-04-26-agent-fix-sprint-plan.md
Normal file
237
decisions/2026-04-26-agent-fix-sprint-plan.md
Normal file
@@ -0,0 +1,237 @@
|
||||
---
|
||||
decision_id: 2026-04-26-agent-fix-sprint-plan
|
||||
date: 2026-04-26
|
||||
authority: PM (Marko) — sprint authorized post-pilot-FAIL
|
||||
type: sprint phasing proposal — PM ratification requested before phase 1 kick
|
||||
predecessor: decisions/2026-04-26-pilot-verdict-FAIL.md
|
||||
---
|
||||
|
||||
# Agent-Fix Sprint — Phasing Plan
|
||||
|
||||
**Goal:** most powerful model-agnostic agent achievable. Bolji od plain prompt-a na svakom modelu (Opus 4.7, Qwen 3.6 35B-A3B, GPT-5.4, future). Production agent (`packages/agent/`) i benchmark setup (`benchmarks/harness/`) konsolidovani — single source of truth.
|
||||
|
||||
**Constraints:**
|
||||
- 1-2 weeks PRIMARY (5-10 working days)
|
||||
- SECONDARY (GEPA) tek posle PRIMARY PASS via re-pilot
|
||||
- Substrate claim (Stage 3 v6 oracle 74%) NE SME da regresuje
|
||||
- Cross-model robustness gate blokira candidate koji improve Claude ali degraduje Qwen >2pp
|
||||
|
||||
---
|
||||
|
||||
## Phase ordering — dependency-driven
|
||||
|
||||
```
|
||||
Phase 1 — Foundations (independent, parallel-safe)
|
||||
Item 1: output-normalize.ts (no deps; pure utility)
|
||||
Item 2: prompt-shapes/ (no deps; pure config + logic)
|
||||
Item 7: run-meta.ts (no deps; mostly utility)
|
||||
|
||||
Phase 2 — Architectural consolidation
|
||||
Item 3: agent-loop unification (consumes Items 1 + 2)
|
||||
— pull scripts/run-pilot-2026-04-26.ts logic into
|
||||
packages/agent/src/agent-loop.ts
|
||||
— deprecate hardcoded "compressed" scaffold in
|
||||
benchmarks/harness/src/cells.ts; refactor to consume
|
||||
packages/agent/ public API
|
||||
|
||||
Phase 3 — Resilience layer
|
||||
Item 4: long-task/{checkpoint,recovery,context-manager}.ts
|
||||
(extends Item 3)
|
||||
|
||||
Phase 4 — Quality refinements
|
||||
Item 5: skills + tools audit (model-aware refactor; uses Item 2)
|
||||
Item 6: failure-classify.ts + report.ts (uses Item 1)
|
||||
|
||||
Phase 5 — Validation
|
||||
Mini re-pilot N=12-20 (5 cells × 3-5 tasks)
|
||||
Acceptance gates from work order
|
||||
|
||||
→ HALT for PM ratification
|
||||
→ SECONDARY: GEPA evaluation with labeled corpus (separate sprint)
|
||||
```
|
||||
|
||||
Phases 1, 4 can have parallel sub-items. Phases 2, 3, 5 are sequential.
|
||||
|
||||
---
|
||||
|
||||
## Phase 1 detail — proposed deliverables (3-4 days)
|
||||
|
||||
### 1.1 — `packages/agent/src/output-normalize.ts` (NEW)
|
||||
|
||||
**API surface:**
|
||||
```typescript
|
||||
export interface NormalizationConfig {
|
||||
stripThinkTags: boolean; // <think>...</think> for Qwen
|
||||
stripAnswerLabels: boolean; // "Answer:", "Final answer:"
|
||||
stripMarkdownFences: boolean; // ```...``` in non-code outputs
|
||||
stripCopiedMetadata: boolean; // [memory:synth], # Recalled Memories
|
||||
unknownAliases: string[]; // → "unknown"
|
||||
preset?: 'production' | 'benchmark-strict' | 'benchmark-lenient';
|
||||
}
|
||||
|
||||
export interface NormalizationResult {
|
||||
raw: string;
|
||||
normalized: string;
|
||||
actions: Array<{ rule: string; before: string; after: string }>;
|
||||
}
|
||||
|
||||
export function normalize(text: string, config: NormalizationConfig): NormalizationResult;
|
||||
export const PRESETS: Record<string, NormalizationConfig>;
|
||||
```
|
||||
|
||||
**Hard rule (from work order):** "unknown" → "" silently strips legitimate abstention signal — DON'T DO. Map "unknown" / "Unknown" / "Unknown." / "N/A" → canonical "unknown" (preserved).
|
||||
|
||||
**Test coverage:** all rules unit-tested with raw/normalized/actions audit; round-trip property tests.
|
||||
|
||||
### 1.2 — `packages/agent/src/prompt-shapes/` (NEW directory)
|
||||
|
||||
```
|
||||
prompt-shapes/
|
||||
├── claude.ts — narrative + explicit step instruction; works well with thinking
|
||||
├── qwen-thinking.ts — minimal scaffolding; thinking handles its own structure
|
||||
├── qwen-non-thinking.ts — explicit step instruction; structured output template
|
||||
├── gpt.ts — terse + structured; reasoning model defaults
|
||||
├── generic-simple.ts — fallback; works for any model
|
||||
├── selector.ts — auto-select by model alias; CLI override
|
||||
└── README.md — how to add a new model class
|
||||
```
|
||||
|
||||
Key principle (from work order): **empirical, ne ideoloski.** If probe shows Qwen does better with markdown than plain, use markdown. Each shape includes a `metadata.evidence_link` field pointing to the empirical evaluation that justified its existence.
|
||||
|
||||
`packages/agent/config/model-prompt-shapes.json` — alias → shape name mapping.
|
||||
|
||||
### 1.3 — `packages/agent/src/run-meta.ts` (NEW)
|
||||
|
||||
Captures: `run_id`, `config_snapshot` (frozen JSON), `dataset_sha256` if applicable, `model_versions`, `provider_routing`, `prompt_shape_per_model`, `seed`, `git_sha`, `timestamp_iso`, `normalization_actions_per_prediction[]`, `raw_api_responses[]` (gzipped), `judge_call_traces[]`.
|
||||
|
||||
Deterministic reproduction for greedy decoding (temperature=0): given run_meta, replay must produce identical predictions.
|
||||
|
||||
---
|
||||
|
||||
## Phase 1 commit boundaries (proposed)
|
||||
|
||||
```
|
||||
commit 1.1: feat(agent): output-normalize layer with raw/normalized/actions audit
|
||||
commit 1.2: feat(agent): model-aware prompt shapes + selector + config
|
||||
commit 1.3: feat(agent): run-meta capture for deterministic reproduction
|
||||
commit 1.* tests: unit + round-trip property tests for 1.1-1.3
|
||||
```
|
||||
|
||||
Each commit type-checks + tests pass; commits don't ship without green CI.
|
||||
|
||||
---
|
||||
|
||||
## Phase 1 acceptance gates (must pass before Phase 2 kick)
|
||||
|
||||
- `npm run test --workspace=@waggle/agent` green for new files
|
||||
- `tsc --noEmit` clean on `packages/agent/`
|
||||
- Existing 121 GEPA-related tests still pass (no regression)
|
||||
- Output normalization round-trip: 100 random adversarial inputs → no semantic drift; abstention signal preserved
|
||||
- Prompt shapes: at least 4 shapes shipped (claude / qwen-thinking / qwen-non-thinking / generic-simple); selector picks correctly for known aliases
|
||||
- Run-meta produces byte-identical replay on greedy decoding (verify with smoke test)
|
||||
|
||||
---
|
||||
|
||||
## Phase 2-5 high-level scope (briefed for PM situational awareness; full plans drafted phase-by-phase)
|
||||
|
||||
### Phase 2 — Multi-step agent loop unification (3-5 days)
|
||||
- Pull `scripts/run-pilot-2026-04-26.ts` `runCellMultiStep` logic into `packages/agent/src/agent-loop.ts` as the unified entry point
|
||||
- Parametrize: `MAX_STEPS` (default 5; configurable), `MAX_RETRIEVALS_PER_STEP` (default 8), per-call halt, multi-step pattern
|
||||
- Production agent in Tauri desktop + MCP server (`packages/server/`) consumes new entry point
|
||||
- `benchmarks/harness/src/cells.ts` deprecates hardcoded "compressed" scaffold; refactors to consume `runAgentLoop` via public API
|
||||
- Pilot wrapper (`scripts/run-pilot-2026-04-26.ts`) becomes thin wrapper around `packages/agent/` (no separate implementation)
|
||||
- Risk: Stage 3 v6 oracle 74% must reproduce — gate on this before Phase 3 kick
|
||||
|
||||
### Phase 3 — Long-task persistence (2-3 days)
|
||||
- `long-task/checkpoint.ts` — serialize state per step
|
||||
- `long-task/recovery.ts` — restore from checkpoint; retry-with-backoff; tool-failure fallback
|
||||
- `long-task/context-manager.ts` — intelligent context summarization; hive-mind retrieval over accumulated state
|
||||
- Progress callbacks + telemetry hooks
|
||||
- Test scenario: simulate process kill mid-step → resume → complete → identical final output
|
||||
|
||||
### Phase 4 — Skills/tools audit + failure taxonomy (1-2 days)
|
||||
- Audit `packages/agent/src/*-tools.ts` for Claude-narrative-shaped descriptions
|
||||
- Refactor to consume prompt-shapes infrastructure
|
||||
- `failure-classify.ts` 10-category classifier (thinking_leakage, correct_answer_with_extra_text, unknown_false_negative, metadata_copy, format_violation, punctuation_or_case_only, wrong_span, wrong_entity, hallucination, retrieval_or_harness_error)
|
||||
- `benchmarks/harness/src/report.ts` (or `packages/agent/src/report.ts`) emits per-cell + per-model summary.{json,md} + predictions.jsonl + failures.jsonl
|
||||
|
||||
### Phase 4 acceptance gate — pilot 2026-04-26 RE-SCORE (PM addendum 2026-04-26, BINDING)
|
||||
|
||||
After Phase 4 ships (output-normalize from 1.1 + failure-classify from 4) and BEFORE Phase 5 mini re-pilot kicks:
|
||||
|
||||
**(i) Re-score 2026-04-26 pilot artefacts** — read all 12 cell JSONLs (4 cells × 3 tasks), apply Phase 1.1 normalization layer to each `candidate_response`, run Phase 4 failure-classifier on each (response, ground-truth-question, materials, judge_rationale) tuple. **NO new LLM calls** — analysis-only on existing artefacts.
|
||||
|
||||
**(ii) Output:** `D:\Projects\PM-Waggle-OS\decisions\2026-04-26-pilot-rescored-delta-report.md` with:
|
||||
- Per-cell raw vs normalized score (which cells now PASS post-normalization)
|
||||
- Failure category distribution per cell (10-bucket histogram per failure-classify taxonomy)
|
||||
- Specifically for the 8 FAIL cells (Tasks 2+3 H2 + all 3 tasks H3 + all 3 tasks H4 = covers `task-2/{B,D}`, `task-3/{B,D}`, `task-1/D`, `task-2/D`, `task-3/D` after de-dup), report counts of:
|
||||
- `thinking_leakage` failures (would be removed by 1.1 normalization)
|
||||
- `unknown_false_negative` failures (would be flagged by classifier)
|
||||
- `metadata_copy` failures (would be removed by 1.1 normalization)
|
||||
- `format_violation` failures (would be flagged by classifier)
|
||||
- **Empirical conclusion:** how many of the pilot reversals were normalization-fixable artefacts vs real harness-design issues.
|
||||
|
||||
**(iii) HALT + PM ratify** the delta report before Phase 5 mini re-pilot kicks. The delta is the empirical signal for PM scope decision: if most reversals are normalization-fixable, Phase 5 re-pilot is a confirmation step; if most are real harness-design issues, Phase 5 may need scope expansion (e.g. raise MAX_STEPS, alternative agent loop pattern).
|
||||
|
||||
**Reason for binding addition:** without this signal we don't know whether Phase 2 loop unification was enough fix or whether Phase 5+ needs extra investigation. The re-score is cheap ($0 LLM cost; pure analysis) and gives PM the data needed for the Phase 5 scope call.
|
||||
|
||||
### Phase 5 — Mini re-pilot validation (1-2 days)
|
||||
- 4-5 cells × 3-5 tasks (3 from current pilot + 1-2 new long-task scenarios)
|
||||
- Pre-registered hypotheses identical to 2026-04-26 + H5 (across-model variance < 0.15) + H6 (long-task scenario completes)
|
||||
- Cost cap $30, halt $25, trio-strict ensemble with max_tokens=3000
|
||||
- Acceptance gates per work order
|
||||
|
||||
---
|
||||
|
||||
## What does NOT happen in this sprint (deferred to SECONDARY)
|
||||
|
||||
- True GEPA self-evolve evaluation (needs labeled corpus authored separately)
|
||||
- Per-model GEPA optimization candidates
|
||||
- Multi-model objective GEPA score function
|
||||
- 4-way eval split (train/dev/holdout/golden)
|
||||
|
||||
These are SECONDARY scope. PM kicks off after PRIMARY confirmed PASS via re-pilot.
|
||||
|
||||
---
|
||||
|
||||
## Halt-and-ping triggers (binding for whole sprint)
|
||||
|
||||
1. **Stage 3 v6 reproduction shows >2pp regression with new agent code** — halt before merging Phase 2; investigate; fix or roll back.
|
||||
2. **Re-pilot reveals problem was NOT multi-step harness pattern** but something else (model capability ceiling, judge methodology, retrieval quality) — flag for PM scope re-evaluation.
|
||||
3. **Long-task scenario reveals hive-mind retrieval gap** — flag (do not block agent fix; hive-mind hooks land as separate post-sprint work).
|
||||
|
||||
---
|
||||
|
||||
## What is NOT permitted (hard rules from work order)
|
||||
|
||||
- Fix that works for Claude but fails for Qwen (current bug)
|
||||
- Hardcoded "this model gets this prompt" without configurable layer
|
||||
- Output normalization that silently changes semantics (`unknown` → `""` strips abstention)
|
||||
- Skills/tools that are Claude-narrative-shaped without audit
|
||||
- Evolution-gates that are not blocking for cross-model regression
|
||||
- `benchmarks/harness/` remaining proxy scaffold separate from production agent
|
||||
- GEPA testing without labeled corpus
|
||||
|
||||
---
|
||||
|
||||
## Proposed start point — PM RATIFICATION REQUESTED
|
||||
|
||||
**Phase 1 (Foundations) is parallel-safe and has no dependencies.**
|
||||
|
||||
Two options for kickoff:
|
||||
|
||||
**Option A (recommended): Phase 1 in 3 sub-commits over 3-4 days.**
|
||||
1.1 output-normalize + tests → commit
|
||||
1.2 prompt-shapes + tests → commit
|
||||
1.3 run-meta + tests → commit
|
||||
HALT → PM ratifies Phase 1 → Phase 2 kicks
|
||||
|
||||
**Option B: All 3 Phase 1 items in a single commit (faster, larger review surface).**
|
||||
Same 3-4 day timeline, harder to review, single rollback handle.
|
||||
|
||||
**Option C: Start Phase 2 first** (loop unification) and let Phase 1 land alongside as needed.
|
||||
Higher risk — unification without normalization + prompt-shapes ready means we're building on incomplete foundations and may have to rework. Not recommended.
|
||||
|
||||
---
|
||||
|
||||
**Awaiting PM ratification of phasing + start option.**
|
||||
175
decisions/2026-04-26-pilot-verdict-FAIL.md
Normal file
175
decisions/2026-04-26-pilot-verdict-FAIL.md
Normal file
@@ -0,0 +1,175 @@
|
||||
---
|
||||
decision_id: 2026-04-26-pilot-verdict-FAIL
|
||||
date: 2026-04-26
|
||||
authority: PM (Marko Marković) — ratified
|
||||
type: pilot close-out + halt expansion
|
||||
predecessors:
|
||||
- briefs/2026-04-26-agentic-knowledge-work-pilot/cc1-brief.md
|
||||
- briefs/2026-04-26-agentic-knowledge-work-pilot/cc1-brief-amendment-2026-04-26.md
|
||||
- briefs/2026-04-26-agentic-knowledge-work-pilot/cc1-brief-amendment-v2-2026-04-26.md
|
||||
manifest_anchor: pilot-2026-04-26-v1
|
||||
---
|
||||
|
||||
# Pilot 2026-04-26 — Final Verdict: FAIL
|
||||
|
||||
**Pilot:** `agentic-knowledge-work-pilot-2026-04-26`
|
||||
**Hypotheses tested:** H2 (Opus multiplier), H3 (Qwen multiplier), H4 (sovereignty bridge)
|
||||
**Sample:** N=3 tasks × 4 cells = 12 candidate runs + 36 judge calls
|
||||
**Outcome:** **🔴 FAIL** — H2 1/3, H3 0/3, H4 0/3, critical_failures 0
|
||||
**Total cost:** $5.5806 / $20 cap (28%) · **Total wall:** ~80 min over 3 sessions
|
||||
|
||||
---
|
||||
|
||||
## 1. Result table (binding)
|
||||
|
||||
| Task | A (Opus solo) | B (Opus + harness) | C (Qwen solo) | D (Qwen + harness) | H2 (B−A) | H3 (D−C) | H4 (D vs A) |
|
||||
|------|---------------|---------------------|----------------|---------------------|----------|----------|--------------|
|
||||
| task-1 strategic synthesis (7 docs) | 4.611 | 4.944 | 4.583 | 4.389 | **+0.333 PASS** | −0.194 ✗ | −0.222 ✗ |
|
||||
| task-2 cross-thread coord (4 threads) | 4.944 | 5.000 | 4.667 | 3.944 | +0.056 ✗ | −0.722 ✗ | **−1.000 ✗** |
|
||||
| task-3 decision support (3 memos) | 4.944 | 4.889 | 4.889 | 4.556 | −0.056 ✗ | −0.333 ✗ | −0.389 ✗ |
|
||||
| **TOTALS** | — | — | — | — | **1/3** | **0/3** | **0/3** |
|
||||
|
||||
Pre-registered PASS criteria (cc1-brief §2): each hypothesis directional sign on ≥ 2 of 3 tasks. **All three hypotheses fail this criterion.**
|
||||
|
||||
---
|
||||
|
||||
## 2. What this pilot did and did not test
|
||||
|
||||
**Tested:**
|
||||
- Multi-step agent loop with retrieval-augmented self-prompting (amendment v1 §1, renamed in amendment v2 §1)
|
||||
- HybridSearch (FTS5 + vec0 RRF) over per-task SessionStore corpus
|
||||
- Trio judge ensemble (Opus + GPT + MiniMax M2.7, κ=0.7878 from Stage 3 v6 calibration)
|
||||
- Two model classes (Claude Opus 4.7, Qwen 3.6 35B-A3B via DashScope direct)
|
||||
- Three task types (strategic synthesis, cross-thread coordination, decision support)
|
||||
|
||||
**Did NOT test:**
|
||||
- True GEPA self-evolve (deferred — no labeled training corpus exists for open-ended N=1 synthesis tasks)
|
||||
- Long-task scenarios (multi-hour runs, checkpoint/recovery, context window exhaustion)
|
||||
- Skills/tools layer (only retrieval was exercised; agent had no tool-use beyond search)
|
||||
- Across-model-variance bound (H5, not pre-registered for this pilot)
|
||||
|
||||
**Implication:** the FAIL verdict is on the multi-step harness pattern as currently implemented. GEPA self-evolve as a separate thesis remains untested and could be evaluated separately if a labeled corpus is authored.
|
||||
|
||||
---
|
||||
|
||||
## 3. Methodology audit findings (binding for future work)
|
||||
|
||||
### 3.1 Original smoke had 2 confounds — addressed via amendment v2
|
||||
|
||||
- **Wrong Qwen model:** original brief specified `qwen3.6-35b-a3b-via-openrouter` which bridges to Qwen 3.5 (`openrouter/qwen/qwen3.5-35b-a3b`) per litellm-config.yaml comment "one-minor regress to 3.5 until OR carries 3.6". Cells C/D Task 1 originally ran on Qwen 3.5.
|
||||
- **Wrong max_tokens for synthesis:** wrapper default 4096 vs Stage 3 v6 lineage 16000-64000. Reasoning headroom may have been silently constrained.
|
||||
|
||||
Amendment v2 §2 corrected both: `qwen3.6-35b-a3b-via-dashscope-direct` + thinking=on + max_tokens=16000. Cells C/D Task 1 restarted under corrected config. Restart shifted Cell C 4.167 → 4.583 (+0.42), Cell D 3.944 → 4.389 (+0.45). H4 reversal magnitude fell from −0.555 → −0.111 — but did not cross zero.
|
||||
|
||||
### 3.2 Confounds NOT eliminated by amendment v2
|
||||
|
||||
- **Cell B Opus loop_exhausted on Tasks 2 + 3:** the 5-step `MAX_STEPS` ceiling was binding. Task-2/B `loop_exhausted=true` (steps=3, retr=2 — but step budget reached without natural finalization). Task-3/B `loop_exhausted=true` (steps=4, retr=3). Opus multi-step was force-finalized on those tasks. May have degraded Cell B output → contributed to H2 reading negative on Tasks 2 + 3.
|
||||
- **Cell A judge ceiling:** Opus solo scored 4.611 / 4.944 / 4.944. With Cell A near 5.0 on Tasks 2 + 3, there is almost no Likert headroom for Cell B to "improve" within the 1-5 scale. Multiplier hypotheses (H2) become hard to test on tasks where the solo baseline is already near judge ceiling.
|
||||
- **MiniMax max_tokens=1024 (initial config)** caused 3 of 12 judge calls to fail mid-JSON. Bumping 1024 → 3000 (PM-ratified post-second-smoke) yielded 11/12 success across the rest of the pilot. **Recommendation:** keep max_tokens=3000 for any successor MiniMax-judge benchmark.
|
||||
|
||||
### 3.3 What the FAIL verdict means
|
||||
|
||||
The harness as currently implemented does not deliver the multiplier we need on synthesis-class tasks. **Possible causes** (not yet disambiguated):
|
||||
|
||||
1. **5-step MAX_STEPS too tight** for Opus on longer-context tasks (Tasks 2 + 3). Raising to 8-10 steps could change H2 reading.
|
||||
2. **Retrieval fragmentation hurts Qwen** — passing retrieved chunks back to the model splits the context vs. a single full-materials prompt. Qwen may reason better over coherent full context than over RAG-fragmented context.
|
||||
3. **Judge ceiling on solo cells** — when solo scores ~4.94, multiplier is bounded by ~0.06 max. H2 reading on Tasks 2 + 3 is dominated by ceiling effect, not real harness capability.
|
||||
4. **Single-shot prompts already include all materials** — for tasks with 3-7 documents fitting comfortably in context window (all 3 pilot tasks did), retrieval-augmented multi-step adds latency + token overhead without information gain.
|
||||
|
||||
These causes are testable in a re-pilot after the harness is fixed.
|
||||
|
||||
---
|
||||
|
||||
## 4. Operational findings (binding for successor benchmarks)
|
||||
|
||||
| Finding | Disposition |
|
||||
|---------|-------------|
|
||||
| MiniMax `max_tokens` = 1024 too tight for dense memo responses | Bump to **3000** for any future MiniMax-judge benchmark |
|
||||
| Loop_exhausted on Cell B/D for longer-context tasks | Raise `MAX_STEPS` to 8-10 (testable via re-pilot) |
|
||||
| Cell A judge ceiling on compact tasks | Use longer / more complex tasks for multiplier benchmarks; or use absolute-difficulty calibration |
|
||||
| Qwen 3.6 via DashScope direct + thinking=on + 16000 tokens = correct synthesis-class config | Inherit for any successor benchmark; binding via amendment v2 §5 |
|
||||
| OR-bridge `qwen3.6-35b-a3b-via-openrouter` regresses to Qwen 3.5 | Use only as documented failover; never as primary route |
|
||||
| `INHERITED_CONFIGS_REQUIRE_TASK_TYPE_AUDIT` rule | New binding rule per amendment v2 §5; PM commits to honor in future brief authoring |
|
||||
|
||||
---
|
||||
|
||||
## 5. Halt expansion — full N=400 multiplier benchmark NOT authorized
|
||||
|
||||
Per cc1-brief §11 + amendment v2 §7, PM action on FAIL is one of three branches:
|
||||
|
||||
- **Branch A — full benchmark NOT authorized:** Expansion would be expensive evidence collection on a known-negative direction. ✓ **PM SELECTION**
|
||||
- **Branch B — full benchmark conditionally authorized after harness fix.** Folded into Branch A logic — re-pilot N=12-20 is the prerequisite (see §6).
|
||||
- **Branch C — pivot to retrieval V2 work.** Folded into the agent-fix sprint (see §6) since retrieval improvements would be evaluated in the same re-pilot as harness improvements.
|
||||
|
||||
**PM selection: Branch A + agent-fix sprint** (per memorandum 2026-04-26 ratification).
|
||||
|
||||
---
|
||||
|
||||
## 6. Successor work — agent-fix sprint authorized
|
||||
|
||||
PM has authorized a 1-2 week sprint covering:
|
||||
|
||||
**PRIMARY (multi-step harness pattern fix in `packages/agent/`):**
|
||||
1. Output normalization layer (`packages/agent/src/output-normalize.ts` — new)
|
||||
2. Model-aware prompt shapes (`packages/agent/src/prompt-shapes/` — new directory)
|
||||
3. Multi-step agent loop unification (consolidate `scripts/run-pilot-2026-04-26.ts` + `benchmarks/harness/src/cells.ts` proxy → `packages/agent/src/agent-loop.ts` as single source of truth)
|
||||
4. Long-task persistence + recovery (`packages/agent/src/long-task/{checkpoint,recovery,context-manager}.ts` — new)
|
||||
5. Skills + tools audit (`packages/agent/src/*-tools.ts` — refactor to model-aware)
|
||||
6. Failure taxonomy + per-cell + per-model reports (`packages/agent/src/failure-classify.ts` — new)
|
||||
7. Run artifacts + reproducibility (`packages/agent/src/run-meta.ts` — new)
|
||||
|
||||
**SECONDARY (after PRIMARY confirmed PASS via re-pilot):**
|
||||
8. GEPA evaluation with proper labeled corpus (per-model + multi-model objective) — extends existing `packages/agent/src/{iterative-optimizer,evolution-gates,eval-dataset}.ts`
|
||||
|
||||
**VALIDATION:** Mini re-pilot N=12-20 — same 4-cell structure, plus H5 (across-model variance < 0.15 Likert) + H6 (long-task scenario completes). Cost cap $30, halt $25.
|
||||
|
||||
**Acceptance gates:**
|
||||
- Stage 3 v6 oracle ceiling reproduces 74% (no substrate regression)
|
||||
- Re-pilot H2/H3/H4 PASS on ≥ 2/3 tasks
|
||||
- Cross-model variance bounded
|
||||
- Long-task scenario completes on all 3 models without data loss
|
||||
- Zero `thinking_leakage` failures in Qwen output post-normalization
|
||||
- `benchmarks/harness/cells.ts` no longer proxy — uses `packages/agent/src/agent-loop.ts`
|
||||
- Pilot wrapper consolidated into `packages/agent/`
|
||||
- Robustness gate blocks candidates that improve one model but degrade another > 2pp
|
||||
|
||||
---
|
||||
|
||||
## 7. Substrate claim — paper claim #1 STANDS
|
||||
|
||||
This pilot does NOT invalidate paper claim #1 (memory substrate quality). Stage 3 v6 LoCoMo apples-to-apples 74% (oracle ceiling) vs Mem0 published 66.9% remains the binding evidence for memory substrate. See `benchmarks/results/v6-self-judge-rebench/apples-to-apples-memo.md` and commit `b7e19c5`.
|
||||
|
||||
The pilot was specifically scoped to paper claim #2 (multiplier on agentic knowledge work). That claim requires further work before it can be made.
|
||||
|
||||
---
|
||||
|
||||
## 8. Artefacts (binding evidence)
|
||||
|
||||
| Path | Description |
|
||||
|------|-------------|
|
||||
| `benchmarks/results/pilot-2026-04-26/pilot-task-{1,2,3}-{A,B,C,D}.jsonl` | 12 binding cell records |
|
||||
| `benchmarks/results/pilot-2026-04-26/pilot-summary.json` | Re-emitted aggregate (covers all 12 cells) |
|
||||
| `benchmarks/results/pilot-2026-04-26/pilot-run.log` | Continuous log: smoke → restart → final chain |
|
||||
| `benchmarks/results/pilot-2026-04-26/invalidated/` | Original smoke C/D records preserved for audit |
|
||||
| `benchmarks/results/pilot-2026-04-26/prompts-archive/` | Per-cell prompts + multi-step traces |
|
||||
| `briefs/2026-04-26-agentic-knowledge-work-pilot/cc1-brief.md` | Original brief (audit-immutable) |
|
||||
| `briefs/2026-04-26-agentic-knowledge-work-pilot/cc1-brief-amendment-2026-04-26.md` | Amendment v1 |
|
||||
| `briefs/2026-04-26-agentic-knowledge-work-pilot/cc1-brief-amendment-v2-2026-04-26.md` | Amendment v2 |
|
||||
| `scripts/run-pilot-2026-04-26.ts` | Pilot orchestrator (to be unified into `packages/agent/`) |
|
||||
|
||||
---
|
||||
|
||||
## 9. Audit chain (per amendment v2 §8)
|
||||
|
||||
```
|
||||
amendment_v2_doc_sha256 = 1ab5082ff773538a26b3c3294f7fbee4e30063a8d994bdb3753bdc9dd6d6cd99
|
||||
amendment_v1_doc_sha256 = 3946d3e00fbb1996fb7e63096ecef51abf1e209e5ff166fd0d8758e9a3a14aad
|
||||
cc1_brief_sha256 = 9805adae478333178d36d71b88795afc37f8fb543c2ebccaecb7b01faf06afee
|
||||
judge_rubric_sha256 = 2e24826eb75e92ef1e64055bb2c632eec64ded8fedf7d5b6897ccaec9ffff2eb
|
||||
head_sha (at pilot run) = b7e19c557fdbc42f2d0a3c3213176aa4d790f7a2
|
||||
manifest_anchor = pilot-2026-04-26-v1
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
**End of verdict. Pilot CLOSED. Successor work scope = agent-fix sprint (PRIMARY → re-pilot → SECONDARY).**
|
||||
Reference in New Issue
Block a user