oleg.m/waggle-os
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
0c3e2ead3beaffba9d5004ef7fd87d871d5e6886
waggle-os/docs/plans/H-AUDIT-1-DESIGN-DOC-2026-04-22.md
Oleg Maslov 0c3e2ead3b
Some checks failed
Installer Smoke / installer-smoke (push) Has been cancelled
Details
moving
2026-09-02 10:10:29 +02:00

20 KiB
Raw Blame History

H-AUDIT-1 Design Doc — turnId Propagation + Stage 2 reasoning_content

Datum: 2026-04-22 Sprint: 11 · Track A · Task A1 Authority chain: briefs/2026-04-22-cc-sprint-11-kickoff.md §3 Track A A1 + decisions/2026-04-22-stage-2-primary-config-locked.md §5 (reasoning_content handling extension) Author: CC-1 Status: DRAFT — awaiting PM ratification before A2 implementation Supersedes memory claim: project_h_audit_1_not_implemented.md — stale. Production chat stack turnId propagation was landed in a prior sprint; see §1 state audit below.

0. Executive summary

Production-chat turnId propagation, the grep ≥6 acceptance target, the full-turn-graph reconstruction test, and the regression guard for the six target files are already landed and green on origin/main as of e1ae0a4. The live surface is 7 files in packages/agent/src + 2 files in packages/server/src = 9 files with turnId references. The existing Vitest suite packages/agent/tests/turn-context.test.ts asserts all four brief acceptance items in isolated test cases.

A2 net-new scope is therefore narrowed to one concern: reasoning_content handling for Stage 2 thinking=on, max_tokens=64000 on qwen3.6-35b-a3b-via-openrouter, per decision doc §5. The Stage 2 batch path runs through benchmarks/harness/src/llm.ts (not the production chat stack), so reasoning_content capture + turnId correlation lands in the harness + its JSONL records, with an explicit rule on production-chat behavior documented below.

1. Current state audit (as of 2026-04-22, HEAD = e1ae0a4)

1.1 turnId generator

File: packages/agent/src/turn-context.ts:29 Contract: export function generateTurnId(): string { return randomUUID(); } — node:crypto.randomUUID() which is UUID v4 by spec (verified by turn-context.test.ts:30-36 regex ^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$).

This satisfies brief §3 Task A1 "generation point: orchestrator turn entry, crypto.randomUUID() v4, ne v7, ne custom."

1.2 Propagation surface (verified grep + read)

File turnId role Lines of interest
packages/agent/src/turn-context.ts generator + logTurnEvent(turnId, payload) helper 29, 45-51
packages/server/src/local/routes/chat.ts generation site (POST /api/chat entry) 318-325
packages/agent/src/agent-loop.ts optional turnId?: string in config; logs agent-loop.enter / .exit / .tool.enter / .tool.exit 75-80, 100, 103, 384, 477-484
packages/agent/src/orchestrator.ts optional turnId?: string in recallMemory opts; logs .enter/.exit + injection-block branch 94-95, 589, 681, 702, 712
packages/agent/src/prompt-assembler.ts optional turnId?: string; logs prompt-assembler.assemble 86-87, 414
packages/agent/src/combined-retrieval.ts optional turnId?: string in CombinedSearchOptions; logs retrieval.enter/.exit (+ KVARK branch) 60-61, 215, 230-254
packages/agent/src/cognify.ts optional turnId?: string in cognify/cognifyFrame/cognifyBatch; logs enter/exit for each 48-55, 106-139
packages/agent/src/index.ts barrel export of generateTurnId 3
packages/server/src/benchmarks/aggregate.ts turnId carried in benchmark aggregate records (via grep)

The six target files from brief §3 are all covered; the list is 9 files total when counting the generator, barrel, benchmark aggregate, and server route.

1.3 Persistence format in chat.ts

Current behavior (chat.ts:324): turnId is generated and logged via logTurnEvent(turnId, { stage: 'chat.turn.start', ... }). Logs use the shared pino logger with turnId as a structured field. There is no dedicated per-turn trace-store row for chat turnId persistence — reconstruction is log-scrape (pino) + the test-only startTurnCapture() buffer from turn-context.ts:61-76.

The broader packages/core/src/mind/execution-traces.ts store exists (for the evolution subsystem) but is not currently keyed on turnId — it uses traceId + runId for evaluation datasets. This is intentional: trace-store was scoped to evolution evals, while turnId is the lightweight correlation key for one POST /api/chat cycle.

1.4 Tests already in place

packages/agent/tests/turn-context.test.ts — three describe blocks:

  1. turn-context helpers (5 tests): UUID v4 shape, silent no-op when undefined, capture buffer semantics, concurrent-turn isolation, stopTurnCapture reverts mode.
  2. H-AUDIT-1 stage threading (end-to-end trace assertion) (1 test, line 80): simulates chat.ts → agent-loop → orchestrator.recallMemory → retrieval → prompt-assembler → tool-call → cognify → agent-loop.exit with a single turnId, asserts new Set(buf.map(e => e.turnId)).size === 1. This is the "reconstruct full turn graph from a single turnId" acceptance test the brief asks for.
  3. H-AUDIT-1 source-tree regression guard (1 test, line 121): reads each of the six required files from disk and asserts every one contains turnId. Fails if any future edit accidentally drops trace plumbing.

1.5 Acceptance gate already satisfied

Brief acceptance item Current state Evidence
grep -n "turnId" packages/**/*.ts ≥ 6 hits ≥50 hits across 9 files Sprint 11 Day-1 grep output, §1.2 table
Unique files with match ≥ 5 9 files §1.2 table
Unit test reconstructs full turn graph from single turnId exists turn-context.test.ts:80-117
Zero regressions on existing suites green pre-sprint (S4 handoff: 4974/4975) will re-verify post-any-change in A2
tsc --noEmit clean green pre-sprint same

The A2 "turnId implementation" acceptance is met on HEAD. A2 becomes a narrow, reasoning_content-only task; see §2.

2. Net-new scope for Stage 2 on/64K — reasoning_content handling

Stage 2 batch runs on qwen3.6-35b-a3b-via-openrouter with thinking=on, max_tokens=64000 (LOCKED 2026-04-22, decision doc §1). Qwen3.6 with thinking enabled emits a reasoning_content field on the response object, separate from the finalized answer. Per decision doc §5, design must cover three rules: persistence, retention, exclusion.

2.1 Execution surface that sees reasoning_content

Stage 2 does not run the production chat stack (orchestrator + cognify + tools + prompt-assembler). It runs through benchmarks/harness/src/llm.ts → LiteLLM → OpenRouter → Qwen. The four cells in benchmarks/harness/src/cells.ts are pure-LLM prompts (no memory tool, no evolution wiring in the harness scaffold as of HEAD).

Consequence: reasoning_content capture lands in the harness layer, not the production chat stack. Production chat (which today doesn't ship with thinking=on by default on any route) gets the exclusion rule only; capture/persistence is explicitly out of this design doc's scope until a future brief.

2.2 Persistence rule

Harness layer (primary):

  • Extend LlmCallResult (in benchmarks/harness/src/llm.ts:12-22) with an optional reasoningContent?: string field. Populated only when the provider response includes it; left undefined otherwise to preserve back-compat for models that don't emit reasoning.
  • Parse from two canonical response shapes:
    • DashScope-intl native: body.choices?.[0]?.message?.reasoning_content (snake_case, top-level in message).
    • OpenRouter unified reasoning API: body.choices?.[0]?.message?.reasoning (note: different key; OR's unified API normalizes cross-provider).
  • Fallback: if neither is present but response body has a top-level body.reasoning_content (older DashScope shape), read that too. Log one reasoning_content_shape_unknown warning if we see a response with thinking=on requested but no reasoning surface, so a future probe can catch provider schema drift.
  • Extend JsonlRecord (in benchmarks/harness/src/types.ts) with an optional reasoning_content?: string and reasoning_content_chars?: number field. The record is keyed by turnId (already there as the first field), so reconstruction from a single turnId reads the JSONL row and gets both answer and reasoning.

Production chat path (out-of-scope for A2, documented for future):

  • If/when a production request is issued against a thinking-enabled route, reasoning_content MUST NOT be persisted to frames, memory, or KnowledgeGraph. It flows through the response and is discarded after the stream completes. If operator logging of reasoning is ever required, a separate design doc and opt-in flag will scope it. No silent write-through.

2.3 Retention policy

Harness JSONL artifacts:

  • The JSONL file under benchmarks/results/*.jsonl is the canonical persistence surface. Retention follows the existing benchmark artifact convention: committed to the repo when landing a benchmark report; otherwise lives in benchmarks/results/ gitignored until the sprint that produced it is closed and a curated subset (summary + representative rows) is moved to preflight-results/. Full reasoning_content is not moved into preflight-results/ — only a summary char-count aggregate, to keep report size manageable.
  • Raw JSONL with reasoning_content stays in benchmarks/results/ locally for the life of the sprint and is deleted/pruned when the sprint close-out report lands on origin/main. No long-term reasoning_content archival.
  • Rotation: daily housekeeping is not automated in this design. If Stage 2 full-run produces ≥2GB of JSONL (estimated ceiling: ~1GB for 2000-call full-run at reasonable reasoning-token sizes), a manual prune step goes in the Sprint 12 close-out runbook. Not a Sprint 11 gate.

Logs (pino):

  • logTurnEvent(turnId, { stage: 'llm.response', reasoningChars: N }) emits the character count only, not the content itself. This gives observability (did reasoning happen? how big?) without polluting logs with possibly-sensitive chain-of-thought.

2.4 Exclusion rule

Reasoning_content MUST NOT appear in:

  1. User-facing output streams — the SSE chat.ts path streams text only, not reasoning. (Already true; guarding against regressions when anyone wires a thinking-enabled route to production chat.)
  2. Judge inputs — benchmarks/harness/src/judge-runner.ts passes only modelAnswer (the final text) to the judge. The judge never sees reasoning_content. This preserves judge neutrality and prevents the judge from being biased by reasoning artifacts that aren't part of the model's final answer.
  3. Public trace viewers / UI — any future chat-UI trace inspector must project JSONL records with reasoning_content stripped unless the caller has an explicit includeReasoning: true permission. v1 guidance: don't add a UI trace inspector at all; reasoning_content lives in CLI-accessible JSONL only for the benchmark team.
  4. Frames / memory / KnowledgeGraph persistence — production chat never writes reasoning_content downstream of the LLM call.
  5. MCP response payloads — MCP tools return structured results; reasoning_content is not a tool-call product.
  6. Summary exports and aggregate reports — benchmarks/harness/src/metrics.ts aggregates cost/latency/accuracy; it computes a reasoning_content_chars aggregate (sum, p50, p95) but does not copy the content into summary JSON. The summary stays <100KB so it fits in briefs.

Permission surface for opt-in inclusion: the optional includeReasoning: boolean parameter lives on JSONL-reader utilities only (not on harness writers). Writers always write reasoning; readers default to stripping it. This puts the gate on the read path, which is where the visibility decision belongs.

2.5 Invariant

turnId is a foreign key — given a JSONL row with turnId T, a consumer can reconstruct:

  • The cell/control + model + instance that generated the row (existing fields)
  • The final model answer (text / model_answer)
  • The judge verdict if judging was enabled (judge_verdict + judge_rationale)
  • The reasoning trace that produced the answer (reasoning_content, net-new)

The four of these together form the "full turn graph" for a Stage 2 batch turn. This is stronger than the production-chat graph (which today reconstructs via log events) because benchmark rows are structured JSONL by design.

3. Test scenario (sample code — not committed yet)

Two net-new test cases land in benchmarks/harness/tests/reasoning-capture.test.ts:

// Test 1: reasoning_content round-trip — harness llm.ts captures, JSONL row persists
it('captures reasoning_content when provider returns it and persists to JSONL under turnId', async () => {
  const fakeLlm = createFakeLlmClient({
    responseBody: {
      choices: [{
        message: {
          content: 'Paris',
          reasoning_content: 'The user asked for capital of France. France capital is Paris.',
        },
      }],
      usage: { prompt_tokens: 10, completion_tokens: 2 },
    },
  });
  const result = await fakeLlm.call({ model: stage2Model, systemPrompt: 'sys', userPrompt: 'q' });
  expect(result.text).toBe('Paris');
  expect(result.reasoningContent).toMatch(/capital of France/);
  expect(result.reasoningContent?.length).toBeGreaterThan(0);
});

// Test 2: full turn-graph reconstruction from single turnId (JSONL round-trip)
it('reconstructs full turn graph (answer + reasoning + cost + latency) from single turnId', async () => {
  // Run one harness turn with a fake LLM emitting reasoning_content.
  // Read back the JSONL. Assert: filter by turnId yields exactly one row
  // containing answer, reasoning_content, cost, latency, judge payload (if judged).
  const turnId = await runOneHarnessTurn({ model: stage2Model, fakeLlm });
  const rows = readJsonl(outputPath).filter(r => r.turnId === turnId);
  expect(rows).toHaveLength(1);
  expect(rows[0].model_answer).toBeDefined();
  expect(rows[0].reasoning_content).toBeDefined();
  expect(rows[0].reasoning_content_chars).toBe(rows[0].reasoning_content!.length);
});

Non-goals for these tests: real API calls (smoke test for that lives in B1 apply), full-run timing (Stage 2 C3 covers that), judge-reasoning interaction (§2.4 exclusion rule covers by construction).

4. Acceptance criteria (updated against current state)

# Criterion Status
1 grep -n "turnId" packages/**/*.ts returns ≥6 hits ✅ already met (≥50 hits, 9 files)
2 Unique files with match ≥5 ✅ already met (9 files)
3 Unit test reconstructs full turn graph from single turnId (production stack) ✅ already met (turn-context.test.ts:80-117)
4 Unit test reconstructs full turn graph including reasoning_content (harness layer) ⬜ A2 net-new — benchmarks/harness/tests/reasoning-capture.test.ts §3
5 LlmCallResult + JsonlRecord extended with reasoningContent / reasoning_content ⬜ A2 net-new
6 Harness captures reasoning_content from Qwen (two response shapes supported) ⬜ A2 net-new
7 Reasoning_content never written to frames/memory/KG/UI/summary reports ⬜ A2 net-new (assert via inspection + regression guard test)
8 pnpm test passes with zero regressions ⬜ A2 gate
9 tsc --noEmit clean on touched packages ⬜ A2 gate
10 Commit message: feat(audit): H-AUDIT-1 reasoning_content capture per design doc 2026-04-22 ⬜ A2 gate

5. Open questions for PM ratification

  1. Confirm narrowed A2 scope. Does PM accept that A2 implementation ships reasoning_content handling only, given that production-chat turnId propagation is already landed? If yes, exit ping filename is sessions/2026-04-22-sprint-11-h-audit-1-exit.md with the §4 criteria 4–7 closed.
  2. Memory note correction. Authorize marking project_h_audit_1_not_implemented.md as superseded by this design doc in the memory index. The note was accurate at write time; Sprint 10 landed the plumbing. The correction prevents future sessions from re-doing completed work.
  3. Cross-cutting note on reasoning/reasoning_content parser. OpenRouter's unified reasoning API uses key reasoning while DashScope native uses reasoning_content. The harness must parse both shapes. If PM prefers exclusive OR (one or the other, not both), flag here so CC-1 picks.
  4. Persistence slot under turnId. Proposed: same JSONL row, net-new field reasoning_content. Alternative considered: separate *.reasoning.jsonl sibling file to keep the main JSONL compact. Stuck with same-row for simplicity unless PM prefers separation.
  5. Retention beyond sprint. Current proposal is to keep raw reasoning_content JSONL local only, delete at sprint close. PM may want a long-term audit archive (compressed .jsonl.gz under benchmarks/archive/) for reproducibility of Stage 2 full-run — decision doc §5 retention rule implies this. Flagged for ratification.

6. Implementation plan for A2 (after ratification)

Surgical, non-speculative:

  1. Add reasoningContent?: string to LlmCallResult in benchmarks/harness/src/llm.ts. Parse from message.reasoning_content OR message.reasoning OR top-level body.reasoning_content (in that order). Log one reasoning_content_shape_unknown warning on miss when thinking was requested.
  2. Add reasoning_content?: string + reasoning_content_chars?: number to JsonlRecord in benchmarks/harness/src/types.ts. Populate in runner.ts from result.reasoningContent.
  3. Update metrics.ts aggregate to compute reasoningContentChars: { sum, p50, p95 } when any row has it.
  4. Ensure judge-runner.ts does not pass reasoning_content to the judge (verify; no change expected per current code).
  5. Land two new tests per §3 in benchmarks/harness/tests/reasoning-capture.test.ts.
  6. pnpm test + tsc --noEmit --project benchmarks/harness/tsconfig.json.
  7. Exit ping: sessions/2026-04-22-sprint-11-h-audit-1-exit.md with grep output + test log + commit SHA.

Budget: $0 for unit tests (fake LLM client). Only real API cost is Task B1 smoke test, already accounted in that task's $0.05 cap. Wall-clock estimate: 2-3h for the net-new slice (the big slice was landed in a prior sprint).

7. Anti-patterns

  • Do not re-implement turnId generator. Use generateTurnId() from @waggle/agent. The harness already imports it in runner.ts:32.
  • Do not thread turnId through cells. Cells are pure prompt-assembly; turnId is passed as a parameter (already) but only the runner needs to emit it into JSONL. Cells don't log.
  • Do not persist reasoning_content to production memory. The §2.4 exclusion rule is a hard contract. If a future task wants this, it needs a separate design doc and PM lock.
  • Do not add reasoning_content to judge input. §2.4 rule (2). Breaking this invalidates the judge-methodology axis that cleared Sprint 11 gate (Fleiss' κ=0.8784 on answer-only input).
  • Do not broaden scope beyond §6. A2 is reasoning_content only. Anything else (tool-call schema extensions, MCP bridge, production thinking-on wiring) is a separate ticket.

8. Related

  • briefs/2026-04-22-cc-sprint-11-kickoff.md §3 Track A A1 + A2
  • decisions/2026-04-22-stage-2-primary-config-locked.md §5 (reasoning_content handling extension)
  • decisions/2026-04-22-sprint-11-scope-locked.md §4.1–4.2 (gate criteria)
  • packages/agent/src/turn-context.ts — generator + logging helpers
  • packages/agent/tests/turn-context.test.ts — existing regression guards (6-file grep, full turn-graph reconstruction)
  • packages/server/src/local/routes/chat.ts:318-325 — generation site
  • benchmarks/harness/src/llm.ts — A2 primary edit target
  • benchmarks/harness/src/runner.ts:32 — turnId already imported + threaded
  • benchmarks/harness/src/types.ts — JsonlRecord extension target

End of A1 design doc. Awaiting PM ratification on §5 open questions before CC-1 moves to A2 implementation.

Reference in New Issue View Git Blame Copy Permalink