chore: remove internal documents and service directories (P2.2 part 1)

The commercial repository carries the product's internal history: audits,
screenshots, briefs, decisions, pricing, publication drafts, personal data
and research runs. None of it belongs in the community edition.

- delete judging/, "UAT 3/", benchmarks/ (including LoCoMo data under
  CC BY-NC), preflight-results/, gepa-phase-5/, notes/, decisions/,
  .understand-anything/, .agents/, .lovable/, the legacy sidecar/ package
  and waggle-cowork/ (an analysis of Claude Code sources)
- delete docs/ except docs/oss, the user guides (without the team-mode
  guide), the reference, methodology, architecture, getting-started and
  contributing pages, which P2.4 rewrites
- delete render.yaml, .parity-allowlist, the retired mind-parity and
  sync-mind workflows and 29 one-off scripts nothing references
- delete tests that only checked removed files (sidecar package, Render
  blueprint, UAT capture, benchmark harness isolation, internal audit and
  signing docs); drop sidecar/ and benchmarks/ from the hive-mind subtree
  split guard; prune ignore, attribute, lint and test globs
- replace the commercial README with an interim one for this edition

Source comments still point at some deleted docs; P2.2 part 2 cleans them
together with names and paths.
This commit is contained in:
shumeiko.it
2026-10-20 01:02:18 +02:00
parent 2e1999a3f6
commit 5f088698dd
1383 changed files with 67 additions and 503605 deletions

View File

@@ -1,338 +0,0 @@
---
name: ax-agent-optimize
description: This skill helps an LLM generate correct AxAgent tuning and evaluation code using @ax-llm/ax. Use when the user asks about agent.optimize(...), judgeOptions, eval datasets, optimization targets, saved optimizedProgram artifacts, or recursive optimization guidance.
version: "19.0.33"
---
# AxAgent Optimize Codegen Rules (@ax-llm/ax)
Use this skill for `agent.optimize(...)` workflows. Prefer short, modern, copyable patterns. Do not repeat general agent-authoring guidance unless the user needs it.
Your job is to help the model choose a good optimization setup for the user's actual goal:
- If the user wants better tool use, prefer action-aware tasks and either a deterministic metric or the built-in judge depending on how objective the scoring is.
- If the user wants better wording only, responder optimization may be enough.
- If the user wants reusable improvements, include artifact save/load.
- If the user wants cost or recursion behavior improved, make the eval tasks expose those tradeoffs explicitly.
## Use These Defaults
- Use `agent.optimize(...)` only after the agent is already configured and runnable.
- Prefer a deterministic custom `metric` when success is easy to score from the prediction and task record.
- Prefer the built-in judge path for open-ended assistant tasks: `judgeAI` plus `judgeOptions`.
- Only reach for a plain typed `AxGen` evaluator when the user needs LLM-as-judge behavior outside the built-in `agent.optimize(...)` flow.
- Default optimize target is `root.actor`; use `target: 'responder'` or explicit program IDs only when the user clearly asks for that.
- Use eval-safe tools or in-memory mocks because optimization replays tasks many times.
- Prefer precise tool return schemas such as `f.object(...)` over vague `f.json(...)` whenever the agent must reason about returned fields.
- Prefer task wording with canonical entity names like "the Atlas project" instead of ambiguous labels like "Atlas" when ambiguity could trigger pointless clarification.
- Save `result.optimizedProgram`, then restore with `new AxOptimizedProgramImpl(...)` and `agent.applyOptimization(...)`.
- When recursive behavior matters, keep `mode: 'advanced'` on the agent and tune against realistic `recursionOptions`.
## Decision Guide
Pick the optimization shape from the user's need:
- "Make the agent use tools correctly" -> optimize `root.actor` with `expectedActions` and `forbiddenActions`.
- "Make final answers read better" -> consider `target: 'responder'`, but only if the task is not mostly tool-selection or clarification behavior.
- "Make the whole agent better" -> use the default actor target first; only broaden target selection when the user clearly wants that extra scope.
- "Tune recursive delegation" -> keep `mode: 'advanced'` and use tasks that actually exercise recursion depth, fan-out, and termination choices.
- "Compare before and after" -> include a held-out task plus artifact save/load and replay.
Choose task design carefully:
- Prefer a small number of realistic tasks over broad but vague datasets.
- Prefer concrete criteria over generic "be helpful" language.
- Prefer explicit action expectations when correctness depends on tools, recipients, dates, or side effects.
- Prefer eval-safe mocks anytime the task touches email, scheduling, external APIs, or persistence.
## Make Agents Optimizable
Optimization works much better when the agent and dataset remove avoidable ambiguity:
- Prefer typed tool outputs over free-form JSON blobs so the actor can rely on exact field names.
- Tell the actor the exact tool fields it may use when payload shape matters.
- Explicitly ban invented fields if the model has any reason to guess hidden IDs or alternate key names.
- If recursive children only see explicit `llmQuery(..., context)` payloads, say that directly in the actor prompt.
- For recursive synthesis, tell the agent what the narrowed context should look like before delegation.
- Keep `maxSubAgentCalls` small in examples unless the user is explicitly testing broad fan-out behavior.
- Use canonical, unambiguous task wording so the model does not burn turns asking for fake clarification.
- In JS-runtime agents, require raw runnable JavaScript only. Ban `javascript:` prefixes, mixed prose/code, and multi-snippet turns.
Good pattern:
- tool schema says exactly what fields exist
- task names the exact entity to look up
- actor prompt says which fields to extract before delegation
- metric or judge penalizes unnecessary recursion and tool misuse
Bad pattern:
- tool returns `json` with an underspecified shape
- task uses overloaded names like `Atlas` without clarifying whether that is a project, team, or account
- recursive child is expected to infer hidden parent state that was never passed in context
- code agent is allowed to mix natural language with JavaScript in the same turn
## Metric vs Judge
Choose the scoring path based on how objectively the task can be measured:
- Use a custom `metric` when you can score success directly from `prediction` and `example`.
- Use the built-in agent judge when success depends on a full-run qualitative review across tool choices, clarifications, and final output.
- Use `judgeOptions.description` to tell the built-in judge what to value most.
- Use helper-based judge code only when the user is not inside `agent.optimize(...)` and still wants LLM judging.
Quick rules:
- Tool correctness with exact expected calls or forbidden calls: prefer a deterministic metric first.
- Simple extraction or classification with known correct answers: prefer a deterministic metric.
- Open-ended assistant quality, nuanced clarification behavior, or broad synthesis quality: prefer the built-in judge.
- GEPA or optimizer flows outside agents that still need LLM judging: use a plain typed `AxGen` evaluator.
Important:
- A custom `metric` overrides the built-in judge path entirely.
- Do not introduce a dedicated judge abstraction in new examples; prefer a plain typed `AxGen`.
- Do not add both a custom `metric` and judge guidance unless the user explicitly wants two separate scoring systems and understands only the custom metric drives optimization.
- If the user builds a plain `AxGen` judge metric, prefer a numeric `score:number` output over a string tier when possible. It is simpler and less fragile in practice.
## Canonical Pattern
```typescript
import {
AxAIGoogleGeminiModel,
AxJSRuntime,
AxOptimizedProgramImpl,
axDefaultOptimizerLogger,
agent,
ai,
f,
fn,
} from '@ax-llm/ax';
const tools = [
fn('sendEmail')
.namespace('email')
.description('Send an email message')
.arg('to', f.string('Recipient email address'))
.arg('body', f.string('Email body text'))
.returns(
f.object({
sent: f.boolean('Whether the email was sent'),
to: f.string('Recipient email address'),
})
)
.handler(async ({ to }) => ({ sent: true, to }))
.build(),
];
const studentAI = ai({
name: 'google-gemini',
apiKey: process.env.GOOGLE_APIKEY!,
config: { model: AxAIGoogleGeminiModel.Gemini25FlashLite, temperature: 0.2 },
});
const judgeAI = ai({
name: 'google-gemini',
apiKey: process.env.GOOGLE_APIKEY!,
config: { model: AxAIGoogleGeminiModel.Gemini3Pro, temperature: 1.0 },
});
const assistant = agent('query:string -> answer:string', {
ai: studentAI,
judgeAI,
contextFields: [],
runtime: new AxJSRuntime(),
functions: { local: tools },
contextPolicy: { preset: 'checkpointed', budget: 'balanced' },
judgeOptions: {
description: 'Prefer correct tool use over polished wording.',
model: 'judge-model',
},
});
const tasks = [
{
input: { query: 'Send an email to Jim saying good morning.' },
criteria: 'Use the email tool and send the message to Jim.',
expectedActions: ['email.sendEmail'],
},
];
const result = await assistant.optimize(tasks, {
target: 'actor',
maxMetricCalls: 12,
verbose: true,
optimizerLogger: axDefaultOptimizerLogger,
onProgress: (progress) => {
console.log(
`round ${progress.round}/${progress.totalRounds} current=${progress.currentScore} best=${progress.bestScore}`
);
},
});
const saved = JSON.stringify(result.optimizedProgram, null, 2);
const restored = new AxOptimizedProgramImpl(JSON.parse(saved));
assistant.applyOptimization(restored);
```
## Deterministic Metric Pattern
Use this when the task has crisp correctness and cost/behavior tradeoffs:
```typescript
const result = await assistant.optimize(tasks, {
target: 'actor',
metric: ({ prediction, example }) => {
if (prediction.completionType !== 'final' || !prediction.output) {
return 0;
}
let score = 0;
if (prediction.output.answer.includes('Jim')) score += 0.4;
if (
prediction.functionCalls.some(
(call) => call.qualifiedName === 'email.sendEmail'
)
) {
score += 0.4;
}
if ((prediction.recursiveStats?.recursiveCallCount ?? 0) === 0) {
score += 0.2;
}
return score;
},
});
```
Use this pattern when:
- the task has a known correct answer or exact action pattern
- recursion cost or tool count must be measured explicitly
- you want repeatable, low-variance optimization runs
## Built-In Judge Pattern
Use this when the agent behavior needs holistic review:
```typescript
const result = await assistant.optimize(tasks, {
judgeAI,
judgeOptions: {
model: AxAIGoogleGeminiModel.Gemini3Pro,
description:
'Be strict about unnecessary delegation, weak clarifications, and incorrect tool choices.',
},
maxMetricCalls: 12,
});
```
Use this pattern when:
- task quality is open-ended or hard to score exactly
- the final answer quality matters together with the action trace
- the user wants a judge to consider clarifications, tool errors, and overall completion quality
## Plain `AxGen` Judge Pattern
Use this only when the user needs LLM judging outside the built-in `agent.optimize(...)` path:
```typescript
import { AxGen, s } from '@ax-llm/ax';
const judgeGen = new AxGen(
s(`
taskInput:json "Task input",
candidateOutput:json "Candidate output",
expectedOutput?:json "Optional reference output"
->
score:number "Normalized score from 0 to 1"
`)
);
judgeGen.setInstruction(
'Score the candidate output from 0 to 1. Reward correctness and task completion. Return only the score field.'
);
const metric = async ({ prediction, example }) => {
const result = await judgeGen.forward(judgeAI, {
taskInput: example,
candidateOutput: prediction,
expectedOutput: example.expectedOutput,
});
return Math.max(0, Math.min(1, result.score));
};
const result = await optimizer.compile(program, train, metric, {
validationExamples: validation,
});
```
Use this pattern when:
- the user is optimizing an `AxGen`, flow, or another program directly
- the user wants LLM judging without the higher-level `agent.optimize(...)` wrapper
- the user wants to inspect judge results directly, not just a numeric score
## Dataset And Judge Rules
- Pass already-loaded tasks. Do not invent a benchmark loader unless the user asks for one.
- Use `expectedActions` and `forbiddenActions` when tool correctness matters.
- `judgeOptions` mirrors normal forward options and supports extra judge guidance through `description`.
- The built-in judge scores from the full agent run, not just the final reply. It can see completion type, clarification payload, final output, action log, normalized function calls, tool errors, and turn count.
- For recursive advanced-mode evals, the built-in judge can also see `recursiveTrace` and `recursiveStats`.
- If the user provides a custom `metric`, that overrides the built-in judge path.
- If the user provides an LLM-based custom metric, keep the output schema as small as possible and prefer a direct numeric score.
Decision rules:
- Prefer a custom metric when the user has deterministic business scoring, exact action expectations, or explicit cost tradeoffs.
- Prefer the built-in judge when the user wants practical assistant-quality tuning and does not already have a trusted metric.
- Prefer a plain typed `AxGen` evaluator when the user is not calling `agent.optimize(...)` but still wants LLM judging.
- Prefer `judgeOptions.description` to steer the judge toward the user's real priority, such as tool correctness, brevity, groundedness, or policy compliance.
## Eval Semantics
- `agent.optimize(...)` runs each evaluation rollout from a clean continuation state.
- Saved runtime state from `getState()` and `setState(...)` is not used during eval rollouts.
- During optimize/eval, `askClarification(...)` is treated as a scored evaluation outcome instead of going through the responder.
- For clarification outcomes in custom metrics, expect `prediction.completionType === 'askClarification'`, populated `prediction.clarification`, and absent `prediction.output`.
- For final outcomes in custom metrics, expect `prediction.completionType === 'final'` and populated `prediction.output`.
- `target: 'responder'` still works, but clarification-heavy tasks are usually low-signal for responder optimization.
## Recursive Optimization Notes
- Recursive-slot artifacts require an agent configured for recursive advanced mode.
- Keep `mode: 'advanced'` top-level; child recursion behavior still follows `recursionOptions`.
- When recursive behavior matters, tune against the same `maxDepth` and tool/discovery structure you expect in production.
- Use recursive traces and recursive stats when the user wants to diagnose where token or delegation cost is coming from.
- For recursion-efficiency tuning, prefer a deterministic metric unless the user specifically needs a qualitative LLM review of decomposition quality.
- Tell the actor that recursive children only see passed context, not parent globals or prior tool results.
- For synthesis-style recursive tasks, specify the desired delegation pattern explicitly, for example "use at most one focused delegated child analysis after narrowing the tool output in JS."
- Penalize over-decomposition directly in the metric or judge prompt.
- If one training task keeps collapsing to zero, inspect that task first instead of adding more optimizer rounds. Most failures come from task ambiguity, weak tool schemas, or vague delegation guidance rather than GEPA itself.
## Artifacts And Replay
- Save `result.optimizedProgram` if the user wants portable artifacts.
- Restore artifacts with `new AxOptimizedProgramImpl(...)`, then call `agent.applyOptimization(...)`.
- For demonstrations, use fresh eval-safe tool state for baseline, optimize, and restored replay so side effects do not leak across phases.
- If the user wants to show improvement, run a held-out task before optimization, then replay it on a freshly restored optimized agent.
## Examples
- [RLM Agent Optimize](https://raw.githubusercontent.com/ax-llm/ax/refs/heads/main/src/examples/rlm-agent-optimize.ts) — Gemini office-assistant tuning with save/load
- [RLM Agent Recursive Optimize](https://raw.githubusercontent.com/ax-llm/ax/refs/heads/main/src/examples/rlm-agent-recursive-optimize.ts) — recursive-slot optimization artifacts
## Do Not Generate
- Do not optimize against production tools with real side effects unless the user explicitly wants that.
- Do not recommend responder-only optimization by default for clarification-heavy workflows.
- Do not omit artifact save/load steps when the user asks for reusable optimized configurations.
- Do not introduce a dedicated judge class or helper abstraction in new agent-optimize examples; prefer the built-in judge path or a plain typed `AxGen`.
- Do not rely on vague `json` tool returns when the agent must reason about specific fields across recursive steps.
- Do not leave recursive child context implicit. If the child needs a fact, pass it explicitly.
- Do not let code-generation agents mix prose and JavaScript if the user is optimizing runtime behavior.

File diff suppressed because it is too large Load Diff

View File

@@ -1,245 +0,0 @@
---
name: ax-ai
description: This skill helps an LLM generate correct AI provider setup and configuration code using @ax-llm/ax. Use when the user asks about ai(), providers, models, presets, embeddings, extended thinking, context caching, or mentions OpenAI/Anthropic/Google/Azure/Groq/DeepSeek/Mistral/Cohere/Together/Ollama/HuggingFace/Reka/OpenRouter with @ax-llm/ax.
version: "19.0.33"
---
# AI Provider Codegen Rules (@ax-llm/ax)
Use this skill to generate AI provider setup, configuration, and chat code. Prefer short, modern, copyable patterns. Do not write tutorial prose unless the user explicitly asks for explanation.
## Quick Setup
```typescript
import { ai } from '@ax-llm/ax';
const openai = ai({ name: 'openai', apiKey: 'sk-...' });
const Codex = ai({ name: 'anthropic', apiKey: 'sk-ant-...' });
const gemini = ai({ name: 'google-gemini', apiKey: 'AIza...' });
const azure = ai({ name: 'azure-openai', apiKey: 'your-key', resourceName: 'your-resource', deploymentName: 'gpt-4' });
const groq = ai({ name: 'groq', apiKey: 'gsk_...' });
const deepseek = ai({ name: 'deepseek', apiKey: 'sk-...' });
const mistral = ai({ name: 'mistral', apiKey: 'your-key' });
const cohere = ai({ name: 'cohere', apiKey: 'your-key' });
const together = ai({ name: 'together', apiKey: 'your-key' });
const openrouter = ai({ name: 'openrouter', apiKey: 'your-key' });
const ollama = ai({ name: 'ollama', url: 'http://localhost:11434' });
const hf = ai({ name: 'huggingface', apiKey: 'hf_...' });
const reka = ai({ name: 'reka', apiKey: 'your-key' });
const grok = ai({ name: 'x-grok', apiKey: 'your-key' });
```
## Model Presets
```typescript
import { ai, AxAIGoogleGeminiModel } from '@ax-llm/ax';
const gemini = ai({
name: 'google-gemini',
apiKey: process.env.GOOGLE_APIKEY!,
config: { model: 'simple' },
models: [
{ key: 'tiny', model: AxAIGoogleGeminiModel.Gemini20FlashLite, description: 'Fast + cheap', config: { maxTokens: 1024, temperature: 0.3 } },
{ key: 'simple', model: AxAIGoogleGeminiModel.Gemini20Flash, description: 'Balanced', config: { temperature: 0.6 } },
],
});
await gemini.chat({ model: 'tiny', chatPrompt: [{ role: 'user', content: 'Hi' }] });
```
## Chat
```typescript
const res = await llm.chat({
chatPrompt: [
{ role: 'system', content: 'You are concise.' },
{ role: 'user', content: 'Write a haiku about the ocean.' },
],
});
console.log(res.results[0]?.content);
```
## Common Options
- `stream` (boolean): enable SSE; true by default
- `thinkingTokenBudget`: `'minimal'` | `'low'` | `'medium'` | `'high'` | `'highest'` | `'none'`
- `showThoughts`: include thoughts in output
- `functionCallMode`: `'auto'` | `'native'` | `'prompt'`
- `debug`, `logger`, `tracer`, `rateLimiter`, `timeout`
## Extended Thinking
```typescript
import { ai, AxAIAnthropicModel } from '@ax-llm/ax';
const Codex = ai({
name: 'anthropic',
apiKey: process.env.ANTHROPIC_APIKEY!,
config: { model: AxAIAnthropicModel.Claude46Opus },
});
const res = await Codex.chat(
{ chatPrompt: [{ role: 'user', content: 'Solve step by step...' }] },
{ thinkingTokenBudget: 'medium', showThoughts: true },
);
console.log(res.results[0]?.thought);
console.log(res.results[0]?.content);
```
### Budget Levels
| Level | Anthropic (tokens) | Gemini (tokens) |
|---|---|---|
| `'none'` | disabled | minimal |
| `'minimal'` | 1,024 | 200 |
| `'low'` | 5,000 | 800 |
| `'medium'` | 10,000 | 5,000 |
| `'high'` | 20,000 | 10,000 |
| `'highest'` | 32,000 | 24,500 |
### Anthropic Model-Specific Behavior
- Opus 4.6: adaptive thinking, effort levels
- Opus 4.5: budget_tokens + effort levels (capped at `'high'`)
- Other thinking models: budget tokens only
### Custom Thinking Levels
```typescript
const Codex = ai({
name: 'anthropic',
apiKey: '...',
config: {
model: AxAIAnthropicModel.Claude46Opus,
thinkingTokenBudgetLevels: {
minimal: 2048,
low: 8000,
medium: 16000,
high: 25000,
highest: 40000,
},
effortLevelMapping: {
minimal: 'low',
low: 'medium',
medium: 'high',
high: 'high',
highest: 'max',
},
},
});
```
## Embeddings
```typescript
const { embeddings } = await llm.embed({
texts: ['hello', 'world'],
embedModel: 'text-embedding-005',
});
```
## Context Caching
```typescript
const result = await gen.forward(llm, { code, language }, {
mem,
sessionId: 'code-review-session',
contextCache: {
ttlSeconds: 3600,
cacheBreakpoint: 'after-examples',
},
});
```
Breakpoint values: `'system'` | `'after-functions'` | `'after-examples'`
Provider behavior:
- Google Gemini: explicit caching with cache resource ID, auto TTL refresh
- Anthropic: implicit via `cache_control` markers
### External Registry (serverless)
```typescript
const registry: AxContextCacheRegistry = {
get: async (key) => { /* redis.get */ },
set: async (key, entry) => { /* redis.set */ },
};
```
## AWS Bedrock
```typescript
import { AxAIBedrock, AxAIBedrockModel } from '@ax-llm/ax-ai-aws-bedrock';
const bedrock = new AxAIBedrock({
region: 'us-east-2',
fallbackRegions: ['us-west-2'],
config: { model: AxAIBedrockModel.ClaudeSonnet4 },
});
```
## Vercel AI SDK Integration
```typescript
import { ai } from '@ax-llm/ax';
import { AxAIProvider } from '@ax-llm/ax-ai-sdk-provider';
import { generateText } from 'ai';
const axAI = ai({ name: 'openai', apiKey: process.env.OPENAI_APIKEY! });
const model = new AxAIProvider(axAI);
const result = await generateText({
model,
messages: [{ role: 'user', content: 'Hello!' }],
});
```
## MCP + AxJSRuntime
```typescript
import { AxMCPClient } from '@ax-llm/ax';
import { axCreateMCPStdioTransport } from '@ax-llm/ax-tools';
const transport = axCreateMCPStdioTransport({
command: 'npx',
args: ['-y', '@anthropic/mcp-server-filesystem'],
});
const client = new AxMCPClient(transport);
```
## Critical Rules
- Use `ai()` factory for all providers.
- Provider names: `'openai'`, `'anthropic'`, `'google-gemini'`, `'azure-openai'`, `'mistral'`, `'groq'`, `'cohere'`, `'together'`, `'deepseek'`, `'ollama'`, `'huggingface'`, `'openrouter'`, `'reka'`, `'x-grok'`
- Thinking constraints on Anthropic: `temperature` and `topK` are ignored; `topP` only sent if >= 0.95.
- Bedrock uses `new AxAIBedrock()`, not `ai()`.
- Vercel AI SDK uses `AxAIProvider` wrapper.
## Examples
Fetch these for full working code:
- [Embeddings](https://raw.githubusercontent.com/ax-llm/ax/refs/heads/main/src/examples/embed.ts) — embedding generation
- [Anthropic Thinking](https://raw.githubusercontent.com/ax-llm/ax/refs/heads/main/src/examples/anthropic-thinking-function.ts) — extended thinking with functions
- [Anthropic Thinking Separation](https://raw.githubusercontent.com/ax-llm/ax/refs/heads/main/src/examples/anthropic-thinking-separation.ts) — thinking separation
- [Anthropic Web Search](https://raw.githubusercontent.com/ax-llm/ax/refs/heads/main/src/examples/anthropic-web-search.ts) — Anthropic web search
- [OpenAI Web Search](https://raw.githubusercontent.com/ax-llm/ax/refs/heads/main/src/examples/openai-web-search.ts) — OpenAI web search
- [OpenAI Responses](https://raw.githubusercontent.com/ax-llm/ax/refs/heads/main/src/examples/openai-responses.ts) — OpenAI responses API
- [o3 Reasoning](https://raw.githubusercontent.com/ax-llm/ax/refs/heads/main/src/examples/reasoning-o3-example.ts) — o3 reasoning
- [Gemini Context Cache](https://raw.githubusercontent.com/ax-llm/ax/refs/heads/main/src/examples/gemini-context-cache.ts) — Gemini context caching
- [Gemini Files](https://raw.githubusercontent.com/ax-llm/ax/refs/heads/main/src/examples/gemini-file-support.ts) — Gemini file handling
- [Grok Live Search](https://raw.githubusercontent.com/ax-llm/ax/refs/heads/main/src/examples/grok-live-search.ts) — Grok live search
- [OpenRouter](https://raw.githubusercontent.com/ax-llm/ax/refs/heads/main/src/examples/openrouter.ts) — OpenRouter provider
- [Vertex AI Auth](https://raw.githubusercontent.com/ax-llm/ax/refs/heads/main/src/examples/vertex-auth-example.ts) — Vertex AI authentication
- [MCP Stdio](https://raw.githubusercontent.com/ax-llm/ax/refs/heads/main/src/examples/mcp-client-memory.ts) — MCP stdio transport
- [MCP HTTP](https://raw.githubusercontent.com/ax-llm/ax/refs/heads/main/src/examples/mcp-client-pipedream.ts) — MCP HTTP transport
- [Telemetry](https://raw.githubusercontent.com/ax-llm/ax/refs/heads/main/src/examples/telemetry.ts) — OpenTelemetry tracing
- [Multi-Modal](https://raw.githubusercontent.com/ax-llm/ax/refs/heads/main/src/examples/multi-modal.ts) — image handling
## Do Not Generate
- Do not use `new AxAIOpenAI(...)` or similar class constructors for standard providers; use `ai()`.
- Do not hardcode provider class names when `ai({ name: ... })` covers the provider.
- Do not mix `thinkingTokenBudget` with explicit `temperature` on Anthropic thinking models.
- Do not use `ai()` for AWS Bedrock; use `new AxAIBedrock()`.
- Do not omit `resourceName` and `deploymentName` for Azure OpenAI.

View File

@@ -1,402 +0,0 @@
---
name: ax-flow
description: This skill helps an LLM generate correct AxFlow workflow code using @ax-llm/ax. Use when the user asks about flow(), AxFlow, workflow orchestration, parallel execution, DAG workflows, conditional routing, map/reduce patterns, or multi-node AI pipelines.
version: "19.0.33"
---
# AxFlow Codegen Rules (@ax-llm/ax)
Use this skill to generate `AxFlow` workflow code. Prefer short, modern, copyable patterns. Do not write tutorial prose unless the user explicitly asks for explanation.
## Use These Defaults
- Use `flow()` factory, not `new AxFlow()`.
- Import: `import { ai, flow, f } from '@ax-llm/ax';`
- `autoParallel: true` is the default; independent executes run in parallel automatically.
- Node results are stored as `${nodeName}Result` in state.
- Always define `.node()` before `.execute()` for that node.
- Use `.returns()` (or `.r()`) as the last step to lock the output type.
- Use descriptive node names: `documentSummarizer`, not `proc1`.
- Use descriptive field names: `userInput`, `responseText`, not `text`, `result`.
## Critical Rules
- Use `flow()` factory syntax for new code.
- Node results in state follow the pattern `state.${nodeName}Result.${fieldName}`.
- `.execute()` maps current state to node inputs; `.map()` transforms state without AI calls.
- `.returns()` maps final state to the flow output type.
- Always define nodes before executing them; reversed order throws at runtime.
- Keep state flat; avoid deep nesting in `.map()`.
- Ensure loop conditions can change to avoid infinite loops.
- Structure independent executes to maximize auto-parallelization.
- Use `flow<InputType, OutputType>()` for typed flows.
- Aliases: `.n()` = `.node()`, `.nx()` = `.nodeExtended()`, `.m()` = `.map()`, `.r()` = `.returns()`.
## Canonical Pattern
```typescript
import { ai, flow } from '@ax-llm/ax';
const llm = ai({ name: 'openai', apiKey: process.env.OPENAI_APIKEY! });
const wf = flow<{ userInput: string }, { responseText: string }>()
.node('testNode', 'userInput:string -> responseText:string')
.execute('testNode', (state) => ({ userInput: state.userInput }))
.returns((state) => ({ responseText: state.testNodeResult.responseText }));
const result = await wf.forward(llm, { userInput: 'Hello world' });
console.log(result.responseText);
```
## Factory Options
```typescript
// Basic
const wf = flow();
// With options
const wf = flow({ autoParallel: false });
// Typed
const wf = flow<InputType, OutputType>();
// Typed with options
const wf = flow<InputType, OutputType>({ autoParallel: true, batchSize: 5 });
```
## State Evolution
State grows with each executed node. Results are stored as `${nodeName}Result`:
```typescript
// Initial state: { userInput: 'Hello' }
flow.execute('processor', (state) => ({ input: state.userInput }));
// State: { userInput: 'Hello', processorResult: { output: '...' } }
flow.execute('analyzer', (state) => ({ text: state.processorResult.output }));
// State: { ..., analyzerResult: { sentiment: '...', confidence: 0.8 } }
```
## Node Definition
```typescript
// String signature (creates AxGen automatically)
flow.node('processor', 'input:string -> output:string');
// Multiple outputs
flow.node('analyzer', 'text:string -> sentiment:string, confidence:number');
// Array outputs
flow.node('extractor', 'documentText:string -> entities:string[]');
// Short alias
flow.n('processor', 'input:string -> output:string');
```
## Extended Nodes (nx)
Add fields to a base signature without rewriting it:
```typescript
import { f, flow } from '@ax-llm/ax';
// Chain-of-thought reasoning
flow.nx('reasoner', 'question:string -> answer:string', {
prependOutputs: [
{ name: 'reasoning', type: f.internal(f.string('Step-by-step reasoning')) },
],
});
// Add confidence scoring
flow.nx('analyzer', 'input:string -> result:string', {
appendOutputs: [{ name: 'confidence', type: f.number('Confidence 0-1') }],
});
// Add optional context input
flow.nx('processor', 'query:string -> response:string', {
appendInputs: [{ name: 'context', type: f.optional(f.string('Extra context')) }],
});
```
Extension options: `prependInputs`, `appendInputs`, `prependOutputs`, `appendOutputs`.
## Execute With Input Mapping
```typescript
flow.execute('summarizer', (state) => ({ documentText: state.document }));
// With AI override (use a different model for this node)
flow.execute('processor', (state) => ({ input: state.data }), { ai: alternativeAI });
```
## Map (State Transformation)
Use `map()` for data shaping without AI calls:
```typescript
// Sync
flow.map((state) => ({ ...state, upperText: state.rawText.toUpperCase() }));
// Async
flow.map(async (state) => {
const data = await fetchFromAPI(state.query);
return { ...state, enrichedData: data };
});
// Parallel async transforms
flow.map([
async (state) => ({ ...state, result1: await api1(state.data) }),
async (state) => ({ ...state, result2: await api2(state.data) }),
], { parallel: true });
```
## Returns (Final Output)
```typescript
const wf = flow<{ input: string }>()
.map((state) => ({ ...state, upper: state.input.toUpperCase(), len: state.input.length }))
.returns((state) => ({ upper: state.upper, isLong: state.len > 20 }));
// Result is typed as { upper: string; isLong: boolean }
const result = await wf.forward(llm, { input: 'test' });
```
## Sequential Processing
```typescript
const wf = flow<{ input: string }, { finalResult: string }>()
.node('step1', 'input:string -> intermediate:string')
.node('step2', 'intermediate:string -> output:string')
.execute('step1', (state) => ({ input: state.input }))
.execute('step2', (state) => ({ intermediate: state.step1Result.intermediate }))
.returns((state) => ({ finalResult: state.step2Result.output }));
```
## Auto-Parallel Execution
Independent executes run in parallel automatically (`autoParallel: true` by default):
```typescript
const wf = flow<{ text: string }, { combined: string }>()
.node('sentimentAnalyzer', 'text:string -> sentiment:string')
.node('topicExtractor', 'text:string -> topics:string[]')
.node('entityRecognizer', 'text:string -> entities:string[]')
// These three run in parallel (all depend only on state.text)
.execute('sentimentAnalyzer', (state) => ({ text: state.text }))
.execute('topicExtractor', (state) => ({ text: state.text }))
.execute('entityRecognizer', (state) => ({ text: state.text }))
// This waits for all three
.returns((state) => ({
combined: JSON.stringify({
sentiment: state.sentimentAnalyzerResult.sentiment,
topics: state.topicExtractorResult.topics,
entities: state.entityRecognizerResult.entities,
}),
}));
// Inspect execution plan
const plan = wf.getExecutionPlan();
console.log(plan.parallelGroups, plan.maxParallelism);
```
Disable auto-parallel:
```typescript
const wf = flow({ autoParallel: false });
// or per execution:
await wf.forward(llm, input, { autoParallel: false });
```
## Conditional Branching
```typescript
const wf = flow<{ query: string; expertMode: boolean }, { response: string }>()
.node('simple', 'query:string -> response:string')
.node('expert', 'query:string -> response:string')
.branch((state) => state.expertMode)
.when(true)
.execute('expert', (state) => ({ query: state.query }))
.when(false)
.execute('simple', (state) => ({ query: state.query }))
.merge()
.returns((state) => ({
response: state.expertResult?.response ?? state.simpleResult?.response,
}));
```
After `.merge()`, only the taken branch's result exists; use optional chaining (`?.`) on untaken branch results.
## While Loops
```typescript
const wf = flow<{ content: string }, { finalContent: string }>()
.node('processor', 'content:string -> processedContent:string')
.node('qualityChecker', 'content:string -> qualityScore:number')
.map((state) => ({ currentContent: state.content, iteration: 0, qualityScore: 0 }))
.while((state) => state.iteration < 3 && state.qualityScore < 0.8)
.map((state) => ({ ...state, iteration: state.iteration + 1 }))
.execute('processor', (state) => ({ content: state.currentContent }))
.execute('qualityChecker', (state) => ({
content: state.processorResult.processedContent,
}))
.map((state) => ({
...state,
currentContent: state.processorResult.processedContent,
qualityScore: state.qualityCheckerResult.qualityScore,
}))
.endWhile()
.returns((state) => ({ finalContent: state.currentContent }));
```
Rules:
- Every `.while()` needs a matching `.endWhile()`.
- Ensure the loop condition can change to avoid infinite loops.
## Feedback Loops (label/feedback)
```typescript
const wf = flow<{ prompt: string }, { result: string }>()
.node('gen', 'prompt:string -> result:string, quality:number')
.map((state) => ({ ...state, tries: 0 }))
.label('retry')
.map((state) => ({ ...state, tries: state.tries + 1 }))
.execute('gen', (state) => ({ prompt: state.prompt }))
.feedback((state) => state.genResult.quality < 0.9 && state.tries < 3, 'retry')
.returns((state) => ({ result: state.genResult.result }));
```
Rules:
- Define the label before referencing it in `.feedback()`.
- Always include a max-iteration guard to avoid infinite loops.
## Explicit Parallel Sub-Flows
```typescript
flow
.parallel([
(sub) => sub.execute('analyzer1', (state) => ({ text: state.input })),
(sub) => sub.execute('analyzer2', (state) => ({ text: state.input })),
(sub) => sub.execute('analyzer3', (state) => ({ text: state.input })),
])
.merge('combinedResults', (r1, r2, r3) => ({
a1: r1.analyzer1Result.analysis,
a2: r2.analyzer2Result.analysis,
a3: r3.analyzer3Result.analysis,
}));
```
## Derive (Batch/Array Processing)
```typescript
const wf = flow<{ items: string[] }, { processed: string[] }>({ batchSize: 3 })
.derive('processed', 'items', (item, index) => `processed-${item}-${index}`, {
batchSize: 2,
});
```
## Dynamic AI Context (Multi-Model)
Route nodes to different AI providers:
```typescript
const fast = ai({ name: 'groq', apiKey: '...' });
const smart = ai({ name: 'anthropic', apiKey: '...' });
const wf = flow<{ text: string }, { out: string }>()
.node('draft', 'text:string -> out:string')
.node('refine', 'text:string -> out:string')
.execute('draft', (state) => ({ text: state.text }), { ai: fast })
.execute('refine', (state) => ({ text: state.draftResult.out }), { ai: smart })
.returns((state) => ({ out: state.refineResult.out }));
```
## Description and toFunction
```typescript
const wf = flow<{ userQuestion: string }, { responseText: string }>()
.node('qa', 'userQuestion:string -> responseText:string')
.execute('qa', (state) => ({ userQuestion: state.userQuestion }))
.returns((state) => ({ responseText: state.qaResult.responseText }))
.description('Question Answerer', 'Answers user questions concisely.');
const fn = wf.toFunction();
// fn.name, fn.parameters (JSON Schema), fn.func
```
## Instrumentation (Tracing)
```typescript
import { ai, flow } from '@ax-llm/ax';
import { context, trace } from '@opentelemetry/api';
const tracer = trace.getTracer('axflow');
const llm = ai({ name: 'openai', apiKey: '...' });
const wf = flow<{ userQuestion: string }>()
.node('summarizer', 'documentText:string -> summaryText:string')
.execute('summarizer', (s) => ({ documentText: s.userQuestion }))
.returns((s) => ({ answer: s.summarizerResult.summaryText }));
const result = await wf.forward(llm, { userQuestion: 'hi' }, {
tracer,
traceContext: context.active(),
});
```
## Program IDs and Demos
```typescript
const wf = flow<{ input: string }>()
.node('summarizer', 'text:string -> summary:string')
.node('classifier', 'text:string -> category:string');
// Discover program IDs
console.log(wf.namedPrograms());
// [{ id: 'root.summarizer', ... }, { id: 'root.classifier', ... }]
// Set demos (TypeScript catches typos)
wf.setDemos([{ programId: 'root.summarizer', traces: [] }]);
// Apply optimization
wf.applyOptimization(optimizedProgram);
```
## Error Handling
```typescript
try {
const result = await wf.forward(llm, input);
} catch (error) {
console.error('Flow execution failed:', error);
}
```
Common errors:
- `"Node 'x' not found"` -- define `.node()` before `.execute()`.
- `"endWhile() without matching while()"` -- every `.while()` needs `.endWhile()`.
- `"when() without matching branch()"` -- `.when()` must be inside `.branch()`/`.merge()`.
- `"merge() without matching branch()"` -- every `.branch()` needs `.merge()`.
- `"Label 'x' not found"` -- define `.label()` before `.feedback()` references it.
## Examples
Fetch these for full working code:
- [Flow](https://raw.githubusercontent.com/ax-llm/ax/refs/heads/main/src/examples/ax-flow.ts) — complete flow usage
- [Auto-Parallel](https://raw.githubusercontent.com/ax-llm/ax/refs/heads/main/src/examples/ax-flow-auto-parallel.ts) — auto-parallelization
- [Async Map](https://raw.githubusercontent.com/ax-llm/ax/refs/heads/main/src/examples/ax-flow-async-map.ts) — async map transforms
- [Enhanced Demo](https://raw.githubusercontent.com/ax-llm/ax/refs/heads/main/src/examples/ax-flow-enhanced-demo.ts) — instance-based nodes
- [Flow as Function](https://raw.githubusercontent.com/ax-llm/ax/refs/heads/main/src/examples/ax-flow-to-function.ts) — flow as callable function
- [Fluent Builder](https://raw.githubusercontent.com/ax-llm/ax/refs/heads/main/src/examples/fluent-flow-example.ts) — fluent builder pattern
- [Flow Logging](https://raw.githubusercontent.com/ax-llm/ax/refs/heads/main/src/examples/flow-logging-simple.ts) — flow logging
- [Load Balancing](https://raw.githubusercontent.com/ax-llm/ax/refs/heads/main/src/examples/balancer.ts) — load balancing
## Do Not Generate
- Do not use `new AxFlow(...)` for new code.
- Do not execute a node before defining it with `.node()`.
- Do not use generic field names like `text`, `result`, `data`, `input`, `output`.
- Do not create deep-nested state objects in `.map()`.
- Do not create loop conditions that can never change.
- Do not add unnecessary dependencies between executes (kills auto-parallelism).
- Do not forget to use optional chaining on branch results after `.merge()`.

View File

@@ -1,323 +0,0 @@
---
name: ax-gen
description: This skill helps an LLM generate correct AxGen code using @ax-llm/ax. Use when the user asks about ax(), AxGen, generators, forward(), streamingForward(), assertions, field processors, step hooks, self-tuning, or structured outputs.
version: "19.0.33"
---
# AxGen Codegen Rules (@ax-llm/ax)
Use this skill to generate `AxGen` code. Prefer short, modern, copyable patterns. Do not write tutorial prose unless the user explicitly asks for explanation.
## Use These Defaults
- Use `ax(...)` factory, not `new AxGen(...)`.
- Always pass an AI instance from `ai(...)` as the first argument to `forward()`.
- Streaming uses `streamingForward()`, not `forward()` with a stream option.
- Assertions auto-retry with error feedback on failure.
- Step hook mutations are applied at the next step boundary (pending pattern).
- `stopFunction` accepts a string or string[] for multiple stop functions.
- Multi-step continues until: all outputs filled, stop function called, or `maxSteps` reached.
## Canonical Pattern
```typescript
import { ai, ax, s } from '@ax-llm/ax';
const llm = ai({
name: 'openai',
apiKey: process.env.OPENAI_APIKEY!,
});
// Inline signature
const gen = ax('input:string -> output:string, reasoning:string');
// Reusable signature
const sig = s('question:string, context:string[] -> answer:string');
const gen2 = ax(sig);
// With options
const gen3 = ax('input -> output', {
description: 'A helpful assistant',
maxRetries: 3,
maxSteps: 10,
temperature: 0.7,
});
const result = await gen.forward(llm, { input: 'Hello world' });
console.log(result.output);
```
## Running AxGen
### `forward()`
```typescript
const result = await gen.forward(llm, { input: '...' });
// With options
const result = await gen.forward(llm, { input: '...' }, {
maxRetries: 5,
model: 'gpt-4.1',
modelConfig: { temperature: 0.9, maxTokens: 1000 },
debug: true,
});
```
### `streamingForward()`
```typescript
const stream = gen.streamingForward(llm, { input: 'Write a long story' });
for await (const chunk of stream) {
if (chunk.delta.output) process.stdout.write(chunk.delta.output);
}
```
## Stopping And Cancellation
```typescript
import { AxAIServiceAbortedError } from '@ax-llm/ax';
const timer = setTimeout(() => gen.stop(), 3_000);
try {
const result = await gen.forward(llm, { topic: 'Long document' }, {
abortSignal: AbortSignal.timeout(10_000),
});
} catch (err) {
if (err instanceof AxAIServiceAbortedError) console.log('Aborted');
}
```
Rules:
- `gen.stop()` gracefully stops multi-step execution at the next step boundary.
- `abortSignal` cancels the underlying AI service call immediately.
- Catch `AxAIServiceAbortedError` when using either mechanism.
## Assertions And Validation
```typescript
// Standard assertion (checked after forward completes)
gen.addAssert(
(args) => args.output.length > 50,
'Output must be at least 50 characters'
);
// Streaming assertion (checked during streaming)
gen.addStreamingAssert(
'output',
(text) => !text.includes('forbidden'),
'Output contains forbidden text'
);
```
Rules:
- Failed assertions cause an automatic retry with the error message fed back to the LLM.
- `addAssert` receives the full output object.
- `addStreamingAssert` targets a specific field and receives the partial text so far.
## Field Processors
```typescript
// Post-processing after generation
gen.addFieldProcessor('summary', (value, context) => value.toUpperCase());
// Streaming field processor (called on each chunk)
gen.addStreamingFieldProcessor('content', (partialValue, context) => {
console.log(`Received ${partialValue.length} chars`);
return partialValue;
});
```
Rules:
- `addFieldProcessor` runs once after the field is fully generated.
- `addStreamingFieldProcessor` runs on each streaming chunk for the target field.
- Both must return the (possibly transformed) value.
## Function Calling
```typescript
const result = await gen.forward(llm, { question: '...' }, {
functions: tools,
functionCallMode: 'auto',
stopFunction: 'finalAnswer',
});
```
Rules:
- `functionCallMode` can be `'auto'`, `'none'`, or a specific function name to force.
- `stopFunction` accepts a string or string[] to halt multi-step on specific function calls.
- Multi-step continues until all outputs filled, stop function called, or `maxSteps` reached.
## Caching
### Response Caching
```typescript
const gen = ax('question:string -> answer:string', {
cachingFunction: async (key, value?) => {
if (value !== undefined) {
await cache.set(key, value);
return;
}
return await cache.get(key);
},
});
```
### Context Caching
```typescript
const result = await gen.forward(llm, { question: '...' }, {
contextCache: { cacheBreakpoint: 'after-examples' },
});
```
Rules:
- `cachingFunction` acts as a get/set: called with `(key)` to read, `(key, value)` to write.
- `contextCache` enables AI provider-level prompt caching for long context.
## Sampling And Result Picker
```typescript
const result = await gen.forward(llm, { question: '...' }, {
sampleCount: 3,
resultPicker: async (samples) => {
// Evaluate each sample and return the index of the best one
return bestIndex;
},
});
```
Rules:
- `sampleCount` generates multiple completions in parallel.
- `resultPicker` receives all samples and must return the index of the chosen result.
## Extended Thinking
```typescript
const result = await gen.forward(llm, { question: '...' }, {
thinkingTokenBudget: 'medium',
showThoughts: true,
});
console.log(result.thought);
```
Rules:
- `thinkingTokenBudget` can be `'low'`, `'medium'`, `'high'`, or a number.
- Set `showThoughts: true` to include the model's reasoning in `result.thought`.
## Step Hooks
```typescript
const result = await gen.forward(llm, values, {
stepHooks: {
beforeStep: (ctx) => {
if (ctx.functionsExecuted.has('complexanalysis')) {
ctx.setModel('smart');
ctx.setThinkingBudget('high');
}
},
afterStep: (ctx) => {
console.log(`Usage: ${ctx.usage.totalTokens} tokens`);
},
},
});
```
### AxStepContext Read-Only Properties
- `stepIndex` - current step number
- `maxSteps` - configured maximum steps
- `isFirstStep` - whether this is the first step
- `functionsExecuted` - `Set<string>` of function names called so far
- `lastFunctionCalls` - array of the most recent function call results
- `usage` - token usage statistics
- `state` - current step state
### AxStepContext Mutators
- `setModel(model)` - change the model for the next step
- `setThinkingBudget(budget)` - adjust thinking budget
- `setTemperature(temp)` - adjust temperature
- `setMaxTokens(max)` - adjust max output tokens
- `setOptions(opts)` - set arbitrary forward options
- `addFunctions(fns)` - add functions for the next step
- `removeFunctions(names)` - remove functions by name
- `stop()` - stop multi-step execution
Rules:
- All mutations are pending and applied at the next step boundary.
- `beforeStep` runs before each LLM call; `afterStep` runs after.
- Use `afterFunctionExecution` to react to specific function results.
## Self-Tuning
```typescript
// Simple: enable all self-tuning
const result = await gen.forward(llm, values, { selfTuning: true });
// Granular: pick what to tune
const result = await gen.forward(llm, values, {
selfTuning: {
model: true,
thinkingBudget: true,
functions: [searchWeb, calculate],
},
});
```
Rules:
- `selfTuning: true` enables automatic model and parameter selection.
- Granular config allows tuning specific aspects independently.
- `selfTuning.functions` provides a pool of functions the tuner may add or remove per step.
## Error Handling
```typescript
import { AxGenerateError } from '@ax-llm/ax';
try {
const result = await gen.forward(llm, { input: '...' });
} catch (error) {
if (error instanceof AxGenerateError) {
console.log(error.details.model, error.details.signature);
}
}
```
Rules:
- `AxGenerateError` includes `details` with `model` and `signature` for debugging.
- `AxAIServiceAbortedError` is thrown on cancellation via `stop()` or `abortSignal`.
## Examples
Fetch these for full working code:
- [Streaming](https://raw.githubusercontent.com/ax-llm/ax/refs/heads/main/src/examples/streaming.ts) — streaming with assertions
- [Assertions](https://raw.githubusercontent.com/ax-llm/ax/refs/heads/main/src/examples/asserts.ts) — output validation
- [Streaming Assertions](https://raw.githubusercontent.com/ax-llm/ax/refs/heads/main/src/examples/streaming-asserts.ts) — streaming with assertion checks
- [Structured Output](https://raw.githubusercontent.com/ax-llm/ax/refs/heads/main/src/examples/structured_output.ts) — fluent API with validation
- [Debug Logging](https://raw.githubusercontent.com/ax-llm/ax/refs/heads/main/src/examples/debug-logging.ts) — debug mode and step hooks
- [Stop Function](https://raw.githubusercontent.com/ax-llm/ax/refs/heads/main/src/examples/stop-function.ts) — stop functions
- [Fibonacci](https://raw.githubusercontent.com/ax-llm/ax/refs/heads/main/src/examples/fibonacci.ts) — streaming with thinking
- [Extraction](https://raw.githubusercontent.com/ax-llm/ax/refs/heads/main/src/examples/extract.ts) — information extraction
- [Multi-Sampling](https://raw.githubusercontent.com/ax-llm/ax/refs/heads/main/src/examples/sample-count.ts) — sample count usage
## Do Not Generate
- Do not use `new AxGen(...)` for new code unless explicitly required.
- Do not pass raw API keys or config objects where an `ai(...)` instance is expected.
- Do not use `forward()` for streaming; use `streamingForward()`.
- Do not forget that assertions auto-retry; avoid manual retry loops around assertion logic.
- Do not mutate step hook context expecting immediate effect; mutations are pending until the next step.
- Do not assume multi-step stops after one LLM call; it continues until outputs are filled, a stop function fires, or `maxSteps` is reached.

View File

@@ -1,260 +0,0 @@
---
name: ax-gepa
description: This skill helps an LLM generate correct AxGEPA optimization code using @ax-llm/ax. Use when the user asks about AxGEPA, GEPA, Pareto optimization, multi-objective prompt tuning, reflective prompt evolution, validationExamples, maxMetricCalls, or optimizing a generator, flow, or agent tree.
version: "19.0.33"
---
# AxGEPA Codegen Rules (@ax-llm/ax)
Use this skill to generate direct `AxGEPA` optimization code. Prefer short, modern, copyable patterns over long explanation.
## Use These Defaults
- Use `new AxGEPA({ studentAI, teacherAI, ... })`.
- Prefer `ai()`, `ax()`, and `flow()` for new code.
- Use a strong `teacherAI` and a cheaper `studentAI`.
- Always pass `validationExamples` to `compile()`.
- Always set `maxMetricCalls` to bound optimizer cost.
- Use scalar metrics for one objective and object metrics for Pareto optimization.
- Apply results with `program.applyOptimization(result.optimizedProgram!)`.
- For tree-wide runs, expect `optimizedProgram.instructionMap`.
## Critical Rules
- `AxGEPA.compile()` works for a single generator and for tree-aware roots such as flows or agents with registered instruction-bearing descendants.
- There is no separate flow-only GEPA optimizer. Use `AxGEPA` for flows too.
- The metric may return either `number` or `Record<string, number>`.
- Keep metrics deterministic and cheap by default.
- Avoid extra LLM calls inside the metric unless the user explicitly wants judge-based evaluation.
- If the user needs LLM-as-judge scoring for a non-agent GEPA run, prefer a plain typed `AxGen` evaluator instead of writing a custom judge abstraction.
- `maxMetricCalls` must be large enough to cover the initial validation pass over `validationExamples`.
- GEPA optimizes instructions. If a tree has no instruction-bearing nodes, optimization will fail.
- Use held-out validation examples for selection. Do not reuse the training set as `validationExamples`.
- `result.optimizedProgram` is the easy-to-apply best candidate. `result.paretoFront` is the full trade-off set for multi-objective runs.
## Metric Selection
Choose the evaluation path deliberately:
- Prefer a deterministic metric when correctness can be read directly from `prediction` and `example`.
- Prefer a deterministic metric when cost, latency, recursion depth, or tool count matters.
- Use a plain typed `AxGen` evaluator only when the task is genuinely qualitative and hard to score exactly.
- For `agent.optimize(...)`, prefer the built-in judge path instead of manually wrapping a judge metric.
Rule of thumb:
- `AxGEPA` on `AxGen` or flow: use a metric first, optionally a plain typed `AxGen` evaluator if needed.
- `agent.optimize(...)`: use custom `metric` for crisp scoring, otherwise `judgeAI` plus `judgeOptions`.
## Canonical Scalar Pattern
```typescript
import { ai, ax, AxAIOpenAIModel, AxGEPA } from '@ax-llm/ax';
const student = ai({
name: 'openai',
apiKey: process.env.OPENAI_APIKEY!,
config: { model: AxAIOpenAIModel.GPT4OMini },
});
const teacher = ai({
name: 'openai',
apiKey: process.env.OPENAI_APIKEY!,
config: { model: AxAIOpenAIModel.GPT4O },
});
const classifier = ax(
'emailText:string -> priority:class "high, normal, low", rationale:string'
);
const train = [
{ emailText: 'URGENT: Server down!', priority: 'high' },
{ emailText: 'Weekly newsletter', priority: 'low' },
];
const validation = [
{ emailText: 'Invoice overdue', priority: 'high' },
{ emailText: 'Lunch plans?', priority: 'low' },
];
const metric = ({ prediction, example }: { prediction: any; example: any }) =>
prediction?.priority === example?.priority ? 1 : 0;
const optimizer = new AxGEPA({
studentAI: student,
teacherAI: teacher,
numTrials: 12,
minibatch: true,
minibatchSize: 4,
earlyStoppingTrials: 4,
sampleCount: 1,
});
const result = await optimizer.compile(classifier, train, metric, {
validationExamples: validation,
maxMetricCalls: 120,
});
classifier.applyOptimization(result.optimizedProgram!);
console.log(result.bestScore);
```
## Canonical Pareto Pattern
```typescript
import { ai, flow, AxAIOpenAIModel, AxGEPA } from '@ax-llm/ax';
const student = ai({
name: 'openai',
apiKey: process.env.OPENAI_APIKEY!,
config: { model: AxAIOpenAIModel.GPT4OMini },
});
const teacher = ai({
name: 'openai',
apiKey: process.env.OPENAI_APIKEY!,
config: { model: AxAIOpenAIModel.GPT4O },
});
const wf = flow<{ emailText: string }>()
.n('classifier', 'emailText:string -> priority:class "high, normal, low"')
.n(
'rationale',
'emailText:string, priority:string -> rationale:string "One concise sentence"'
)
.e('classifier', (state) => ({ emailText: state.emailText }))
.e('rationale', (state) => ({
emailText: state.emailText,
priority: state.classifierResult.priority,
}))
.r((state) => ({
priority: state.classifierResult.priority,
rationale: state.rationaleResult.rationale,
}));
const train = [
{ emailText: 'URGENT: Server down!', priority: 'high' },
{ emailText: 'Weekly newsletter', priority: 'low' },
];
const validation = [
{ emailText: 'Invoice overdue', priority: 'high' },
{ emailText: 'Lunch plans?', priority: 'low' },
];
const metric = ({ prediction, example }: { prediction: any; example: any }) => {
const accuracy = prediction?.priority === example?.priority ? 1 : 0;
const rationale = typeof prediction?.rationale === 'string'
? prediction.rationale
: '';
const brevity = rationale.length <= 40 ? 1 : rationale.length <= 80 ? 0.5 : 0.1;
return { accuracy, brevity };
};
const result = await new AxGEPA({
studentAI: student,
teacherAI: teacher,
numTrials: 16,
minibatch: true,
minibatchSize: 6,
earlyStoppingTrials: 5,
sampleCount: 1,
}).compile(wf, train, metric, {
validationExamples: validation,
maxMetricCalls: 240,
});
for (const point of result.paretoFront) {
console.log(point.scores, point.configuration);
}
wf.applyOptimization(result.optimizedProgram!);
console.log(result.optimizedProgram?.instructionMap);
```
## Metric Patterns
```typescript
// Scalar objective
const scalarMetric = ({ prediction, example }) =>
prediction.answer === example.answer ? 1 : 0;
// Multi-objective
const multiMetric = ({ prediction, example }) => ({
accuracy: prediction.answer === example.answer ? 1 : 0,
brevity:
typeof prediction?.reasoning === 'string' &&
prediction.reasoning.length < 120
? 1
: 0.2,
});
```
- Return plain numbers or plain object literals.
- Keep objective names stable across calls.
- Prefer normalized scores such as `0..1` so trade-offs are easy to reason about.
## Result Handling
```typescript
const { optimizedProgram, paretoFront } = result;
program.applyOptimization(optimizedProgram!);
// Save for later
const saved = JSON.stringify(optimizedProgram);
// Load later and re-apply
const loaded = JSON.parse(saved);
program.applyOptimization(loaded);
```
- Single-target runs usually populate both `optimizedProgram.instruction` and `optimizedProgram.instructionMap`.
- Tree-wide runs rely on `instructionMap`, keyed by full program ID.
- Pareto points expose candidate configs under `point.configuration.instructionMap`.
## Useful Options
```typescript
const optimizer = new AxGEPA({
studentAI,
teacherAI,
numTrials: 20,
minibatch: true,
minibatchSize: 5,
minibatchFullEvalSteps: 5,
earlyStoppingTrials: 5,
minImprovementThreshold: 0,
sampleCount: 1,
seed: 42,
verbose: true,
});
```
- `numTrials`: number of reflection/evolution rounds.
- `minibatch`: reduce per-round evaluation cost.
- `minibatchSize`: examples per minibatch.
- `earlyStoppingTrials`: stop after repeated non-improvement.
- `minImprovementThreshold`: reject tiny gains below this threshold.
- `seed`: stabilize sampling during demos and tests.
## Budgeting and Validation
- Always create distinct `train` and `validationExamples` arrays.
- Size `maxMetricCalls` for at least one full validation pass plus several rounds.
- If the user wants a strict budget, say so explicitly and set `maxMetricCalls`.
- For expensive trees, start with `auto: 'light'` or fewer `numTrials`, then scale up.
## Troubleshooting
- Error about `maxMetricCalls` being too small: increase it until the initial validation pass fits.
- Empty or poor Pareto front: verify the metric returns numbers for every example.
- No tree optimization effect: ensure child programs are registered under the root and have instructions to mutate.
- Saved optimization applies only partly: use `program.applyOptimization(...)`, not just `setInstruction(...)`, so `instructionMap` reaches the full tree.
## Good Example Targets
- `/Users/vr/src/ax/src/examples/gepa.ts`
- `/Users/vr/src/ax/src/examples/gepa-flow.ts`
- `/Users/vr/src/ax/src/examples/gepa-train-inference.ts`
- `/Users/vr/src/ax/src/examples/gepa-quality-vs-speed-optimization.ts`

View File

@@ -1,268 +0,0 @@
---
name: ax-learn
description: This skill helps an LLM generate correct AxLearn code using @ax-llm/ax. Use when the user asks about self-improving agents, trace-backed learning, feedback-aware updates, or AxLearn modes.
version: "19.0.33"
---
# AxLearn Codegen Rules (@ax-llm/ax)
Use this skill to generate `AxLearn` code that matches the current API.
## Core Model
- `AxLearn` wraps an `AxGen`.
- `teacher` is for judging, synthesis, and reflection.
- `runtimeAI` is the model being improved.
- `forward()` and `streamingForward()` are inference-time APIs and auto-log traces when tracing is enabled.
- `optimize()` is offline learning.
- `applyUpdate()` is a bounded update API for `continuous` and `playbook` modes.
- `ready()` should be awaited before assuming checkpoints have been restored.
- `improvement` is the score delta from the previous/restored state.
## Required Inputs
- Always provide `name`.
- Always provide `storage`.
- Always provide `teacher`.
- Always provide `runtimeAI` if you call `optimize()` or `applyUpdate()`.
## Modes
- `batch`: offline prompt learning only.
- `continuous`: offline optimization plus bounded feedback-aware `applyUpdate(...)`.
- `playbook`: structured context/playbook learning plus `applyUpdate(...)`.
## Preferred Construction
```typescript
import {
AxLearn,
ax,
ai,
type AxCheckpoint,
type AxStorage,
type AxTrace,
} from '@ax-llm/ax';
const storage: AxStorage = {
save: async (_name, _item) => {
// persist trace/checkpoint
},
load: async (_name, _query) => {
// return traces/checkpoints
return [];
},
};
const teacher = ai({
name: 'openai',
apiKey: process.env.OPENAI_APIKEY!,
});
const runtimeAI = ai({
name: 'openai',
apiKey: process.env.OPENAI_APIKEY!,
});
const gen = ax(`
customerQuery:string "User message" ->
supportReply:string "Agent reply"
`);
const agent = new AxLearn(gen, {
name: 'support-bot-v1',
storage,
teacher,
runtimeAI,
mode: 'continuous',
budget: 12,
examples: [
{
customerQuery: 'Where is my order?',
supportReply: 'Your order is in transit and should arrive in 2 days.',
},
{
customerQuery: 'I need a refund.',
supportReply: 'I can help with that. Please share your order number.',
},
],
generateExamples: false,
});
await agent.ready();
```
## Runtime Pattern
```typescript
const prediction = await agent.forward(runtimeAI, {
customerQuery: 'My package is late.',
});
const traces = await agent.getTraces({ limit: 1 });
if (traces[0]) {
await agent.addFeedback(traces[0].id, {
score: 0,
label: 'needs-empathy',
comment: 'Acknowledge the frustration more directly.',
});
}
```
## Offline Optimization
```typescript
const result = await agent.optimize({
// Optional overrides
budget: 20,
});
console.log(result.mode);
console.log(result.score);
console.log(result.improvement);
console.log(result.checkpointVersion);
```
`result.improvement` is the gain relative to the prior/restored score.
## Continuous Update
Use `applyUpdate(...)` only in `continuous` or `playbook` mode.
- In `continuous` mode, `example` may be input-only.
- `prediction` is the observed runtime output being critiqued.
- If `example` includes expected output fields, that expected-output row stays eligible for scored optimization.
- The observed `prediction` row is feedback/reflection context, not a scored train/validation row by itself.
- Feedback-bearing scored examples should stay in the training pool when non-feedback rows can fill validation.
- In `playbook` mode, `getInstruction()` returns the active composed prompt.
```typescript
const update = await agent.applyUpdate({
example: {
customerQuery: 'My package is late.',
},
prediction,
feedback: {
score: 0,
label: 'needs-empathy',
comment: 'Acknowledge the frustration more directly.',
},
});
```
## Playbook Mode
- Use `mode: 'playbook'` when the learned artifact should be structured guidance, not just an instruction tweak.
- Playbook checkpoints restore through `ready()`.
- `applyUpdate(...)` in playbook mode performs an online structured update.
- `getInstruction()` should be treated as the active composed runtime prompt, even before optimization if the base prompt lives in the signature description.
- `artifact.playbookSummary` should match the persisted checkpoint `state.artifactSummary`.
## How Learning Data Is Used
- `examples` and usable traces become scored optimization rows.
- Feedback stored with `addFeedback(...)` becomes reflection feedback for later optimization.
- In continuous updates, `example + prediction + feedback` is used as an observed feedback event.
- Input-only update examples are useful for reflection, but they are not promoted into scored examples unless expected outputs are present.
## Important Options
```typescript
const agent = new AxLearn(gen, {
name: 'agent-id',
storage,
teacher,
runtimeAI,
mode: 'batch', // 'batch' | 'continuous' | 'playbook'
budget: 20,
metric: async ({ prediction, example }) => {
return prediction.supportReply === example.supportReply ? 1 : 0;
},
criteria: 'accuracy and tone',
judgeOptions: {},
examples: [],
useTraces: true,
generateExamples: false,
synthCount: 20,
validationSplit: 0.2,
continuousOptions: {
feedbackWindowSize: 25,
maxRecentTraces: 100,
updateBudget: 4,
},
playbookOptions: {
maxEpochs: 2,
},
onTrace: (trace) => {
console.log(trace.id);
},
onProgress: (progress) => {
console.log(progress.round, progress.score);
},
});
```
## Result Shape
```typescript
type AxLearnResult = {
mode: 'batch' | 'continuous' | 'playbook';
score: number;
improvement: number;
checkpointVersion: number;
stats: {
trainingExamples: number;
validationExamples: number;
feedbackExamples: number;
durationMs: number;
mode: 'batch' | 'continuous' | 'playbook';
};
state?: {
mode: 'batch' | 'continuous' | 'playbook';
instruction?: string;
baseInstruction?: string;
score?: number;
continuous?: {
feedbackTraceCount?: number;
lastUpdateAt?: string;
};
playbook?: Record<string, unknown>;
artifactSummary?: Record<string, unknown>;
};
artifact?: {
playbook?: Record<string, unknown>;
playbookSummary?: {
feedbackEvents: number;
historyBatches: number;
bulletCount: number;
updatedAt?: string;
};
lastUpdateAt?: string;
feedbackExamples?: number;
};
};
```
## Storage Notes
- `AxStorage.save(name, item)` receives either a trace or checkpoint.
- `AxStorage.load(name, query)` should return arrays of traces or checkpoints.
- Checkpoints may be returned unsorted. `AxLearn` restores the newest one client-side.
## Do This
- Use `runtimeAI` explicitly.
- Await `ready()` before relying on restored state.
- Run `optimize()` off the hot path.
- Use `continuous` mode when you want bounded feedback-aware updates.
- Use `playbook` mode when you want persistent structured guidance.
- Pass the real observed model output as `prediction` in `applyUpdate(...)`.
- Treat `getInstruction()` in playbook mode as the live composed prompt, not just the raw base instruction.
## Avoid This
- Do not assume `teacher` is the optimized runtime model.
- Do not call `applyUpdate()` in `batch` mode.
- Do not claim feedback affects learning unless you are storing it with `addFeedback(...)` or passing it to `applyUpdate(...)`.
- Do not assume checkpoints load synchronously in the constructor.
- Do not treat `prediction` as the gold answer in continuous updates.

View File

@@ -1,192 +0,0 @@
---
name: ax-signature
description: This skill helps an LLM generate correct DSPy signature code using @ax-llm/ax. Use when the user asks about signatures, s(), f(), field types, string syntax, fluent builder API, validation constraints, or type-safe inputs/outputs.
version: "19.0.33"
---
# Ax Signature Reference
## Signature Syntax
```
[description] input1:type, input2:type -> output1:type, output2:type
```
## Field Types
| Type | Syntax | TypeScript | Example |
|------|--------|-----------|---------|
| String | `:string` | `string` | `userName:string` |
| Number | `:number` | `number` | `score:number` |
| Boolean | `:boolean` | `boolean` | `isValid:boolean` |
| JSON | `:json` | `any` | `metadata:json` |
| Date | `:date` | `Date` | `birthDate:date` |
| DateTime | `:datetime` | `Date` | `timestamp:datetime` |
| Image | `:image` | `{mimeType, data}` | `photo:image` (input only) |
| Audio | `:audio` | `{format?, data}` | `recording:audio` (input only) |
| File | `:file` | `{mimeType, data}` | `document:file` (input only) |
| URL | `:url` | `string` | `website:url` |
| Code | `:code` | `string` | `pythonScript:code` |
| Class | `:class "a, b, c"` | `"a" \| "b" \| "c"` | `mood:class "happy, sad"` |
## Arrays, Optional, and Internal Fields
```typescript
'tags:string[] -> processedTags:string[]' // arrays
'query:string, context?:string -> response:string' // optional with ?
'problem:string -> reasoning!:string, solution:string' // internal with !
```
## Three Ways to Create Signatures
### 1. String-Based (Recommended for simple cases)
```typescript
import { ax, s } from '@ax-llm/ax';
const gen = ax('input:string -> output:string');
const sig = s('query:string -> response:string');
```
### 2. Pure Fluent Builder API
```typescript
import { f } from '@ax-llm/ax';
const sig = f()
.input('userMessage', f.string('User input'))
.input('contextData', f.string('Additional context').optional())
.input('tags', f.string('Keywords').array())
.output('responseText', f.string('AI response'))
.output('confidenceScore', f.number('Confidence 0-1'))
.output('debugInfo', f.string('Debug info').internal())
.build();
```
### 3. Hybrid
```typescript
import { s, f } from '@ax-llm/ax';
const sig = s('base:string -> result:string')
.appendInputField('extra', f.json('Metadata').optional())
.appendOutputField('score', f.number('Quality score'));
```
## Fluent API Reference
Type creators:
- `f.string(desc)`, `f.number(desc)`, `f.boolean(desc)`, `f.json(desc)`
- `f.image(desc)`, `f.audio(desc)`, `f.file(desc)`, `f.url(desc)`
- `f.email(desc)`, `f.date(desc)`, `f.datetime(desc)`
- `f.class(['a','b','c'], desc)`, `f.code(desc)`
- `f.object({ field: f.string() }, desc)`
Chainable modifiers (method chaining only, no nesting):
- `.optional()` - make field optional
- `.array()` / `.array('list description')` - make field an array
- `.internal()` - output only, hidden from final output
- `.cache()` - input only, mark for prompt caching
```typescript
// Correct: pure fluent chaining
f.string('description').optional().array()
f.string('context').cache().optional()
f.object({ field: f.string() }, 'item desc').array('list desc')
// Wrong: nested function calls (removed)
f.array(f.string('description')) // REMOVED
f.optional(f.string('description')) // REMOVED
f.internal(f.string('description')) // REMOVED
```
## Validation Constraints
### String Constraints
```typescript
f.string('username').min(3).max(20)
f.string('email').email()
f.string('website').url()
f.string('birthDate').date()
f.string('timestamp').datetime()
f.string('pattern').regex('^[A-Z0-9]')
```
### Number Constraints
```typescript
f.number('age').min(18).max(120)
f.number('score').min(0).max(100)
```
### Complete Validation Example
```typescript
const sig = f()
.input('formData', f.string('Raw form data'))
.output('user', f.object({
username: f.string('Username').min(3).max(20),
email: f.string('Email').email(),
age: f.number('Age').min(18).max(120),
bio: f.string('Bio').max(500).optional(),
website: f.string('Website').url().optional(),
tags: f.string('Tag').min(2).max(30).array()
}, 'User profile'))
.build();
```
## Cached Input Fields
```typescript
const sig = f()
.input('staticContext', f.string('Context').cache())
.input('userQuery', f.string('Dynamic query'))
.output('answer', f.string('Response'))
.build();
```
## Field Naming Rules
Good: `userQuestion`, `customerEmail`, `analysisResult`, `confidenceScore`
Bad: `text`, `data`, `input`, `output`, `a`, `x`, `val` (too generic), `1field` (starts with number)
## Media Type Restrictions
- Media types (image, audio, file) are **top-level input fields only**
- Cannot be nested in objects
- Cannot be output fields
## Common Patterns
```typescript
// Chain of Thought
'problem:string -> reasoning!:string, solution:string'
// Classification
'email:string -> priority:class "urgent, normal, low"'
// Multi-modal
'imageData:image, question?:string -> description:string, objects:string[]'
// Data Extraction
'invoiceText:string -> invoiceNumber:string, totalAmount:number, lineItems:json[]'
// With description
'"Answer TypeScript questions" question:string -> answer:string, confidence:number'
```
## Critical Rules
- Use `f()` fluent builder, NOT nested `f.array(f.string())` -- those are removed.
- Field names must be descriptive (not generic like `text`, `data`, `input`).
- Media types are input-only, top-level only.
- `.internal()` is output-only (for chain-of-thought reasoning).
- `.cache()` is input-only (for prompt caching).
- Validation errors trigger auto-retry with correction feedback.
- `f.email()`, `f.url()`, `f.date()`, `f.datetime()` are shorthand for `f.string().email()` etc.
## Examples
Fetch these for full working code:
- [Fluent Signature](https://raw.githubusercontent.com/ax-llm/ax/refs/heads/main/src/examples/fluent-signature-example.ts) — fluent f() API
- [Structured Output](https://raw.githubusercontent.com/ax-llm/ax/refs/heads/main/src/examples/structured_output.ts) — structured output with validation
- [Debug Schema](https://raw.githubusercontent.com/ax-llm/ax/refs/heads/main/src/examples/debug_schema.ts) — JSON schema validation

View File

@@ -1,292 +0,0 @@
---
name: ax
description: This skill helps with using the @ax-llm/ax TypeScript library for building LLM applications. Use when the user asks about ax(), ai(), f(), s(), agent(), flow(), AxGen, AxAgent, AxFlow, signatures, streaming, or mentions @ax-llm/ax.
version: "19.0.33"
---
# Ax Library (@ax-llm/ax) Quick Reference
Ax is a TypeScript library for building LLM-powered applications with type-safe signatures, streaming support, and multi-provider compatibility.
> **Detailed skills available:** ax-ai (providers), ax-signature (signatures/types), ax-gen (generators), ax-agent (agents/runtime), ax-agent-optimize (agent tuning/eval), ax-flow (workflows), ax-gepa (Pareto optimization), ax-learn (self-improving agents).
## Imports & Factories
```typescript
// Prefer factory functions: ax(), ai(), agent(), flow() — not new AxGen(), new AxAI(), etc.
import { ax, ai, f, s, fn, agent, flow, AxMemory, AxMCPClient, AxLearn } from '@ax-llm/ax';
// AI provider
const llm = ai({ name: 'openai', apiKey: process.env.OPENAI_APIKEY });
// Generator (from string signature)
const gen = ax('question:string -> answer:string');
// Generator (from fluent signature)
const gen = ax(
f()
.input('question', f.string('User question'))
.output('answer', f.string('AI response'))
.build()
);
// Reusable signature
const sig = s('question:string, context:string[] -> answer:string');
// Agent
const myAgent = agent('userInput:string -> response:string', {
name: 'helper',
description: 'A helpful assistant',
});
// Flow
const wf = flow<{ input: string }, { output: string }>()
.node('step1', 'input:string -> output:string')
.execute('step1', (state) => ({ input: state.input }))
.returns((state) => ({ output: state.step1Result.output }));
// Function tool
const tool = fn('search')
.description('Search the web')
.arg('query', f.string('Search query'))
.returns(f.string('Search results'))
.handler(({ query }) => searchWeb(query))
.build();
```
## Running
```typescript
// Forward (blocking)
const result = await gen.forward(llm, { question: 'What is 2+2?' });
// Streaming
for await (const chunk of gen.streamingForward(llm, { question: 'Tell a story' })) {
if (chunk.delta.answer) process.stdout.write(chunk.delta.answer);
}
```
## Forward Options Quick Reference
| Goal | Option | Example |
|------|--------|---------|
| Model override | `model` | `{ model: 'gpt-4o-mini' }` |
| Temperature | `modelConfig.temperature` | `{ modelConfig: { temperature: 0.8 } }` |
| Max tokens | `modelConfig.maxTokens` | `{ modelConfig: { maxTokens: 500 } }` |
| Retry on failure | `maxRetries` | `{ maxRetries: 3 }` |
| Max agent steps | `maxSteps` | `{ maxSteps: 10 }` |
| Fail fast | `fastFail` | `{ fastFail: true }` |
| Thinking budget | `thinkingTokenBudget` | `{ thinkingTokenBudget: 'medium' }` |
| Show thoughts | `showThoughts` | `{ showThoughts: true }` |
| Context caching | `contextCache` | `{ contextCache: { cacheBreakpoint: 'after-examples' } }` |
| Multi-sampling | `sampleCount` | `{ sampleCount: 5 }` |
| Debug logging | `debug` | `{ debug: true }` |
| Abort signal | `abortSignal` | `{ abortSignal: controller.signal }` |
| Memory | `mem` | `{ mem: new AxMemory() }` |
| Stop function | `stopFunction` | `{ stopFunction: 'finalAnswer' }` |
| Function mode | `functionCallMode` | `{ functionCallMode: 'auto' }` |
## Memory and Context
```typescript
import { AxMemory } from '@ax-llm/ax';
const memory = new AxMemory();
// Multi-turn conversation
await gen.forward(llm, { userMessage: 'My name is Alice' }, { mem: memory });
const r = await gen.forward(llm, { userMessage: 'What is my name?' }, { mem: memory });
```
## Few-Shot Examples
```typescript
const classifier = ax('reviewText:string -> sentiment:class "positive, negative, neutral"');
classifier.setExamples([
{ reviewText: 'I love this!', sentiment: 'positive' },
{ reviewText: 'Terrible.', sentiment: 'negative' },
{ reviewText: 'It works.', sentiment: 'neutral' },
]);
```
## Common Patterns
### Classification
```typescript
const classifier = ax(
f()
.input('text', f.string())
.output('category', f.class(['spam', 'ham', 'uncertain']))
.output('confidence', f.number().min(0).max(1))
.build()
);
```
### Extraction
```typescript
const extractor = ax(
f()
.input('text', f.string())
.output('entities', f.object({
people: f.string().array(),
organizations: f.string().array(),
locations: f.string().array()
}))
.build()
);
```
### Multi-modal (Images)
```typescript
const analyzer = ax(
f()
.input('image', f.image('Image to analyze'))
.input('question', f.string('Question').optional())
.output('description', f.string())
.output('objects', f.string().array())
.build()
);
const result = await analyzer.forward(llm, {
image: { mimeType: 'image/jpeg', data: base64Data },
question: 'What objects are in this image?'
});
```
### Chaining Generators
```typescript
const researcher = ax('topic:string -> research:string, keyFacts:string[]');
const writer = ax('research:string, keyFacts:string[] -> article:string');
const research = await researcher.forward(llm, { topic: 'AGI' });
const draft = await writer.forward(llm, { research: research.research, keyFacts: research.keyFacts });
```
## Error Handling
```typescript
import { AxGenerateError, AxAIServiceError, AxAIServiceAbortedError } from '@ax-llm/ax';
try {
const result = await gen.forward(llm, { input: 'test' });
} catch (error) {
if (error instanceof AxGenerateError) {
console.error('Generation failed:', error.details.model, error.details.signature);
} else if (error instanceof AxAIServiceAbortedError) {
console.log('Request was aborted');
} else if (error instanceof AxAIServiceError) {
console.error('AI service error:', error.message);
}
}
```
## Debugging
```typescript
import { axCreateDefaultColorLogger } from '@ax-llm/ax';
const result = await gen.forward(llm, { input: 'test' }, {
debug: true,
logger: axCreateDefaultColorLogger(),
// OpenTelemetry
tracer: openTelemetryTracer,
meter: openTelemetryMeter,
});
```
## MCP Integration
```typescript
import { AxMCPClient } from '@ax-llm/ax';
import { AxMCPStdioTransport } from '@ax-llm/ax-tools';
// Stdio transport (local MCP server)
const transport = new AxMCPStdioTransport({
command: 'npx',
args: ['-y', '@modelcontextprotocol/server-memory'],
});
const mcpClient = new AxMCPClient(transport, { debug: false });
await mcpClient.init();
// Use with agent
const myAgent = agent('userMessage:string -> response:string', {
name: 'assistant',
description: 'An assistant with MCP tools',
functions: [mcpClient],
});
```
### HTTP Transport (Remote MCP)
```typescript
import { AxMCPStreambleHTTPTransport } from '@ax-llm/ax/mcp/transports/httpStreamTransport.js';
const transport = new AxMCPStreambleHTTPTransport('https://remote.mcp.pipedream.net', {
headers: { 'x-pd-project-id': projectId },
authorization: `Bearer ${accessToken}`,
});
```
### MCP Capabilities
| Capability | Prefix | Description |
|---|---|---|
| Tools | *(none)* | Function calls |
| Prompts | `prompt_` | Prompt templates |
| Resources | `resource_` | File/data access |
```typescript
const caps = mcpClient.getCapabilities();
const functions = mcpClient.toFunction();
```
### Function Overrides
```typescript
const mcpClient = new AxMCPClient(transport, {
functionOverrides: [
{ name: 'search_documents', updates: { name: 'findDocs', description: 'Search docs' } }
]
});
```
## Type Reference
```typescript
class AxGen<IN, OUT> {
forward(ai: AxAIService, values: IN, options?: AxProgramForwardOptions): Promise<OUT>;
streamingForward(ai: AxAIService, values: IN, options?: AxProgramStreamingForwardOptions): AsyncGenerator<{ delta: Partial<OUT> }>;
setExamples(examples: Array<Partial<IN & OUT>>): void;
addAssert(fn: (output: OUT) => boolean, message?: string): void;
addFieldProcessor(field: keyof OUT, fn: (value: any) => any): void;
addStreamingFieldProcessor(field: keyof OUT, fn: (chunk: string, ctx: any) => void): void;
stop(): void;
}
class AxAgent<IN, OUT> {
forward(ai: AxAIService, values: IN, options?: AxAgentOptions): Promise<OUT>;
streamingForward(ai: AxAIService, values: IN, options?: AxAgentOptions): AsyncGenerator<{ delta: Partial<OUT> }>;
getFunction(): AxFunction;
}
class AxFlow<IN, OUT> {
node(name: string, signature: string | AxSignature): AxFlow;
execute(name: string, mapper: (state) => any): AxFlow;
returns(mapper: (state) => OUT): AxFlow;
forward(ai: AxAIService, values: IN): Promise<OUT>;
}
```
## Examples
Fetch these for full working code:
- [Chat](https://raw.githubusercontent.com/ax-llm/ax/refs/heads/main/src/examples/chat.ts) — multi-turn conversation
- [Marketing](https://raw.githubusercontent.com/ax-llm/ax/refs/heads/main/src/examples/marketing.ts) — product use case
- [MCP Integration](https://raw.githubusercontent.com/ax-llm/ax/refs/heads/main/src/examples/mcp-client-memory.ts) — MCP integration

View File

@@ -13,6 +13,5 @@ UI-UX
*.md *.md
!README.md !README.md
docker-compose*.yml docker-compose*.yml
render.yaml
.env* .env*
*.log *.log

57
.gitattributes vendored
View File

@@ -1,60 +1,3 @@
# Sprint 12 Task 1 Blocker #1 (2026-04-22):
# Pin benchmark canonical archive + meta to byte-exact treatment. The
# dataset_version hash is computed over the raw bytes at runtime (see
# benchmarks/harness/src/datasets.ts getDatasetVersion). If Git converts
# line endings on checkout the hash drifts per-platform, breaking
# H-AUDIT-2 replication across Windows/macOS/Linux clones. `-text` marks
# these files as binary — no auto-conversion regardless of autocrlf.
benchmarks/data/locomo/locomo-1540.jsonl -text
benchmarks/data/locomo/locomo-1540.meta.json -text
# Sprint 12 Task 2.5 Stage 3 (2026-04-24):
# Same byte-exact treatment for pre-registration manifests. The YAML +
# md SHA-256 recorded in the anchor commit message is computed against
# these files' on-disk bytes; we need checkout bytes to match across
# platforms so any auditor can re-verify the hash after `git clone`.
benchmarks/results/manifest-*.md -text
benchmarks/results/manifest-*.yaml -text
benchmarks/results/stage*-gate-*.md -text
benchmarks/results/stage*-gate-*.jsonl -text
# Sprint 12 Task 2.5 Stage 3 (v5 emission 2026-04-24):
# Pre-registration manifests relocated to benchmarks/preregistration/ per
# PM brief §Step 1. Same byte-exact treatment as v4.
benchmarks/preregistration/manifest-*.md -text
benchmarks/preregistration/manifest-*.yaml -text
# Sprint 12 Task 2.5 Stage 3 §1.3f (2026-04-24):
# Vertex Batch eligibility probe artefacts — byte-exact audit.
benchmarks/probes/**/*.log -text
benchmarks/probes/**/*.jsonl -text
benchmarks/probes/**/*.py -text
benchmarks/probes/**/*.md -text
# Sprint 12 Task 2.5 Stage 3 v6 N=400 final deliverables (2026-04-25):
# Same byte-exact treatment for the agentic-cell evidence JSONL + Phase C
# 5-cell summary / Fisher analysis / memo. The agentic JSONL feeds the
# H1 secondary-endpoint S4 reproducibility chain (see commit message body
# for the file-level SHA-256 capture).
benchmarks/results/stage3-n400-v6-*.md -text
benchmarks/results/agentic-locomo-2026-04-25T*.jsonl -text
benchmarks/results/agentic-locomo-2026-04-25T*.summary.json -text
# v6 self-judge re-evaluation (2026-04-25): apples-to-apples vs Mem0.
# Same byte-exact treatment for the 2000-record results JSONL + comparison
# md + memo. SHA-256 of these files captured in commit message body.
benchmarks/results/v6-self-judge-rebench/*.jsonl -text
benchmarks/results/v6-self-judge-rebench/*.md -text
# Agentic knowledge work pilot 2026-04-26 (FAIL verdict). All artefacts
# byte-exact pinned for audit. SHA-256s captured in commit body.
benchmarks/results/pilot-2026-04-26/*.jsonl -text
benchmarks/results/pilot-2026-04-26/*.json -text
benchmarks/results/pilot-2026-04-26/*.log -text
benchmarks/results/pilot-2026-04-26/prompts-archive/*.md -text
benchmarks/results/pilot-2026-04-26/invalidated/*.jsonl -text
decisions/*.md -text
# Installer arc (steal #5, 2026-07-10): the shell installer + process manager # Installer arc (steal #5, 2026-07-10): the shell installer + process manager
# are consumed by `curl | bash` on Linux/macOS. A CRLF checkout would break the # are consumed by `curl | bash` on Linux/macOS. A CRLF checkout would break the
# shebang and `read`/`printf` parsing, so pin them to LF regardless of core.autocrlf. # shebang and `read`/`printf` parsing, so pin them to LF regardless of core.autocrlf.

241
.github/sync.md vendored
View File

@@ -1,241 +0,0 @@
> **⚠️ DEPRECATED (2026-04-30 monorepo migration) — HISTORICAL/AUDIT REFERENCE ONLY.**
> This manual describes the retired dual-repo bidirectional-sync mechanism. The canonical
> substrate now lives only at **`packages/hive-mind-core/src/{mind,harvest}/`**. Its public
> mirror has a different layout and is updated through a maintainer-curated forward-port that
> removes excluded files and interleaved `install_audit` logic. A raw subtree split is unsafe
> and must never be pushed; [`scripts/oss-subtree-split.sh`](../scripts/oss-subtree-split.sh)
> is inspection-only. The workflows below are inert deprecation anchors because their old
> `packages/core/src/...` trigger paths no longer exist. Everything after this banner is
> historical and must not be treated as current instructions. See AGENTS.md §7.5 and
> [`packages/hive-mind-core/CONTRIBUTING.md`](../packages/hive-mind-core/CONTRIBUTING.md).
---
# Memory Substrate Sync — `waggle-os` ↔ `hive-mind`
This document is the operating manual for the two GitHub Actions workflows
that keep `packages/core/src/mind/` and `packages/core/src/harvest/` in
sync with the OSS release artifact at
[`marolinik/hive-mind`](https://github.com/marolinik/hive-mind).
> **Audience:** anyone modifying files under `packages/core/src/mind/` or
> `packages/core/src/harvest/`. If you only touch `packages/agent/`,
> `packages/server/`, `apps/web/`, etc., none of this applies — those
> paths are explicitly Waggle-only per
> [`hive-mind/EXTRACTION.md`](https://github.com/marolinik/hive-mind/blob/master/EXTRACTION.md).
---
## TL;DR
| What you did | What happens |
|--------------|--------------|
| Modify `packages/core/src/mind/foo.ts` and open a PR | `mind-parity-check` runs hive-mind's tests against your change. Failure blocks merge unless allowlisted. |
| Merge the PR to `main` | `sync-mind-to-hive-mind` opens a PR on `marolinik/hive-mind` with the filtered diff. Manual review there before merge. |
| Want to skip a parity test that's intentionally divergent | Add the basename to `.parity-allowlist` with a comment explaining why. |
| Modify `packages/core/src/mind/vault.ts` (NOT-extracted) | Sync workflow filters this out automatically — nothing leaks to hive-mind. |
---
## Why this exists
Both repos carry their own copy of `packages/core/src/mind/` and
`packages/core/src/harvest/`. The audit at
`PM-Waggle-OS/decisions/2026-04-26-memory-sync-audit.md` documented the
status quo before this workflow shipped:
- No automated sync existed; bug fixes flowed in both directions ad-hoc.
- Two production bug fixes that landed in hive-mind never made it back
to waggle-os until the manual Step 1 backport (commits `89c1004` +
`fed4a20`).
- The OSS release artifact and the production substrate had silently
drifted in 2/19 mind/ files.
The two workflows below close that gap. Their goal is **detection +
human-reviewed propagation**, never automatic merge.
---
## Workflow 1 — `mind-parity-check.yml`
**File:** `.github/workflows/mind-parity-check.yml`
**Trigger:** PR or push to `main` that touches `packages/core/src/mind/`,
`packages/core/src/harvest/`, or `packages/core/tests/mind/`.
**Outcome:** test-pass = ✅ block-clear; test-fail = ❌ merge blocked.
### What it does, step by step
1. Checks out **both** repos — waggle-os in `./waggle-os/`, hive-mind
master in `./hive-mind/`.
2. Runs the **waggle-os baseline** mind/ test suite. This is the
committed Step 2 ports plus all pre-existing waggle-os mind/ tests.
If this fails, the PR is rejected on a regular regression — same as
any other failing test.
3. **Injects** the latest hive-mind tests into the waggle-os checkout
under `<basename>-hive-mind.test.ts` filenames, with import paths
adapted via `sed` from `./*.js` to `../../src/mind/*.js`. Three rules
govern what gets injected:
- **`.parity-allowlist`**: filenames listed here are skipped entirely.
- **Already-committed `-hive-mind` file**: kept as-is. The committed
version (typically a Step 2 port with bespoke header comments
documenting provenance and adaptation rationale) is what the parity
check exercises — overwriting it with the latest hive-mind verbatim
content would silently drop those headers and any waggle-os-side
adaptations.
- **No collision**: copy hive-mind file as `<basename>-hive-mind.test.ts`
into `tests/mind/`, sed the imports.
4. Runs the **combined suite** (waggle-os baseline + injected
hive-mind tests). If hive-mind has added new test cases since the
last Step 2 port, they'll surface here. Failure here means waggle-os
has accidentally diverged from hive-mind's surface contract.
5. Emits an **informational diff** of shared substrate file sizes —
not a gate, just visibility.
### When it fails
| Failure mode | What it means | What to do |
|--------------|---------------|------------|
| Baseline waggle-os mind/ tests fail | Regular regression | Fix your change |
| `<x>-hive-mind.test.ts` (suffixed) fails | hive-mind tests a behavior waggle-os doesn't honor any more | Decide: (a) accept and fix waggle-os to match hive-mind, OR (b) document intentional divergence and add to `.parity-allowlist` |
| Test fails because of import-path adaptation drift | hive-mind reorganized imports | Update the `sed` rules in `mind-parity-check.yml` |
| `db-hive-mind.test.ts` fails (always — see allowlist) | The proprietary-tables-must-be-absent assertion mismatches Waggle's schema | This is in `.parity-allowlist` already; if you removed it, restore it |
### `.parity-allowlist` policy
The file at the repo root lists test basenames whose verbatim hive-mind
copy is intentionally skipped during the parity check.
```
# Comment describing why this entry exists
file-name.test.ts
```
**Adding an entry** requires a comment line directly above with the
divergence rationale and at least one cross-reference (EXTRACTION.md
section, PM-Waggle-OS memo, or related PR).
**Removing an entry** is allowed only when the divergence is resolved
— either hive-mind upstream changed or waggle-os adopted the upstream
behavior. Re-running parity check should pass without the entry first.
The current single entry is `db-hive-mind.test.ts` because hive-mind's
`db.test.ts` asserts proprietary tables (ai_interactions,
execution_traces, evolution_runs, improvement_signals, install_audit)
MUST BE ABSENT — its OSS-scrub guarantee. Waggle-os carries those
tables legitimately. The waggle-os adaptation lives in committed
`db.test.ts` (no suffix) which splits the original assertion into "OSS
shared must exist" (verbatim from hive-mind) + "Waggle-specific must
exist" (inverted). See
`PM-Waggle-OS/decisions/2026-04-26-memory-sync-step2-test-port-results.md`.
---
## Workflow 2 — `sync-mind.yml`
**File:** `.github/workflows/sync-mind.yml`
**Trigger:** push to `main` that touches `packages/core/src/mind/` or
`packages/core/src/harvest/`.
**Outcome:** opens a PR on `marolinik/hive-mind` with the filtered
diff. **Never auto-merges.** The PR sits for manual review on the
hive-mind side.
### Direction note (binding)
This workflow implements ONLY the **waggle-os → hive-mind** direction.
The empirically primary direction (**hive-mind → waggle-os**) requires
a workflow living **inside the hive-mind repo**, which is out of scope
for the waggle-os Step 3 PR. It will be added via a sibling PR to
hive-mind once Step 3 here is ratified by PM. The current Memory Sync
Repair audit shows hive-mind is the more active substrate repo (ahead
in 2/5 audit dimensions, +14 organic test files), so the
hive-mind-side workflow carries the heavier production burden.
### Filter list — NOT-extracted paths
The workflow excludes these paths from the patch — leaking them into
hive-mind would put Waggle-specific code into the Apache-2.0 release
artifact:
```
packages/core/src/mind/vault.ts
packages/core/src/mind/evolution-runs.ts
packages/core/src/mind/execution-traces.ts
packages/core/src/mind/improvement-signals.ts
packages/core/src/compliance/**
```
This list mirrors the "NOT Extracted" section of
[`hive-mind/EXTRACTION.md`](https://github.com/marolinik/hive-mind/blob/master/EXTRACTION.md).
**If you add a new NOT-extracted file, update the workflow's
`excluded_paths` array AND EXTRACTION.md in the same PR** — otherwise
the file will leak on the next mind/ push.
### Setup — `HIVE_MIND_SYNC_TOKEN`
This workflow needs a fine-grained PAT scoped to `marolinik/hive-mind`
with `pull_request: write` + `contents: write` permissions. Marko
configures it via:
```bash
gh secret set HIVE_MIND_SYNC_TOKEN --repo marolinik/waggle-os
gh variable set MIND_SYNC_ENABLED --body 'true' --repo marolinik/waggle-os
```
The `MIND_SYNC_ENABLED` repository variable is the kill switch — set
it to `false` to disable the workflow entirely without removing the
secret. The workflow's `if:` condition checks this before running.
If `HIVE_MIND_SYNC_TOKEN` is unset, the workflow fails fast with a
clear error rather than silently skipping.
### When the patch doesn't apply
`git apply --3way` falls back to a 3-way merge when the index
mismatch is small. If even that fails, the workflow exits with an
error pointing to the source SHA. Manual reconciliation:
1. Check out hive-mind master locally.
2. `git apply` the patch from the workflow run's
`sync-patch-<sha>` artifact.
3. Resolve conflicts; commit on a branch named
`auto-sync/waggle-os-<short-sha>`.
4. Open the PR by hand following the same body template.
This is rare in practice because hive-mind's mind/ files are mostly
verbatim extractions of waggle-os's — any genuine conflict means
hive-mind has its own change at the same lines, which is exactly the
case the human review is supposed to catch.
---
## Bidirectional bug fix protocol
When you find a bug whose fix should go into BOTH repos:
1. Fix it in waggle-os first (production-impacted).
2. Merge to waggle-os main → `sync-mind.yml` auto-opens a hive-mind PR.
3. Review and merge the hive-mind PR.
4. Confirm the next `mind-parity-check` on waggle-os main is green —
that closes the loop.
When the bug originates in hive-mind (e.g. an upstream contributor
reports it):
1. Wait for the hive-mind PR (or open it yourself).
2. After it merges, the eventual hive-mind→waggle-os auto-PR (when
that workflow ships) will pick it up.
3. Until then, manual cherry-pick to waggle-os, identical to Step 1
of the original sync repair (see Step 1 results memo at
`PM-Waggle-OS/decisions/2026-04-26-memory-sync-step1-results.md`
for the canonical pattern: commit message references hive-mind SHA
verbatim).
---
## Cross-references
- Audit + 3-step plan: `PM-Waggle-OS/decisions/2026-04-26-memory-sync-audit.md`
- Step 1 results (forward-port + bidirectional audit): `PM-Waggle-OS/decisions/2026-04-26-memory-sync-step1-results.md`
- Step 2 results (test port): `PM-Waggle-OS/decisions/2026-04-26-memory-sync-step2-test-port-results.md`
- Step 3 results (this workflow): `PM-Waggle-OS/decisions/2026-04-26-memory-sync-step3-cicd-results.md`
- EXTRACTION.md: `D:\Projects\hive-mind\EXTRACTION.md` (or [GitHub link](https://github.com/marolinik/hive-mind/blob/master/EXTRACTION.md))

View File

@@ -1,183 +0,0 @@
name: mind-parity-check
# DEPRECATED 2026-04-30 — CC Sesija B monorepo migration §2.6 Task B22.
#
# This workflow ran the (then-external) hive-mind repo's mind/+harvest/ tests
# against waggle-os's substrate to verify behavioral parity while the same code
# lived in both repos. After CC Sesija B migration, the substrate lives ONLY in
# waggle-os/packages/hive-mind-core/. The OSS mirror at
# github.com/marolinik/hive-mind is a separately curated layout produced from
# that canonical source. It is intentionally not byte-identical: imports and
# layout differ, and excluded files plus interleaved `install_audit` logic are
# removed during the maintainer-reviewed forward-port.
#
# This workflow is preserved as the deprecation anchor. It will NOT fire on
# push because trigger paths (packages/core/src/mind/**) no longer exist as
# tracked content. See sync-mind.yml's deprecation note for the full migration
# context. Do not reactivate this injection workflow as a substitute for the
# current drift review, curated export, and public-mirror test gates.
# Memory Sync Repair Step 3.1. Verifies that hive-mind's mind/ + harvest/
# tests pass against waggle-os's substrate. The check runs the waggle-os
# committed Step 2 ports (which include adapted versions like db.test.ts)
# AS BASELINE, then ALSO injects the latest hive-mind tests into the
# waggle-os checkout under `<basename>-hive-mind.test.ts` filenames so
# any NEW hive-mind cases since the Step 2 port get exercised.
#
# Triggers ONLY when shared substrate paths change. Failure blocks merge
# unless the failing test is allowlisted in `.parity-allowlist` at the
# repo root with a documented reason.
#
# See `.github/sync.md` for full design rationale and allowlist policy.
on:
push:
branches: [main]
paths:
- 'packages/core/src/mind/**'
- 'packages/core/src/harvest/**'
- 'packages/core/tests/mind/**'
pull_request:
branches: [main]
paths:
- 'packages/core/src/mind/**'
- 'packages/core/src/harvest/**'
- 'packages/core/tests/mind/**'
concurrency:
group: mind-parity-${{ github.ref }}
cancel-in-progress: true
jobs:
parity-check:
# Permanently inert deprecation anchor. The old test-injection model is not a
# safe substitute for the curated forward-port and drift-review workflow.
if: ${{ false }}
name: hive-mind ↔ waggle-os mind substrate parity
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- name: Checkout waggle-os
uses: actions/checkout@v4
with:
path: waggle-os
- name: Checkout hive-mind master
uses: actions/checkout@v4
with:
repository: marolinik/hive-mind
ref: master
path: hive-mind
- name: Setup Node 20
uses: actions/setup-node@v4
with:
node-version: 20
- name: Cache npm dependencies
uses: actions/cache@v4
with:
path: ~/.npm
key: ${{ runner.os }}-npm-${{ hashFiles('waggle-os/**/package-lock.json') }}
restore-keys: |
${{ runner.os }}-npm-
- name: Install waggle-os dependencies
working-directory: waggle-os
run: npm install
- name: Run waggle-os baseline mind/ tests (committed Step 2 ports)
working-directory: waggle-os
run: npx vitest run --reporter=default packages/core/tests/mind/
- name: Inject latest hive-mind tests as -hive-mind suffix files
working-directory: waggle-os
run: |
set -euo pipefail
allowlist_file=".parity-allowlist"
allowlist_basenames=()
if [ -f "$allowlist_file" ]; then
while IFS= read -r line; do
# Strip comments + leading/trailing whitespace; skip blanks
clean="${line%%#*}"
clean="$(echo "$clean" | tr -d '[:space:]')"
[ -z "$clean" ] && continue
allowlist_basenames+=("$clean")
done < "$allowlist_file"
echo "Allowlist entries: ${allowlist_basenames[@]:-<none>}"
else
echo "No .parity-allowlist file present — no skips."
fi
# Helper: is a basename in the allowlist?
is_allowlisted() {
local name="$1"
for a in "${allowlist_basenames[@]:-}"; do
[ "$a" = "$name" ] && return 0
done
return 1
}
target_dir="packages/core/tests/mind"
source_dir="../hive-mind/packages/core/src/mind"
injected=0
skipped=0
already_committed=0
for src in "$source_dir"/*.test.ts; do
[ -e "$src" ] || continue
base="$(basename "$src" .test.ts)"
target_name="${base}-hive-mind.test.ts"
target_path="$target_dir/$target_name"
if is_allowlisted "$target_name"; then
echo " SKIP (allowlist): $target_name"
skipped=$((skipped + 1))
continue
fi
if [ -f "$target_path" ]; then
# File is committed (Step 2 port). Preserve its bespoke
# header comments + any waggle-os adaptations. The
# committed version IS what the parity check should
# exercise — it's exactly what landed in main.
already_committed=$((already_committed + 1))
echo " KEEP (committed): $target_name"
continue
fi
cp "$src" "$target_path"
# Adapt import paths from hive-mind's adjacent style (`./x.js`) to
# waggle-os's separate-tests-folder style (`../../src/mind/x.js`).
sed -i "s|from \"\\./|from \"../../src/mind/|g" "$target_path"
sed -i "s|from '\\./|from '../../src/mind/|g" "$target_path"
injected=$((injected + 1))
echo " INJECT: $target_name"
done
echo ""
echo "Injected: $injected new hive-mind test file(s) under -hive-mind suffix"
echo "Kept: $already_committed already-committed Step 2 port file(s)"
echo "Skipped: $skipped allowlisted file(s)"
- name: Run combined waggle-os + injected hive-mind suite
working-directory: waggle-os
run: npx vitest run --reporter=default packages/core/tests/mind/
- name: Informational diff — shared substrate file sizes
working-directory: waggle-os
if: always()
run: |
echo "## Substrate file size comparison (informational)"
for f in db.ts schema.ts frames.ts search.ts knowledge.ts identity.ts awareness.ts sessions.ts scoring.ts reconcile.ts ontology.ts concept-tracker.ts entity-normalizer.ts embedding-provider.ts inprocess-embedder.ts; do
wf="packages/core/src/mind/$f"
hf="../hive-mind/packages/core/src/mind/$f"
if [ -f "$wf" ] && [ -f "$hf" ]; then
wsize=$(wc -c < "$wf")
hsize=$(wc -c < "$hf")
diff_pct=$(awk -v w="$wsize" -v h="$hsize" 'BEGIN { if (h == 0) print "n/a"; else printf "%.1f%%", ((w - h) / h) * 100 }')
echo " $f: waggle-os=$wsize hive-mind=$hsize delta=$diff_pct"
fi
done

View File

@@ -1,233 +0,0 @@
name: sync-mind-to-hive-mind
# DEPRECATED 2026-04-30 — CC Sesija B monorepo migration §2.6 Task B22.
#
# This workflow was the bidirectional-sync mechanism between waggle-os and the
# public `marolinik/hive-mind` repo while substrate code lived in BOTH
# places (packages/core/src/mind/ + packages/core/src/harvest/ in waggle-os,
# duplicated in hive-mind/packages/core/src/{mind,harvest}/).
#
# After CC Sesija B migration (commits ff5b4aa..b59d188 on
# feature/hive-mind-monorepo-migration), the substrate lives ONLY in
# waggle-os/packages/hive-mind-core/. The public mirror now has a curated layout
# and is updated through a maintainer-reviewed forward-port that removes excluded
# files and interleaved `install_audit` logic. Raw subtree branches are unsafe
# publish sources; see `packages/hive-mind-core/CONTRIBUTING.md`.
#
# This workflow is preserved for AUDIT TRAIL purposes (the historical
# trigger paths and concurrency settings are referenced in EXTRACTION.md and
# the .github/sync.md operating manual). It will NOT fire on push because the
# trigger paths (packages/core/src/mind/** + packages/core/src/harvest/**) no
# longer exist as tracked content — they were git-mv'd to packages/hive-mind-core/
# on commit 3b556c0.
#
# DO NOT delete this file as part of cleanup — leave it as the deprecation
# anchor. Do not reactivate it by changing trigger paths: its filtered-patch
# model cannot safely remove interleaved proprietary schema logic. Build any
# future automation from the curated-forward-port contract instead.
# Memory Sync Repair Step 3.2 — waggle-os → hive-mind direction.
#
# Triggered when waggle-os main receives a push that touches shared
# substrate paths (mind/ or harvest/), this workflow extracts the
# filtered diff (excluding NOT-extracted files per EXTRACTION.md), opens
# a PR against marolinik/hive-mind master with the patch applied, and
# tags the PR with the source SHA.
#
# This is the SECONDARY direction empirically — Steps 1+2 found
# hive-mind is the more active substrate repo (ahead in 2/5 audit
# dimensions, +14 organic test files). The PRIMARY direction
# (hive-mind push → auto-PR ka waggle-os) is intentionally NOT
# implemented in this file because it requires a workflow living in
# the hive-mind repo, which is out of CC-2 scope. It will be added
# via a sibling PR to hive-mind once Step 3 here is ratified.
#
# See `.github/sync.md` for full design rationale, EXTRACTION.md
# filter list, and how to extend bidirectional sync.
on:
push:
branches: [main]
paths:
- 'packages/core/src/mind/**'
- 'packages/core/src/harvest/**'
concurrency:
group: sync-mind-${{ github.ref }}
# Don't cancel — every main push to mind/ or harvest/ deserves its own
# sync attempt (the resulting hive-mind PR is per-commit traceable).
cancel-in-progress: false
jobs:
open-hive-mind-pr:
# Permanently inert deprecation anchor. This filtered-patch publisher cannot
# enforce the current interleaved proprietary-content boundary.
name: Open auto-sync PR to marolinik/hive-mind
runs-on: ubuntu-latest
timeout-minutes: 10
# `HIVE_MIND_SYNC_TOKEN` is a fine-grained PAT scoped to
# marolinik/hive-mind with `pull_request: write` + `contents: write`.
# Configured by Marko via `gh secret set HIVE_MIND_SYNC_TOKEN`.
# Without the secret, the job fails fast with a documented error
# rather than silently skipping.
if: ${{ false }}
steps:
- name: Checkout waggle-os (full history for the diff)
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Verify HIVE_MIND_SYNC_TOKEN is configured
run: |
if [ -z "${{ secrets.HIVE_MIND_SYNC_TOKEN }}" ]; then
echo "::error::HIVE_MIND_SYNC_TOKEN secret is not configured."
echo "::error::Run: gh secret set HIVE_MIND_SYNC_TOKEN --repo marolinik/waggle-os"
echo "::error::See .github/sync.md for full setup instructions."
exit 1
fi
- name: Compute filtered diff (exclude NOT-extracted paths)
id: diff
run: |
set -euo pipefail
# Files that MUST be excluded from sync per EXTRACTION.md
# "NOT Extracted" section. Sync attempts that include these paths
# would leak Waggle-specific code (vault, compliance, evolution,
# tier system) into the hive-mind Apache-2.0 release artifact.
excluded_paths=(
'packages/core/src/mind/vault.ts'
'packages/core/src/mind/evolution-runs.ts'
'packages/core/src/mind/execution-traces.ts'
'packages/core/src/mind/improvement-signals.ts'
'packages/core/src/compliance'
)
# Build the rev range. `${{ github.event.before }}` is the parent
# commit of this push; `${{ github.sha }}` is the new HEAD. For a
# branch's first push or a force-push, `before` may be all-zeros;
# in that case fall back to the previous merge-base via reflog.
before_sha='${{ github.event.before }}'
after_sha='${{ github.sha }}'
if [ "$before_sha" = "0000000000000000000000000000000000000000" ]; then
echo "::warning::push has no `before` SHA — falling back to HEAD~1"
before_sha="$(git rev-parse HEAD~1)"
fi
# All shared-substrate files in this push, before exclusion.
changed=$(git diff --name-only "$before_sha" "$after_sha" -- \
'packages/core/src/mind/**' 'packages/core/src/harvest/**')
# Apply EXTRACTION.md "NOT extracted" exclusions.
filtered_files=()
while IFS= read -r f; do
[ -z "$f" ] && continue
skip=false
for excl in "${excluded_paths[@]}"; do
case "$f" in
"$excl"|"$excl"/*)
skip=true
break
;;
esac
done
$skip || filtered_files+=("$f")
done <<< "$changed"
if [ ${#filtered_files[@]} -eq 0 ]; then
echo "No syncable changes after EXTRACTION.md filtering."
echo "skip=true" >> "$GITHUB_OUTPUT"
exit 0
fi
echo "Files to sync:"
printf ' %s\n' "${filtered_files[@]}"
# Generate a patch limited to the filtered files. This patch will
# apply to hive-mind because the directory layout matches:
# waggle-os `packages/core/src/{mind,harvest}/...` ↔
# hive-mind `packages/core/src/{mind,harvest}/...`.
mkdir -p .sync-output
git diff "$before_sha" "$after_sha" -- "${filtered_files[@]}" > .sync-output/patch.diff
# Capture the original commit subjects for the PR description.
git log --format='- %h %s' "$before_sha".."$after_sha" -- "${filtered_files[@]}" > .sync-output/commits.txt
echo "skip=false" >> "$GITHUB_OUTPUT"
echo "after_sha=$after_sha" >> "$GITHUB_OUTPUT"
echo "after_short=$(git rev-parse --short "$after_sha")" >> "$GITHUB_OUTPUT"
echo "first_subject=$(git log -1 --format=%s "$after_sha")" >> "$GITHUB_OUTPUT"
- name: Checkout hive-mind for patch application
if: steps.diff.outputs.skip != 'true'
uses: actions/checkout@v4
with:
repository: marolinik/hive-mind
ref: master
token: ${{ secrets.HIVE_MIND_SYNC_TOKEN }}
path: hive-mind
fetch-depth: 0
- name: Apply patch to hive-mind branch + push
if: steps.diff.outputs.skip != 'true'
working-directory: hive-mind
env:
GH_TOKEN: ${{ secrets.HIVE_MIND_SYNC_TOKEN }}
run: |
set -euo pipefail
branch_name="auto-sync/waggle-os-${{ steps.diff.outputs.after_short }}"
git config user.name 'waggle-os-sync-bot'
git config user.email 'sync-bot@waggle-os.ai'
git checkout -b "$branch_name"
# Apply the filtered patch. The waggle-os layout is identical
# under packages/core/src/{mind,harvest}/ so the patch applies
# directly without `--directory` rewriting.
if ! git apply --3way ../.sync-output/patch.diff; then
echo "::error::Patch did not apply cleanly. Manual reconciliation required."
echo "::error::Source SHA: ${{ steps.diff.outputs.after_sha }}"
exit 1
fi
git add -A
git commit -m "chore(sync): auto-sync from waggle-os@${{ steps.diff.outputs.after_short }}
Source: marolinik/waggle-os main @ ${{ steps.diff.outputs.after_sha }}
Subject: ${{ steps.diff.outputs.first_subject }}
See PR description for the full list of waggle-os commits in this batch."
git push origin "$branch_name"
- name: Open PR on hive-mind
if: steps.diff.outputs.skip != 'true'
working-directory: hive-mind
env:
GH_TOKEN: ${{ secrets.HIVE_MIND_SYNC_TOKEN }}
run: |
set -euo pipefail
branch_name="auto-sync/waggle-os-${{ steps.diff.outputs.after_short }}"
commit_list="$(cat ../.sync-output/commits.txt)"
gh pr create \
--repo marolinik/hive-mind \
--base master \
--head "$branch_name" \
--title "auto-sync from waggle-os@${{ steps.diff.outputs.after_short }}: ${{ steps.diff.outputs.first_subject }}" \
--body "$(printf 'Automated cross-repo sync — Memory Sync Repair Step 3.2 (waggle-os → hive-mind direction).\n\n## Source\n- Repo: marolinik/waggle-os\n- Branch: main\n- HEAD: %s\n- Workflow run: %s/%s/actions/runs/%s\n\n## Filtering\nThis patch was filtered to exclude paths listed under "NOT Extracted" in EXTRACTION.md:\n- packages/core/src/mind/vault.ts\n- packages/core/src/mind/evolution-runs.ts\n- packages/core/src/mind/execution-traces.ts\n- packages/core/src/mind/improvement-signals.ts\n- packages/core/src/compliance\n\n## Originating commits in this batch\n\n%s\n\n## Review checklist\n- [ ] Patch applied cleanly to hive-mind master without 3-way conflicts\n- [ ] No NOT-extracted paths sneaked through (sanity-check the diff against EXTRACTION.md)\n- [ ] Test deltas (if any) make sense for the OSS surface; no Waggle-specific test fixtures\n- [ ] mind-parity-check on waggle-os side is GREEN before merging this PR (otherwise the sync would re-introduce a regression)' \
"${{ steps.diff.outputs.after_sha }}" \
"${{ github.server_url }}" \
"${{ github.repository }}" \
"${{ github.run_id }}" \
"$commit_list")"
- name: Upload patch as artifact (debug aid)
if: always() && steps.diff.outputs.skip != 'true'
uses: actions/upload-artifact@v4
with:
name: sync-patch-${{ steps.diff.outputs.after_short }}
path: .sync-output/
retention-days: 30

169
.gitignore vendored
View File

@@ -38,142 +38,6 @@ marketplace.db
*.pfx *.pfx
app/src-tauri/.thumbprint.txt app/src-tauri/.thumbprint.txt
# Preflight gate per-query raw JSONs carry retrieved frame previews that
# originate from Marko's personal AI exports. The report .md stays commit-
# eligible (adapter counts, timing, verbatim questions + model answers
# only); per-query JSONs and scratch dirs do not.
preflight-results/stage-0-query-*.json
preflight-results/stage-0-tmp/
preflight-results/stage-*-attempt-*.md
preflight-results/stage-*-exclusions.md
# Benchmark harness — data + run outputs stay local
benchmarks/data/*
!benchmarks/data/.gitkeep
# Committed sample-lock artifacts (stratified picks from public LoCoMo). Small,
# diff-reviewable, required for reproducibility. Raw LoCoMo dumps stay local.
!benchmarks/data/preflight-locomo-50.json
!benchmarks/data/failure-mode-calibration-10.jsonl
# Sprint 12 Task 1 Blocker #1 — canonical LoCoMo eval archive (1531 instances,
# derived from snap-research/locomo10.json via scripts/build-locomo-canonical.ts).
# Committed for hash determinism across clones + CI + H-AUDIT-2 replication.
!benchmarks/data/locomo/
!benchmarks/data/locomo/locomo-1540.jsonl
!benchmarks/data/locomo/locomo-1540.meta.json
# LME V1 canonical outputs — meta.json is tracked; large JSONL + raw JSON are not.
# Re-ignore the dir's contents (the `!dir/` negation re-includes everything; the
# single-level `data/*` above does not cover nested files) then whitelist meta.
!benchmarks/data/longmemeval/
benchmarks/data/longmemeval/*
!benchmarks/data/longmemeval/longmemeval.meta.json
# BEAM canonical meta — same pattern
!benchmarks/data/beam/
benchmarks/data/beam/*
!benchmarks/data/beam/beam-128K.meta.json
!benchmarks/data/beam/beam-1M.meta.json
# Memori replication corpus + outputs stay local (large; not source).
benchmarks/memori-replication/
benchmarks/results/*
!benchmarks/results/.gitkeep
# Sprint 12 Task 2.5 Stage 3 — per-stage manifest-adjacent audit artefacts:
# pre-registration anchors, clarification memos, RCA memos, probe logs.
# These are tamper-evident, NOT run output, so they MUST be tracked.
# Broadened 2026-04-24 for Stage 3 re-kick (Option A): clarification +
# RCA memos + Gate P+ probe log all need the same treatment.
!benchmarks/results/manifest-*.md
!benchmarks/results/manifest-*.yaml
!benchmarks/results/stage*-gate-*.md
!benchmarks/results/stage*-gate-*.jsonl
# Stage 3 v6 N=400 final deliverables (2026-04-25): cell-by-cell summary,
# Fisher H1 analysis, ≤300-word memo, and the agentic-cell evidence JSONL.
# Tracked under PM-RATIFY-V6-N400-COMPLETE; tamper-evident audit chain.
!benchmarks/results/stage3-n400-v6-*.md
!benchmarks/results/agentic-locomo-2026-04-25T*.jsonl
!benchmarks/results/agentic-locomo-2026-04-25T*.summary.json
# v6 self-judge re-evaluation (2026-04-25): apples-to-apples vs Mem0
# methodology. 2000 records, side-by-side comparison, ≤300-word memo.
!benchmarks/results/v6-self-judge-rebench/
!benchmarks/results/v6-self-judge-rebench/*.jsonl
!benchmarks/results/v6-self-judge-rebench/*.md
# Agentic knowledge work pilot 2026-04-26 (FAIL verdict, see
# decisions/2026-04-26-pilot-verdict-FAIL.md). 12 cell JSONLs + summary +
# run log + prompts archive + invalidated originals (audit-preserved).
!benchmarks/results/pilot-2026-04-26/
!benchmarks/results/pilot-2026-04-26/*.jsonl
!benchmarks/results/pilot-2026-04-26/*.json
!benchmarks/results/pilot-2026-04-26/*.log
!benchmarks/results/pilot-2026-04-26/prompts-archive/
!benchmarks/results/pilot-2026-04-26/prompts-archive/*.md
!benchmarks/results/pilot-2026-04-26/invalidated/
!benchmarks/results/pilot-2026-04-26/invalidated/*.jsonl
# GEPA Faza 1 corpus + run logs + spot-audit reports + NULL/Gen1/held-out
# artefacts (per launch decision §G + manifest v7 §gepa). All instances +
# generation logs + checkpoint reports tracked for audit chain.
# Exception applies to BOTH the line-47 base pattern AND the line-98
# `**/benchmarks/results/*` nested-mis-CWD catch (mirrors pilot-2026-04-26).
!benchmarks/results/gepa-faza1/
!benchmarks/results/gepa-faza1/**/*.jsonl
!benchmarks/results/gepa-faza1/**/*.json
!benchmarks/results/gepa-faza1/**/*.md
!benchmarks/results/gepa-faza1/**/*.log
!benchmarks/results/gepa-faza1/**/*.yaml
!**/benchmarks/results/gepa-faza1
!**/benchmarks/results/gepa-faza1/
!**/benchmarks/results/gepa-faza1/**
# §1.3f Vertex Batch eligibility probe (2026-04-24): standalone probe
# artefacts outside §11 frozen paths. Logs + JSONL + script + memo all
# tracked; override the *.log line-3 wildcard for this specific folder.
!benchmarks/probes/**/*.log
!benchmarks/probes/**/*.jsonl
!benchmarks/probes/**/*.py
!benchmarks/probes/**/*.md
# Also catch nested mis-CWD writes (e.g. ran from benchmarks/harness/).
# Use a trailing `*` (not `/`) so the `!` exceptions above still apply.
**/benchmarks/results/*
# Canonical LoCoMo SOTA evidence — intentionally committed (see in-dir README). Do NOT remove:
# this exact swallow hid the 87.66 result from the repo (docs/analysis/locomo-sota-evidence-drift-2026-06-30.md).
!**/benchmarks/results/locomo-sota-2026-06/
!**/benchmarks/results/locomo-sota-2026-06/**
**/benchmarks/data/*
!**/benchmarks/data/preflight-locomo-50.json
!**/benchmarks/data/failure-mode-calibration-10.jsonl
!**/benchmarks/data/locomo/
!**/benchmarks/data/locomo/locomo-1540.jsonl
!**/benchmarks/data/locomo/locomo-1540.meta.json
!**/benchmarks/data/longmemeval/
**/benchmarks/data/longmemeval/*
!**/benchmarks/data/longmemeval/longmemeval.meta.json
!**/benchmarks/data/beam/
**/benchmarks/data/beam/*
!**/benchmarks/data/beam/beam-128K.meta.json
!**/benchmarks/data/beam/beam-1M.meta.json
!**/benchmarks/data/.gitkeep
!**/benchmarks/results/.gitkeep
!**/benchmarks/results/manifest-*.md
!**/benchmarks/results/manifest-*.yaml
!**/benchmarks/results/stage*-gate-*.md
!**/benchmarks/results/stage*-gate-*.jsonl
!**/benchmarks/results/stage3-n400-v6-*.md
!**/benchmarks/results/agentic-locomo-2026-04-25T*.jsonl
!**/benchmarks/results/agentic-locomo-2026-04-25T*.summary.json
!**/benchmarks/results/v6-self-judge-rebench/
!**/benchmarks/results/v6-self-judge-rebench/*.jsonl
!**/benchmarks/results/v6-self-judge-rebench/*.md
!**/benchmarks/results/pilot-2026-04-26/
!**/benchmarks/results/pilot-2026-04-26/*.jsonl
!**/benchmarks/results/pilot-2026-04-26/*.json
!**/benchmarks/results/pilot-2026-04-26/*.log
!**/benchmarks/results/pilot-2026-04-26/prompts-archive/
!**/benchmarks/results/pilot-2026-04-26/prompts-archive/*.md
!**/benchmarks/results/pilot-2026-04-26/invalidated/
!**/benchmarks/results/pilot-2026-04-26/invalidated/*.jsonl
# Sprint 11 A2 (ratification §Q5 Tier 2): `benchmarks/archive/` holds
# gzipped JSONL from launch-claim-supporting runs (H-42a/b full-run).
# Retention: 12 months minimum from commit per ratification §Q5. Files here
# ARE committed — do NOT gitignore the folder. Spring 11 only adds the
# folder + README; actual archival runs land in a later sprint.
!benchmarks/archive/
!benchmarks/archive/**
test-regression-*.log test-regression-*.log
# SQLite WAL/SHM companions — always match the *.db they accompany # SQLite WAL/SHM companions — always match the *.db they accompany
marketplace.db-shm marketplace.db-shm
@@ -256,21 +120,6 @@ apps/www/lighthouse-report.json
# Vision-harness capture output (regenerated per run) # Vision-harness capture output (regenerated per run)
tests/vision/artifacts/ tests/vision/artifacts/
# UX Refactor v2.1 handoff package — D10 doc-authority ruling (ratified
# 2026-06-10; register: docs/ux-refactor/deltas/open-questions.md, section
# "UX Refactor v2.1 — Ratification of Decision Register D1–D15").
# Text sources stay TRACKED: PRD .md, Implementation Handoff .md,
# _blueprint_extracted.txt, NAMING-ERRATUM.md, and
# Waggle_OS_Handoff_Assets/{README.md,ASSET_MANIFEST.json}.
# Binary exports stay LOCAL-ONLY (~85 MB: pdf/docx/pptx + generated mockup
# png/jpg — mockups are visual direction, not authority; see the package's
# NAMING-ERRATUM.md for the authority chain).
docs/Waggle_OS_UX_Refactor_Master_Handoff_Package/*.pdf
docs/Waggle_OS_UX_Refactor_Master_Handoff_Package/*.docx
docs/Waggle_OS_UX_Refactor_Master_Handoff_Package/*.pptx
docs/Waggle_OS_UX_Refactor_Master_Handoff_Package/Waggle_OS_Handoff_Assets/*.png
docs/Waggle_OS_UX_Refactor_Master_Handoff_Package/Waggle_OS_Handoff_Assets/*.jpg
# Internal / pre-OSS artifacts — kept on disk, excluded from the public repo. # Internal / pre-OSS artifacts — kept on disk, excluded from the public repo.
# These are currently tracked; run `git rm --cached <file>` once to stop tracking # These are currently tracked; run `git rm --cached <file>` once to stop tracking
# them (the entries below keep them from being re-added). Root-anchored so nested # them (the entries below keep them from being re-added). Root-anchored so nested
@@ -284,26 +133,8 @@ docs/Waggle_OS_UX_Refactor_Master_Handoff_Package/Waggle_OS_Handoff_Assets/*.jpg
# divergent dependency graph from a stray `bun install`. # divergent dependency graph from a stray `bun install`.
bun.lock bun.lock
# /understand tool trash — regenerable deleted-graph JSON, not for the public repo
.understand-anything/.trash*
.gstack/ .gstack/
# Local plaintext API keys — never commit # Local plaintext API keys — never commit
AI API KEYS.txt AI API KEYS.txt
# Sesija C external dependencies — vendored per-clone, not committed
# (ARE platform installed under external/meta-agents-research-environments;
# pinned SHA tracked in benchmarks/gaia2/README.md)
external/
# Sesija C per-run output dirs — large JSONL traces, not committed
# (Phase 2 smoke evidence summarized in benchmarks/gaia2/smoke-evidence.md;
# Phase 4 dry run JSONLs stay local + summarized in dry-run-results-memo.md)
benchmarks/gaia2/runs/*
!benchmarks/gaia2/runs/.gitkeep
# Sesija C per-clone Gaia2 task dumps — produced by dump-tasks.py from HF,
# large JSONL (2.44–2.67M chars/scenario), not committed. Phase 3b-B
# regenerates locally; Phase 4 Docker run uses ARE-native HF loader directly.
benchmarks/gaia2/data/*
!benchmarks/gaia2/data/.gitkeep

View File

@@ -1,23 +0,0 @@
## Make App Windows Draggable
### Problem
Windows currently use fixed `defaultPosition` via framer-motion `animate`, so they can't be repositioned by the user.
### Approach
Use framer-motion's built-in `drag` prop on the window container, constrained by a `dragConstraints` ref (the desktop). The title bar acts as the drag handle via `dragListener={false}` on the main div + `dragControls` triggered from the title bar.
### Changes
**`src/components/os/AppWindow.tsx`**
- Add `useState` for `position` (initialized from `defaultPosition`)
- Use `useDragControls` from framer-motion
- Add `drag` prop to `motion.div` with `dragControls`, `dragMomentum={false}`, `dragListener={false}`
- Add `dragConstraints` prop (parent bounds or screen-based object)
- Title bar gets `onPointerDown` to start drag via `dragControls.start()`
- When maximized, disable drag
- Track position via `onDragEnd` to persist position across re-renders
- Remove the `x`/`y`/`top`/`left` from the `animate` prop (use `style` for initial positioning instead, so drag offset works correctly)
- Set `cursor-grab` / `cursor-grabbing` on title bar

View File

@@ -1,29 +0,0 @@
# Memory Sync Repair — `mind-parity-check` allowlist
#
# Lines list test filenames (basename, e.g. `db-hive-mind.test.ts`) whose
# verbatim hive-mind copy is INTENTIONALLY skipped during the parity check
# because waggle-os has an adapted version under the natural name (e.g.
# `db.test.ts`) that handles a legitimate API divergence per
# `D:\Projects\hive-mind\EXTRACTION.md`.
#
# Format: one filename per line. `#` starts a comment. Blank lines ignored.
#
# Adding an entry: include a comment line directly above the entry that
# explains *why* the divergence exists (link to EXTRACTION.md section, PR,
# or PM-Waggle-OS decisions/ memo).
#
# Removing an entry: only when the divergence has been resolved (either
# hive-mind upstream changed or waggle-os adopted the upstream behavior).
# Re-running parity check should pass without the allowlist entry before
# removal lands.
# `db-hive-mind.test.ts` — hive-mind asserts proprietary tables
# (ai_interactions, execution_traces, evolution_runs, improvement_signals,
# install_audit) MUST BE ABSENT, which is its OSS-scrub guarantee. Waggle-os
# legitimately carries those tables per EXTRACTION.md "NOT extracted"
# section. The waggle-os adaptation lives in committed `db.test.ts` (no
# suffix) which splits the original into "OSS shared must exist" (verbatim)
# + "Waggle-specific must exist" (inverted). See:
# - PM-Waggle-OS/decisions/2026-04-26-memory-sync-step2-test-port-results.md §2
# - hive-mind EXTRACTION.md "NOT Extracted" section
db-hive-mind.test.ts

View File

@@ -1,191 +0,0 @@
# .understandignore — patterns for files/dirs to exclude from analysis
# Syntax: same as .gitignore (globs, # comments, ! negation, trailing / for dirs)
# Lines below are suggestions — uncomment to activate.
# Use ! prefix to force-include something excluded by defaults.
#
# Built-in defaults (always excluded unless negated):
# node_modules/, .git/, dist/, build/, obj/, *.lock, *.min.js, etc.
#
# --- From .gitignore (uncomment to exclude) ---
# .mcp.json
# logs
# npm-debug.log*
# yarn-debug.log*
# yarn-error.log*
# pnpm-debug.log*
# lerna-debug.log*
# dist-ssr
# *.local
# .env
# .env.local
# .env*.local
# .waggle/
# *.mind
# marketplace.db
# *.pfx
# app/src-tauri/.thumbprint.txt
# preflight-results/stage-0-query-*.json
# preflight-results/stage-0-tmp/
# preflight-results/stage-*-attempt-*.md
# preflight-results/stage-*-exclusions.md
# benchmarks/data/*
# !benchmarks/data/.gitkeep
# !benchmarks/data/preflight-locomo-50.json
# !benchmarks/data/failure-mode-calibration-10.jsonl
# !benchmarks/data/locomo/
# !benchmarks/data/locomo/locomo-1540.jsonl
# !benchmarks/data/locomo/locomo-1540.meta.json
# !benchmarks/data/longmemeval/
# benchmarks/data/longmemeval/*
# !benchmarks/data/longmemeval/longmemeval.meta.json
# !benchmarks/data/beam/
# benchmarks/data/beam/*
# !benchmarks/data/beam/beam-128K.meta.json
# !benchmarks/data/beam/beam-1M.meta.json
# benchmarks/memori-replication/
# benchmarks/results/*
# !benchmarks/results/.gitkeep
# !benchmarks/results/manifest-*.md
# !benchmarks/results/manifest-*.yaml
# !benchmarks/results/stage*-gate-*.md
# !benchmarks/results/stage*-gate-*.jsonl
# !benchmarks/results/stage3-n400-v6-*.md
# !benchmarks/results/agentic-locomo-2026-04-25T*.jsonl
# !benchmarks/results/agentic-locomo-2026-04-25T*.summary.json
# !benchmarks/results/v6-self-judge-rebench/
# !benchmarks/results/v6-self-judge-rebench/*.jsonl
# !benchmarks/results/v6-self-judge-rebench/*.md
# !benchmarks/results/pilot-2026-04-26/
# !benchmarks/results/pilot-2026-04-26/*.jsonl
# !benchmarks/results/pilot-2026-04-26/*.json
# !benchmarks/results/pilot-2026-04-26/*.log
# !benchmarks/results/pilot-2026-04-26/prompts-archive/
# !benchmarks/results/pilot-2026-04-26/prompts-archive/*.md
# !benchmarks/results/pilot-2026-04-26/invalidated/
# !benchmarks/results/pilot-2026-04-26/invalidated/*.jsonl
# !benchmarks/results/gepa-faza1/
# !benchmarks/results/gepa-faza1/**/*.jsonl
# !benchmarks/results/gepa-faza1/**/*.json
# !benchmarks/results/gepa-faza1/**/*.md
# !benchmarks/results/gepa-faza1/**/*.log
# !benchmarks/results/gepa-faza1/**/*.yaml
# !**/benchmarks/results/gepa-faza1
# !**/benchmarks/results/gepa-faza1/
# !**/benchmarks/results/gepa-faza1/**
# !benchmarks/probes/**/*.log
# !benchmarks/probes/**/*.jsonl
# !benchmarks/probes/**/*.py
# !benchmarks/probes/**/*.md
# **/benchmarks/results/*
# **/benchmarks/data/*
# !**/benchmarks/data/preflight-locomo-50.json
# !**/benchmarks/data/failure-mode-calibration-10.jsonl
# !**/benchmarks/data/locomo/
# !**/benchmarks/data/locomo/locomo-1540.jsonl
# !**/benchmarks/data/locomo/locomo-1540.meta.json
# !**/benchmarks/data/longmemeval/
# **/benchmarks/data/longmemeval/*
# !**/benchmarks/data/longmemeval/longmemeval.meta.json
# !**/benchmarks/data/beam/
# **/benchmarks/data/beam/*
# !**/benchmarks/data/beam/beam-128K.meta.json
# !**/benchmarks/data/beam/beam-1M.meta.json
# !**/benchmarks/data/.gitkeep
# !**/benchmarks/results/.gitkeep
# !**/benchmarks/results/manifest-*.md
# !**/benchmarks/results/manifest-*.yaml
# !**/benchmarks/results/stage*-gate-*.md
# !**/benchmarks/results/stage*-gate-*.jsonl
# !**/benchmarks/results/stage3-n400-v6-*.md
# !**/benchmarks/results/agentic-locomo-2026-04-25T*.jsonl
# !**/benchmarks/results/agentic-locomo-2026-04-25T*.summary.json
# !**/benchmarks/results/v6-self-judge-rebench/
# !**/benchmarks/results/v6-self-judge-rebench/*.jsonl
# !**/benchmarks/results/v6-self-judge-rebench/*.md
# !**/benchmarks/results/pilot-2026-04-26/
# !**/benchmarks/results/pilot-2026-04-26/*.jsonl
# !**/benchmarks/results/pilot-2026-04-26/*.json
# !**/benchmarks/results/pilot-2026-04-26/*.log
# !**/benchmarks/results/pilot-2026-04-26/prompts-archive/
# !**/benchmarks/results/pilot-2026-04-26/prompts-archive/*.md
# !**/benchmarks/results/pilot-2026-04-26/invalidated/
# !**/benchmarks/results/pilot-2026-04-26/invalidated/*.jsonl
# !benchmarks/archive/
# !benchmarks/archive/**
# test-regression-*.log
# marketplace.db-shm
# marketplace.db-wal
# *.db-shm
# *.db-wal
# .workspace/
# .scratch/
# .mind/
# .planning/
# external/
# *.tsbuildinfo
# scripts/.cache/
# app/src-tauri/resources/node
# app/src-tauri/resources/node.exe
# app/src-tauri/resources/native/
# app/src-tauri/resources/service.js
# app/src-tauri/resources/service.js.map
# .vscode/*
# !.vscode/extensions.json
# .DS_Store
# *.suo
# *.ntvs*
# *.njsproj
# *.sln
# *.sw?
# .claude/
# .playwright-mcp/
# .superpowers/
# ~$*
# playwright-report/
# test-results/
# tests/screenshots/
# s*.png
# screen*.png
# tmp_bench_results.json
# tmp_bench_results-v5.json
# tmp_hermes-skill-reuse.json
# tmp_hermes-pilot*.json
# tmp_hermes-pilot*.log
# tmp_lpv*.log
# tmp_lpv*.json
# apps/www/public/brand/_backup-*/
# .vercel/
# apps/www/lighthouse-report.json
# tests/vision/artifacts/
# docs/Waggle_OS_UX_Refactor_Master_Handoff_Package/*.pdf
# docs/Waggle_OS_UX_Refactor_Master_Handoff_Package/*.docx
# docs/Waggle_OS_UX_Refactor_Master_Handoff_Package/*.pptx
# docs/Waggle_OS_UX_Refactor_Master_Handoff_Package/Waggle_OS_Handoff_Assets/*.png
# docs/Waggle_OS_UX_Refactor_Master_Handoff_Package/Waggle_OS_Handoff_Assets/*.jpg
# --- Detected directories (uncomment to exclude) ---
# docs/
# scripts/
# tests/
# --- Test file patterns (uncomment to exclude) ---
# JS / TS
# *.test.*
# *.spec.*
# *.snap
# C# / .NET
# **/*Tests.cs
# **/*Test.cs
# **/*Fixture.cs
# **/*.Tests.csproj
# Java / Kotlin
# **/src/test/**
# **/*Test.java
# **/*IT.java
# **/*Spec.kt
# Go
# **/*_test.go

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

View File

@@ -1,6 +0,0 @@
{
"lastAnalyzedAt": "2026-06-26T09:19:56.431Z",
"gitCommitHash": "18aebe1f4bd035cfe2646173db7d2fa5326d2ec9",
"version": "1.0.0",
"analyzedFiles": 2896
}

View File

@@ -66,7 +66,7 @@ npm workspaces with the root `package-lock.json`. Node `^20.19.0 || >=22.12.0`.
| `packages/hive-mind-*`, `memory-mcp`, `wiki-compiler` | Excluded once P1.5 moves all memory access to the contract. | | `packages/hive-mind-*`, `memory-mcp`, `wiki-compiler` | Excluded once P1.5 moves all memory access to the contract. |
| `packages/admin-web`, `apps/www` | Excluded. | | `packages/admin-web`, `apps/www` | Excluded. |
| `apps/browser-ext` | Not decided. Do not extend it. | | `apps/browser-ext` | Not decided. Do not extend it. |
| `sidecar/`, `benchmarks/`, `judging/`, most of `docs/` | Internal history of the commercial product, removed in P2.2. Community-edition docs live in `docs/oss/`. | | `docs/` | `docs/oss/` holds the community-edition records. The remaining guides, reference, architecture, getting-started and contributing pages predate this edition and are rewritten in P2.4. Internal history (`sidecar/`, `benchmarks/`, `judging/`, internal docs) was removed in P2.2. |
### Where things live ### Where things live

226
README.md
View File

@@ -1,201 +1,89 @@
# Waggle OS # Waggle OS Community Edition
Workspace-native AI agent platform with persistent memory, model-agnostic orchestration, and skill-extensible capabilities. The current desktop release scope is Windows-first: a Tauri 2.0 app with a Vite-bundled web UI and bundled Node.js sidecar; macOS packaging and certification remain roadmap work. Waggle is a local-first AI workspace: an agent with persistent memory, personas, skills,
connectors and scheduled automations. It runs as a Node.js sidecar with a React web UI, and
can be wrapped in a Tauri desktop shell. You bring your own model: a local model through
Ollama or a provider API key.
## Current Release Scope > **Status: work in progress.** This branch is being converted from the commercial Waggle
> OS codebase into an open-source community edition. Billing, KVARK, compliance reporting
> and self-learning are already removed; memory is moving to a local mem0 store, and
> installers for Windows, macOS and Linux are not ready yet. The rules for this conversion
> are in [`AGENTS.md`](AGENTS.md) and the decision records in
> [`docs/oss/adr/`](docs/oss/adr/README.md).
The active launch gate is **Windows Solo**. Its in-scope external-agent release cohort is **Claude Code, Codex, and Hermes**. Each tool uses the user's own installation and authentication; Waggle does not redistribute provider credentials or bypass provider terms. ## Repository layout
- **Cursor and OpenClaw are roadmap integrations.** They remain registered for detection and future development, but the production launcher, hooks, Fleet/task path, and direct run API do not offer them.
- Claude Desktop, Codex Desktop, and Hermes Desktop may appear as detected convenience launch surfaces; they are not separate memory-hook or agent-acceptance targets in this release gate.
- **ChatGPT/OpenAI is a model and memory-import surface**, not a separate local coding-agent launcher.
- The Windows Solo launch contract requires an exact-revision Windows installer qualification receipt to prove its bundled Node sidecar, no-Python OpenAI-compatible proxy, Waggle-managed local runtime/model, default in-process embedding path, and freedom from developer Node, Docker, Python, an external LiteLLM service, or a separately installed Ollama. A separate revision-bound router receipt must prove the smart-router primary, compact-tool-context, budget, and fallback paths. Router, persona, and authentication receipts may cover a later candidate only through an independently reviewed bounded no-impact attestation proving that no covered runtime surface changed; otherwise they must be rerun. A user-installed Ollama remains optional.
- Docker/LiteLLM deployment files remain optional server and team deployment choices; they are not desktop prerequisites.
Release status, revision-bound receipts, and any bounded carry-forward attestations are governed only by the current [launch recommendation](docs/production-readiness/09-LAUNCH_RECOMMENDATION.md). If it does not say **GO**, do not describe Waggle as production-ready or reuse historical scores or receipts as current release evidence.
### Current Windows Solo internal RC evidence — 2026-08-27
The current runtime candidate is private `main` commit
`23ad3fa5f99bddce648b84750a41365299aeb0da`. Its internal-pilot NSIS installer
(102,923,936 bytes; SHA-256
`7BFA9F9B13633A51CD3336B42E3EF904B7F7A568C6DEE4F6CED967CBD4F40A59`)
passed **64/64** clean-profile checks. The receipt proves bundled sidecar and offline
npm, FREE/Solo first boot, in-process embeddings, the Waggle-managed local runtime and
`qwen2.5:0.5b`, local-model chat, proxy restart, repair, relaunch, data preservation,
Exit/cleanup, and uninstall. Docker, Python, developer Node.js, external LiteLLM, and a
separately installed Ollama were not prerequisites.
- Private PR #66 merged as `23ad3fa5` after every blocking check passed: primary CI,
Playwright smoke and full E2E, Windows and both macOS Tauri verification targets, and
Hive Mind install/smoke on Windows, Ubuntu, and macOS. The tested PR head and merged
source have the same tree.
- Exact-current full and production dependency audits contain **0 Critical and 0 High**
findings. Lower-severity maintenance remains documented.
- The historical persona collection at `4c712ff6` still records all ten personas x3 at
or above 95/100 after documented independent adjudication, but it is not relabeled as
an exact-current seal. PR #66 changed memory behavior, so public release qualification
requires a fresh receipt or an explicit bounded semantic-impact attestation.
- Smart-router primary, compact-tool-context, durable-budget and fallback evidence, plus
the Claude Code, Codex, and Hermes official-user-auth canaries, remain historical
scoped evidence. No later change touched provider credential handling; the auth
harness read or copied no credential files.
- The curated Hive Mind mirror is hardened through private-to-maintainer PR #53, merged
on public `master` as `3410327800db3ea23f875d547a0c7f4d08826b7e`; Windows, macOS,
Ubuntu, and Ubuntu first-run smoke were green. The immutable drift baseline still
reports reviewed release blockers, so the next OSS package release remains a separate
maintainer-curated operation and is not implied by this Windows Solo RC.
Detailed local receipt paths, hashes, limitations, and integration gates are recorded in
the current [launch recommendation](docs/production-readiness/09-LAUNCH_RECOMMENDATION.md).
The internal signer (`CN=Egzakta Internal Pilot`) and DigiCert timestamp prove the pilot
pipeline but are not publicly trusted Authenticode. Public release is **not yet approved**:
a protected hosted build with a publicly trusted signer, an exact-candidate sealed
managed Deep Security report, current persona qualification, and either fresh or
explicitly attested smart-router and official-auth qualification remain mandatory.
## Architecture
``` ```
waggle-os/ waggle-os/
├── apps/ ├── apps/web/ # Web UI (React 19, Vite, Tailwind 4)
│ ├── web/ # Main web app UI (React 19 + Vite + Tailwind 4 + base-ui/react) ├── app/ # Tauri 2 desktop shell (Rust); loads the apps/web build
│ ├── www/ # Marketing site (Next.js) ├── packages/ # npm workspaces: agent, server, core, shared, marketplace,
│ └── browser-ext/ # Browser extension (unpacked; not an npm workspace) │ # memory-contract, memory-mem0, cli, launcher, sdk, weaver, ...
├── packages/ # 28 workspace packages (see "Packages" below) └── docs/ # Guides, reference, and docs/oss for the community edition
├── app/ # Tauri 2.0 desktop shell (Rust) — loads the apps/web build
├── sidecar/ # Node.js sidecar bundled into the Tauri desktop binary
└── docs/ # Architecture, contributing, threat model, and guides
``` ```
### Packages `AGENTS.md` lists the status of every package in this edition.
The monorepo has **28 packages** under `packages/`. They split into two groups. ## Run from source
**Product packages (15, MIT):** Prerequisites: Node.js `^20.19.0` or `>=22.12.0`, and npm.
| Package | Purpose |
|---|---|
| `agent` | Agent loop, orchestrator, tools, personas, workflows, and the evolution subsystem |
| `core` | Config, the encrypted vault, cron store, file store, telemetry, and compliance/audit |
| `server` | Fastify sidecar — local (solo) and team routes, SSE streaming, Stripe, KVARK client |
| `shared` | Shared types, Zod schemas, the tier system, and the MCP catalog |
| `marketplace` | Package catalog with the `SecurityGate` installer |
| `optimizer` | GEPA prompt optimization |
| `weaver` | Memory consolidation daemon |
| `waggle-dance` | Multi-agent coordination protocol |
| `worker` | Background job processor (BullMQ, team mode) |
| `sdk` | Plugin / skill SDK |
| `cli` | Command-line REPL |
| `launcher` | AI-tool launcher / dock backend |
| `admin-web` | Admin dashboard for team deployments |
| `wiki-compiler` | Knowledge / wiki compiler |
| `memory-mcp` | MCP server exposing the memory substrate to external agents |
**Memory substrate — `hive-mind-*` (13, Apache-2.0):** the persistent-memory core, mirrored to the public OSS repo [`marolinik/hive-mind`](https://github.com/marolinik/hive-mind).
| Package | Purpose |
|---|---|
| `hive-mind-core` | The memory substrate: `FrameStore`, `HybridSearch`, `KnowledgeGraph`, `IdentityLayer`, `AwarenessLayer`, plus Harvest ingestion (`src/mind` + `src/harvest`) |
| `hive-mind-cli` | CLI for the substrate |
| `hive-mind-mcp-server` | MCP server for the substrate |
| `hive-mind-shim-core` | Signal-emitter shim library |
| `hive-mind-wiki-compiler` | Wiki compiler (OSS) |
| `hive-mind-hooks-core` | Shared hook library |
| `hive-mind-hooks-*` | Per-tool capture hooks: `claude-code`, `claude-desktop`, `codex`, `codex-desktop`, `cursor`, `hermes`, `openclaw` |
> The memory substrate is developed **here** and mirrored out — never the reverse.
> See the "Memory Substrate Sync" section of [`CLAUDE.md`](./CLAUDE.md) before touching `packages/hive-mind-core`.
## Quick Start
### Windows Solo desktop
Use only the signed Windows installer and SHA-256 identified by a **GO** [launch recommendation](docs/production-readiness/09-LAUNCH_RECOMMENDATION.md). If that recommendation is not GO, no packaged desktop artifact is release-approved; use the source-development instructions below.
### Self-host from the private repository (Linux / macOS)
```bash ```bash
gh repo clone marolinik/waggle-os
cd waggle-os
bash install.sh
```
This path is for maintainers with authenticated access to the private repository.
Do not publish an anonymous raw-file installer until the source/licensing decision is explicit.
Best for a VPS or homelab — this runs a headless Waggle server (no desktop shell):
- **Checks prerequisites, clones, builds, and starts** the Node.js sidecar, then prints the URL (`http://127.0.0.1:3333`). A 5-question wizard — install dir, port, data dir, build web UI, start now — is all Enter-defaulted; pass `--yes` to accept every default non-interactively.
- **Boots with zero API keys** in echo mode so the UI works immediately. Add a provider key later under **Settings → API Keys**, where it is stored in the encrypted vault — keys are never passed on the command line.
- **No sudo, ever.** A missing prerequisite prints the exact per-OS install command and exits; the installer never installs system packages for you.
Manage the running server with the installed wrapper: `scripts/waggle-server.sh status | logs | stop | start`. Re-running the one-liner against an existing install prints an upgrade hint instead of reinstalling.
### Run from source (development)
```bash
# Prerequisites: Node.js ^20.19.0 or >=22.12.0, npm
npm install npm install
npm run build:packages # workspace packages export dist/; the sidecar needs it
# (Optional) copy the env template. Provider API keys are normally set in-app npm run build # builds the web UI into dist/
# (Settings → API Keys), which stores them in the encrypted vault — so you do npm run dev:server # the sidecar on http://localhost:3333, serving the UI
# NOT need to put keys in .env for a basic local run.
cp .env.example .env
# Terminal 1 — backend sidecar (http://localhost:3333)
npm run dev:server
# Terminal 2 — web app (http://localhost:8080)
npm run dev:web
# Open http://localhost:8080
``` ```
The source tree follows the root `package.json` Node engine above. The packaged Open http://localhost:3333. Rebuild the UI and restart the sidecar to see UI changes:
Windows Solo desktop carries its own pinned Node.js 22.23.2 runtime, so an the Vite dev server (`npm run dev:web`) cannot authenticate against the sidecar yet
installed user does not need a separate Node.js installation. (UF-003 in [`docs/oss/upstream-fixes.md`](docs/oss/upstream-fixes.md)).
`npm run dev:server` runs the Fastify sidecar via `tsx` (equivalent to Provider keys are entered in the app under **Settings → API Keys** and stored in the
`cd packages/server && npx tsx src/local/start.ts`). `npm run dev:web` runs the encrypted vault, so a basic local run needs no `.env` file. `.env.example` lists the
Vite dev server for `apps/web`. optional variables.
**Embeddings.** By default `EMBEDDING_PROVIDER=auto` downloads a small in-process **Embeddings.** By default `EMBEDDING_PROVIDER=auto` downloads a small in-process model
model (~23 MB, cached under `~/.waggle/models/`) and works fully offline. For (about 23 MB, cached under `~/.waggle/models/`) and works offline. To use Ollama instead,
better recall — and to reproduce any hive-mind benchmark — install install [Ollama](https://ollama.com), run `ollama pull nomic-embed-text`, and set
[Ollama](https://ollama.com), run `ollama pull nomic-embed-text`, and set
`EMBEDDING_PROVIDER=ollama`. `EMBEDDING_PROVIDER=ollama`.
> **Windows:** if the sidecar fails to start with an esbuild platform error, see | Variable | Description |
> [Troubleshooting](docs/CONTRIBUTING.md#troubleshooting) in the contributing guide. |---|---|
| `ANTHROPIC_API_KEY` | Claude API key; can be set in the app instead. |
| `OPENAI_API_KEY` | Enables OpenAI models and optional OpenAI embeddings. |
| `EMBEDDING_PROVIDER` | `auto` (default), `inprocess`, `ollama`, `voyage`, `openai` or `mock`. |
| `LITELLM_BASE_URL` | Optional external LiteLLM-compatible proxy. |
## Environment Variables ## Checks
Provider keys are stored in the encrypted vault (set via **Settings → API Keys**) ```bash
and hydrated into the process at boot, so most of these are optional for a local npm run build:packages # build all workspace packages
run. See [`.env.example`](./.env.example) for the full contract. npm run typecheck:web # type-check the web UI
npm run lint # lint the packages
npm run check:deps # every import is a declared dependency
npm test # unit tests
```
| Variable | Required | Description | Known failures and the numbers to compare against are in
|----------|----------|-------------| [`docs/oss/baseline.md`](docs/oss/baseline.md).
| `ANTHROPIC_API_KEY` | Recommended | Claude API key. Optional in `.env` — can be set in-app instead (vault). |
| `OPENAI_API_KEY` | No | Enables OpenAI models and optional OpenAI embeddings. |
| `EMBEDDING_PROVIDER` | No | `auto` (default) · `inprocess` · `ollama` · `voyage` · `openai` · `mock`. `auto` tries in-process → Ollama → API → mock. |
| `LITELLM_BASE_URL` | No | Optional external LiteLLM-compatible proxy URL. The Windows Solo desktop uses its bundled no-Python proxy unless explicitly configured otherwise. |
| `DATABASE_URL` | Team only | PostgreSQL connection string. |
| `REDIS_URL` | Team only | Redis for the background job queue. |
## Documentation ## Documentation
- [Getting Started](docs/GETTING-STARTED.md) — first-run walkthrough - [Getting Started](docs/GETTING-STARTED.md)
- [Architecture](docs/ARCHITECTURE.md) — package structure, data flow, extension points - [Architecture](docs/ARCHITECTURE.md)
- [Contributing](docs/CONTRIBUTING.md) — setup, tests, PR process, troubleshooting - [Contributing](docs/CONTRIBUTING.md)
- [Threat Model](THREAT_MODEL.md) — trust boundary and security controls - [Threat Model](THREAT_MODEL.md)
- [Security Policy](SECURITY.md) — how to report a vulnerability - [Security Policy](SECURITY.md)
- [Code of Conduct](CODE_OF_CONDUCT.md) - [Code of Conduct](CODE_OF_CONDUCT.md)
The guides predate the community edition and still describe some removed features; they
are rewritten before the first release.
## License ## License
This repository is licensed under the [MIT License](./LICENSE), **except** the This repository is licensed under the [MIT License](./LICENSE), **except** the
`hive-mind-*` packages under `packages/`, which are licensed under **Apache-2.0**. `hive-mind-*` packages under `packages/`, which are licensed under **Apache-2.0**.
Each `hive-mind-*` package carries its own `LICENSE` file, which governs that Each `hive-mind-*` package carries its own `LICENSE` file, which governs that
package. See [Packages](#packages) for the split. package.

View File

@@ -114,7 +114,7 @@ content or a user is resolved to concrete IP(s); the fetch is **refused if any r
address is loopback / private (RFC1918 + CGNAT) / link-local / unique-local / multicast / address is loopback / private (RFC1918 + CGNAT) / link-local / unique-local / multicast /
reserved / unspecified**. This closes the **cloud instance-metadata** exfiltration path — reserved / unspecified**. This closes the **cloud instance-metadata** exfiltration path —
`169.254.169.254` (link-local) reaching IAM credentials — which matters because the `169.254.169.254` (link-local) reaching IAM credentials — which matters because the
cloud/TEAMS sidecar binds `0.0.0.0` (`docker-compose.production.yml`, `render.yaml`). cloud/TEAMS sidecar binds `0.0.0.0` (`docker-compose.production.yml`).
Coverage: Coverage:
- **Scheme allowlist:** only `http:`/`https:` (blocks `file:`/`gopher:`/`ftp:` redirect tricks). - **Scheme allowlist:** only `http:`/`https:` (blocks `file:`/`gopher:`/`ftp:` redirect tricks).
- **Obfuscated literals** (octal `0177.0.0.1`, decimal `2130706433`, hex `0x7f000001`) are - **Obfuscated literals** (octal `0177.0.0.1`, decimal `2130706433`, hex `0x7f000001`) are

Binary file not shown.

Before

Width:  |  Height:  |  Size: 1.3 MiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 1.3 MiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 1.1 MiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 1.1 MiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 1.3 MiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 288 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 1.3 MiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 116 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 1.0 MiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 842 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 468 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 466 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 478 KiB

View File

@@ -1,45 +0,0 @@
import { readFileSync } from 'node:fs';
import { describe, expect, it } from 'vitest';
const playbook = readFileSync(
new URL('../../docs/code-signing-pilot-and-launch.md', import.meta.url),
'utf8',
);
const pilotScript = readFileSync(
new URL('./sign-windows-pilot.ps1', import.meta.url),
'utf8',
);
const macPilotScript = readFileSync(
new URL('./sign-macos-adhoc.sh', import.meta.url),
'utf8',
);
describe('internal pilot signing guidance', () => {
it('routes Windows builds through the explicit pilot-signing override', () => {
expect(playbook).toContain('npm run tauri:build:win:pilot-signed');
expect(playbook).not.toMatch(/^npm run tauri:build:win$/m);
expect(pilotScript).toContain(
"Write-Host ' 1. npm run tauri:build:win:pilot-signed'",
);
expect(pilotScript).not.toMatch(
/Write-Host '[ ]{2}1\. npm run tauri:build:win'\s*$/m,
);
expect(pilotScript).not.toMatch(/Write-Host '[ ]+1\. cd app/);
});
it('does not claim an ordinary macOS build loads the signing override', () => {
expect(playbook).toContain('macOS is deferred');
expect(playbook).toContain('npm run tauri:sign:pilot:mac:adhoc');
expect(playbook).not.toContain(
'so every `npm run tauri:build:mac` produces an ad-hoc-signed `.app` automatically',
);
expect(playbook).not.toContain(
'ships the macOS ad-hoc identity in the build-override config by default',
);
expect(macPilotScript).toContain(
'Treat the input as unsigned until this script signs and verifies it.',
);
expect(macPilotScript).not.toContain('already passes');
});
});

View File

@@ -1,34 +0,0 @@
# benchmarks/archive/
Long-term archival folder for launch-claim-supporting benchmark runs.
## Retention contract (per H-AUDIT-1 ratification §Q5)
- **Tier 2 only.** This folder holds gzipped JSONL from runs that support a
published LoCoMo or SWE-ContextBench claim (e.g. H-42a/b Stage 2 full-run
when it lands). Sprint-internal probes (Stage 1 mikro-eval, Stage 2 4-cell
mini, reruns) stay in `benchmarks/results/` and are pruned at sprint close.
- **12 months minimum** from commit date. Longer at PM discretion for
EU AI Act / legal / compliance alignment.
- **Gzipped JSONL only.** Expected ≤300MB per full run (~20-30% of raw).
- **Includes `reasoning_content`** when Stage 2 thinking=on runs land here —
external reviewers asking "why did the model answer this way on item N"
need the provider's reasoning chain. Exclusion rules (§2.4 of design doc)
apply to the read path, not to the archive write path.
## Naming
```
h-42a-stage-2-full-YYYY-MM-DD.jsonl.gz # Stage 2 main-run artifact
h-42b-stage-2-full-YYYY-MM-DD.jsonl.gz # Comparative run (if cut)
```
## NOT gitignored
Both the folder and its contents ARE committed — see `.gitignore` exemption
rule (Sprint 11 A2).
## Sprint 11 state
Empty. Sprint 11 only provisions the folder. Real archival files land in the
sprint that executes H-42a/b.

View File

@@ -1,76 +0,0 @@
{
"verdict": "PASS",
"k_opus_gpt": 0.847958297132928,
"k_opus_minimax": 0.8548922056384745,
"k_gpt_minimax": 0.7877758913412564,
"k_conservative_trio": 0.7877758913412564,
"minimax_parse_success": 100,
"minimax_n_total": 100,
"minimax_lat_p50_ms": 11927,
"minimax_lat_p95_ms": 31426,
"minimax_routing_errors": 0,
"minimax_retries_total": 0,
"minimax_prompt_tokens_total": 53855,
"minimax_completion_tokens_total": 48920,
"cost_actual_usd": 0.0749,
"per_cell": [
{
"cell": "no-context",
"n": 20,
"mm_parsed": 20,
"k_og": 1.0,
"k_om": 1.0,
"k_gm": 1.0
},
{
"cell": "oracle-context",
"n": 20,
"mm_parsed": 20,
"k_og": 0.7058823529411763,
"k_om": 0.7916666666666667,
"k_gm": 0.7058823529411763
},
{
"cell": "full-context",
"n": 20,
"mm_parsed": 20,
"k_og": 0.8,
"k_om": 0.7,
"k_gm": 0.7058823529411763
},
{
"cell": "retrieval",
"n": 20,
"mm_parsed": 20,
"k_og": 1.0,
"k_om": 0.8936170212765956,
"k_gm": 0.8936170212765956
},
{
"cell": "agentic",
"n": 20,
"mm_parsed": 20,
"k_og": 0.782608695652174,
"k_om": 0.8979591836734693,
"k_gm": 0.6874999999999999
}
],
"confusion_opus_gpt": {
"correct_correct": 32,
"correct_incorrect": 7,
"incorrect_correct": 0,
"incorrect_incorrect": 61
},
"confusion_opus_minimax": {
"correct_correct": 37,
"correct_incorrect": 2,
"incorrect_correct": 5,
"incorrect_incorrect": 56
},
"confusion_gpt_minimax": {
"correct_correct": 32,
"correct_incorrect": 0,
"incorrect_correct": 10,
"incorrect_incorrect": 58
}
}

View File

@@ -1,312 +0,0 @@
"""
Manifest v6 Phase 2 pre-flight — cold alias probes
====================================================
Validates both v6 judge aliases are production-ready with a minimal
probe sample before any N=400 commit. 6 calls total (3 MiniMax + 3
Kimi) on the first 3 split instances from §1.3h sample.
Probes use direct HTTP to upstream endpoints (same methodology as
§1.3g / §1.3h / §1.3h-C / v6 κ re-cal). LiteLLM proxy is NOT in the
loop — isolates upstream routing/parse behavior from middleware.
Kimi backup has never been exercised in production under v6 authority;
this is the first production-class test.
Scope: §11-compliant (read-only access to frozen files; new artefact
under benchmarks/calibration/v6-kappa-recal/).
"""
from __future__ import annotations
import json
import re
import sys
import time
import urllib.error
import urllib.request
from datetime import datetime, timezone
from pathlib import Path
try:
sys.stdout.reconfigure(encoding="utf-8", errors="replace")
sys.stderr.reconfigure(encoding="utf-8", errors="replace")
except Exception:
pass
OUT_PATH = Path("D:/Projects/waggle-os/benchmarks/calibration/v6-kappa-recal/phase2-cold-probes.jsonl")
# Reuse first 3 split instances from §1.3h — they exercised both providers before
SAMPLE_PATH = Path("D:/Projects/waggle-os/benchmarks/probes/judge-swap-validation/split-cases-sample.jsonl")
JUDGE_PROMPT_TEMPLATE = "\n".join([
"You are evaluating whether an LLM's answer is correct against ground truth.",
"",
"## Question",
"{question}",
"",
"## Ground-truth answer",
"{ground_truth}",
"",
"## Ground-truth supporting context (excerpt shown to the model)",
"{context}",
"",
"## Model's answer",
"{model_answer}",
"",
"## Your task",
"",
"Step 1: Determine if the model's answer is correct.",
"- \"correct\" means the model's answer contains all required facts from ground truth, with no additional incorrect claims.",
"- Minor phrasing differences, synonyms, or alternative but equivalent formulations are acceptable.",
"- Extra detail is acceptable ONLY if it is factually correct.",
"",
"Step 2: If incorrect, assign exactly one failure mode using this decision tree:",
"",
"1. Does the model explicitly refuse or say it does not know? -> F1 (ABSTAIN)",
"2. Does the model answer a DIFFERENT question than was asked (coherent but off-topic)? -> F5 (OFF-TOPIC)",
"3. Does the model rely on entities, names, dates, or claims that do NOT appear in the ground-truth context (fabrication)? -> F4 (HALLUCINATED)",
"4. Does the model correctly state SOME required facts but miss others, without stating any incorrect facts? -> F2 (PARTIAL)",
"5. Otherwise (model states facts derived from the context but gets them wrong): -> F3 (INCORRECT)",
"",
"Step 3: Return JSON only, no prose, in this exact schema:",
"",
"{{",
" \"verdict\": \"correct\" | \"incorrect\",",
" \"failure_mode\": null | \"F1\" | \"F2\" | \"F3\" | \"F4\" | \"F5\",",
" \"rationale\": \"one sentence explaining the verdict\"",
"}}",
"",
"If verdict is \"correct\", failure_mode MUST be null.",
"If verdict is \"incorrect\", failure_mode MUST be one of F1-F5.",
])
def ts() -> str:
return datetime.now(timezone.utc).isoformat()
def logmsg(msg: str) -> None:
print(f"{ts()} {msg}", flush=True)
def load_env() -> dict[str, str]:
env_path = Path("D:/Projects/waggle-os/.env")
out: dict[str, str] = {}
for line in env_path.read_text(encoding="utf-8").splitlines():
line = line.strip()
if not line or line.startswith("#") or "=" not in line:
continue
k, _, v = line.partition("=")
out[k.strip()] = v.strip().strip('"').strip("'")
return out
def extract_json_body(raw: str) -> dict | None:
if not raw:
return None
trimmed = raw.strip()
if trimmed.startswith("```"):
m = re.match(r"^```(?:json)?\s*\n?(.*?)```\s*$", trimmed, re.DOTALL)
if m:
trimmed = m.group(1).strip()
try:
return json.loads(trimmed)
except Exception:
pass
first = trimmed.find("{")
last = trimmed.rfind("}")
if first != -1 and last != -1 and last > first:
try:
return json.loads(trimmed[first:last + 1])
except Exception:
return None
return None
def parse_verdict(raw: str) -> tuple[str | None, str | None, str | None]:
body = extract_json_body(raw)
if not isinstance(body, dict):
return (None, None, None)
v = body.get("verdict")
fm = body.get("failure_mode")
rat = body.get("rationale")
if v not in ("correct", "incorrect"):
return (None, None, None)
if fm is not None and fm not in ("F1", "F2", "F3", "F4", "F5"):
fm = None
return (v, fm, rat if isinstance(rat, str) else None)
def http_post_json(url: str, headers: dict, body: dict, timeout_s: int = 60) -> tuple[int, dict | str]:
req = urllib.request.Request(
url, data=json.dumps(body).encode("utf-8"), method="POST",
headers={"Content-Type": "application/json", **headers},
)
try:
with urllib.request.urlopen(req, timeout=timeout_s) as resp:
raw = resp.read().decode("utf-8", errors="replace")
try:
return resp.status, json.loads(raw)
except Exception:
return resp.status, raw
except urllib.error.HTTPError as e:
try:
return e.code, e.read().decode("utf-8", errors="replace")[:2000]
except Exception:
return e.code, ""
except Exception as e:
return 0, f"{type(e).__name__}: {e}"
def call_minimax(prompt: str, or_key: str) -> dict:
url = "https://openrouter.ai/api/v1/chat/completions"
headers = {"Authorization": f"Bearer {or_key}"}
body = {
"model": "minimax/minimax-m2.7",
"messages": [{"role": "user", "content": prompt}],
"temperature": 0.0,
"max_tokens": 4096,
}
started = time.time()
status, resp = http_post_json(url, headers, body)
latency = int((time.time() - started) * 1000)
if status == 200 and isinstance(resp, dict):
choices = resp.get("choices") or []
if choices:
msg = choices[0].get("message") or {}
content = msg.get("content") or msg.get("reasoning_content") or ""
usage = resp.get("usage", {})
return {
"raw_text": content,
"status": 200,
"latency_ms": latency,
"prompt_tokens": usage.get("prompt_tokens"),
"completion_tokens": usage.get("completion_tokens"),
"provider": "minimax",
"alias": "minimax-m27-via-openrouter",
"routing": "openrouter_direct_http",
"error": None,
}
return {
"raw_text": "",
"status": status,
"latency_ms": latency,
"error": str(resp)[:300],
"provider": "minimax",
"alias": "minimax-m27-via-openrouter",
"routing": "openrouter_direct_http",
"prompt_tokens": None,
"completion_tokens": None,
}
def call_kimi(prompt: str, moonshot_key: str) -> dict:
"""Kimi K2.6 via Moonshot direct intl endpoint. First production-class
run of the v6 kimi-k26-direct alias equivalent (LiteLLM proxy not in
loop; routes directly to upstream)."""
url = "https://api.moonshot.ai/v1/chat/completions"
headers = {"Authorization": f"Bearer {moonshot_key}"}
body = {
"model": "kimi-k2.6",
"messages": [{"role": "user", "content": prompt}],
"max_tokens": 4096,
}
started = time.time()
status, resp = http_post_json(url, headers, body)
latency = int((time.time() - started) * 1000)
if status == 200 and isinstance(resp, dict):
choices = resp.get("choices") or []
if choices:
msg = choices[0].get("message") or {}
content = msg.get("content") or msg.get("reasoning_content") or ""
usage = resp.get("usage", {})
return {
"raw_text": content,
"status": 200,
"latency_ms": latency,
"prompt_tokens": usage.get("prompt_tokens"),
"completion_tokens": usage.get("completion_tokens"),
"provider": "kimi",
"alias": "kimi-k26-direct",
"routing": "moonshot_direct_http",
"error": None,
}
return {
"raw_text": "",
"status": status,
"latency_ms": latency,
"error": str(resp)[:300],
"provider": "kimi",
"alias": "kimi-k26-direct",
"routing": "moonshot_direct_http",
"prompt_tokens": None,
"completion_tokens": None,
}
def main() -> int:
logmsg("[cold-probes] Phase 2 pre-flight START")
env = load_env()
or_key = env.get("OPENROUTER_API_KEY", "").strip()
moonshot_key = env.get("MOONSHOT_API_KEY", "").strip()
if not or_key or not moonshot_key:
logmsg("[cold-probes] FATAL missing keys (OR or MOONSHOT)")
return 2
sample = []
with SAMPLE_PATH.open("r", encoding="utf-8") as f:
for line in f:
line = line.strip()
if line:
sample.append(json.loads(line))
sample = sample[:3] # first 3 instances
logmsg(f"[cold-probes] loaded {len(sample)} probe instances from §1.3h split sample")
rows = []
for i, s in enumerate(sample):
prompt = JUDGE_PROMPT_TEMPLATE.format(
question=s["question"], ground_truth=s["ground_truth"],
context=s["context"], model_answer=s["model_answer"],
)
for fn, label in ((call_minimax, "minimax"), (call_kimi, "kimi")):
resp = fn(prompt, or_key if label == "minimax" else moonshot_key)
verdict, fm, rat = parse_verdict(resp["raw_text"])
rows.append({
"instance_id": s["instance_id"],
"cell": s["cell"],
"provider": resp["provider"],
"alias": resp["alias"],
"routing": resp["routing"],
"http_status": resp["status"],
"error": resp.get("error"),
"latency_ms": resp["latency_ms"],
"prompt_tokens": resp.get("prompt_tokens"),
"completion_tokens": resp.get("completion_tokens"),
"parsed_verdict": verdict,
"parsed_failure_mode": fm,
"parsed_rationale": rat,
"raw_text": resp["raw_text"],
"opus_verdict_ref": s.get("opus_verdict"),
"gpt_verdict_ref": s.get("gpt_verdict"),
})
logmsg(
f"[cold-probes] {label:7} {i+1}/3 {s['instance_id']} status={resp['status']} "
f"verdict={verdict} lat={resp['latency_ms']}ms"
)
OUT_PATH.parent.mkdir(parents=True, exist_ok=True)
with OUT_PATH.open("w", encoding="utf-8") as f:
for r in rows:
f.write(json.dumps(r, ensure_ascii=False) + "\n")
# Summary
mm_rows = [r for r in rows if r["provider"] == "minimax"]
km_rows = [r for r in rows if r["provider"] == "kimi"]
mm_parsed = sum(1 for r in mm_rows if r["parsed_verdict"] is not None)
km_parsed = sum(1 for r in km_rows if r["parsed_verdict"] is not None)
logmsg(f"[cold-probes] SUMMARY MiniMax: {mm_parsed}/3 parsed Kimi: {km_parsed}/3 parsed")
logmsg("[cold-probes] END")
return 0 if (mm_parsed == 3 and km_parsed == 3) else 3
if __name__ == "__main__":
sys.exit(main())

View File

@@ -1,100 +0,0 @@
{"instance_id": "locomo_conv-50_q086", "cell": "no-context", "question": "What sports activity is Calvin planning to try after the tour with Frank Ocean?", "ground_truth": "Skiing", "context": "Session 9 (3:15 pm on 21 June, 2023):\nCalvin: Nope, not yet! I haven't been there before, but it's on my to-do list for after my tour with Frank Ocean ends. Can't wait to see them in person! Look at that snowy peak. [image: a photo of a person on skis on a snowy mountain]", "model_answer": "Surfing", "opus_verdict": "incorrect", "opus_failure_mode": "F3", "gpt_verdict": "incorrect", "gpt_failure_mode": "F3", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F4"}
{"instance_id": "locomo_conv-26_q059", "cell": "no-context", "question": "Would Caroline be considered religious?", "ground_truth": "Somewhat, but not extremely religious", "context": "Session 12 (1:50 pm on 17 August, 2023):\nCaroline: Hey Mel! How're ya doin'? Recently, I had a not-so-great experience on a hike. I ran into a group of religious conservatives who said something that really upset me. It made me think how much work we still have to do for LGBTQ rights. It's been so helpful to have people around me who accept and support me, so I know I'll be ok!\n\nSession 14 (1:33 pm on 25 August, 2023):\nCaroline: Thanks! It was made for a local church and shows time changing our lives. I made it to show my own journey as a transgender woman and how we should accept growth and change. [image: a photo of a large stained glass window in a church]", "model_answer": "Cannot be determined without context.", "opus_verdict": "incorrect", "opus_failure_mode": "F1", "gpt_verdict": "incorrect", "gpt_failure_mode": "F1", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F1"}
{"instance_id": "locomo_conv-44_q000", "cell": "no-context", "question": "Which year did Audrey adopt the first three of her dogs?", "ground_truth": "2020", "context": "Session 1 (1:10 pm on 27 March, 2023):\nAudrey: I've had them for 3 years! Their names are Pepper, Precious and Panda. I can't live without my little ones!", "model_answer": "1954", "opus_verdict": "incorrect", "opus_failure_mode": "F4", "gpt_verdict": "incorrect", "gpt_failure_mode": "F4", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F4"}
{"instance_id": "locomo_conv-41_q080", "cell": "no-context", "question": "What did Maria make for her home to remind her of a trip to England?", "ground_truth": "painting of a castle on a hill", "context": "Session 8 (6:03 pm on 6 March, 2023):\nMaria: Thanks, John! I got the idea from that trip to England a few years ago - I was mesmerized by the castles. I wanted to bring that same magic to my home, like a reminder of the world's beauty. I made a painting too! [image: a photography of a painting of a castle on a hill]", "model_answer": "A wind chime", "opus_verdict": "incorrect", "opus_failure_mode": "F4", "gpt_verdict": "incorrect", "gpt_failure_mode": "F3", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F4"}
{"instance_id": "locomo_conv-42_q001", "cell": "no-context", "question": "What kind of interests do Joanna and Nate share?", "ground_truth": "Watching movies, making desserts", "context": "Session 1 (7:31 pm on 21 January, 2022):\nJoanna: Yeah! Besides writing, I also enjoy reading, watching movies, and exploring nature. Anything else you enjoy doing, Nate?\nNate: Playing video games and watching movies are my main hobbies.\nJoanna: Cool, Nate! So we both have similar interests. What type of movies do you like best?\n\nSession 3 (9:27 am on 7 February, 2022):\nNate: Thanks, Joanna. Not much has changed for me, but I just discovered that I can make coconut milk icecream and gave it a try. It was actually pretty good, so I'm proud of myself. [image: a photo of a bowl of ice cream with a spoon in it]\n\nSession 4 (1:07 pm on 25 February, 2022):\nNate: Thanks, Joanna! It means a lot that you enjoy the desserts I bake.\n\nSession 10 (11:54 am on 2 May, 2022):\nJoanna: Not much is new other than the screenplay. Been working on some projects and testing out dairy-free dessert recipes for friends and fam. Here's a pic of a cake I made recently! [image: a photo of a cake with white frosting on a wooden table]\n\nSession 20 (6:03 pm on 5 September, 2022):\nJoanna: Hey Nate! Cute turtles! Bummer about the setback. Any positive vibes comin' your way? I just revised on of my old recipes and made this! [image: a photo of a piece of cake with strawberries and chocolate]", "model_answer": "Reading", "opus_verdict": "incorrect", "opus_failure_mode": "F3", "gpt_verdict": "incorrect", "gpt_failure_mode": "F3", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F3"}
{"instance_id": "locomo_conv-43_q155", "cell": "no-context", "question": "How did John feel after being able to jog without pain?", "ground_truth": "It was a huge success.", "context": "Session 24 (3:37 pm on 16 December, 2023):\nJohn: It was great! After being out for so long, jogging without any pain was a huge success. My wife and I hosted a small get-together with friends and family to celebrate.", "model_answer": "Relieved.", "opus_verdict": "incorrect", "opus_failure_mode": "F3", "gpt_verdict": "incorrect", "gpt_failure_mode": "F3", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F4"}
{"instance_id": "locomo_conv-50_q015", "cell": "no-context", "question": "Why does Dave regularly visit parks?", "ground_truth": "because it relaxes and calms him", "context": "Session 1 (11:53 am on 23 March, 2023):\nDave: That's great, can't wait for you to return from your trip. I've been spending lots of time at this beautiful park lately - so calming. Have you been there before? [image: a photography of a park with a lake and a few boats]\n\nSession 8 (2:31 pm on 9 June, 2023):\nDave: Hey, nice photo of the studio! Working in a team yields incredible results, well done! Keep pushing it! I've been doing good - thanks for asking. I've been exploring some parks on the weekends to relax - it's so peaceful being surrounded by nature. Are there any chill spots you enjoy in Boston?", "model_answer": "Unknown.", "opus_verdict": "incorrect", "opus_failure_mode": "F1", "gpt_verdict": "incorrect", "gpt_failure_mode": "F1", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F1"}
{"instance_id": "locomo_conv-48_q020", "cell": "no-context", "question": "Which games have Jolene and her partner played together?", "ground_truth": "Detroit, Walking Dead, Battlefield 1, It Takes Two, Overcooked 2", "context": "Session 2 (9:49 am on 27 January, 2023):\nJolene: They are very unusual pets! Here's me and my partner gaming last week - it's so fun. We played the game \"Detroit\" on the console. We are both crazy about this activity! [image: a photo of a person laying in bed with a dog watching tv]\nJolene: We are planning to play \"Walking Dead\" next Saturday.\n\nSession 15 (7:37 pm on 9 July, 2023):\nJolene: Yeah, we love playing \"It takes two\" together! It's a fun team-strategy game and it's competitive. Plus, it's a great way for us to bond. Do you have any activities you like doing with people?\n\nSession 19 (12:52 am on 19 August, 2023):\nJolene: Oh, I forgot to mention Overcooked 2 - this is a good co-op game if you're into hilarious and chaotic cooking. My partner and I often play for bets! I once won three large pizzas!\n\nSession 20 (9:11 am on 21 August, 2023):\nJolene: Long time no talk! We were given a new game for the console last week, it is Battlefield 1. What's been up with you?", "model_answer": "Animal Crossing and Minecraft", "opus_verdict": "incorrect", "opus_failure_mode": "F4", "gpt_verdict": "incorrect", "gpt_failure_mode": "F4", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F4"}
{"instance_id": "locomo_conv-48_q055", "cell": "no-context", "question": "Which pet did Jolene adopt more recently - Susie or Seraphim?", "ground_truth": "Seraphim", "context": "Session 2 (9:49 am on 27 January, 2023):\nJolene: I bought it a year ago in Paris.\nJolene: Even as a child I learned to play on my own.\n\nSession 16 (9:26 am on 1 August, 2023):\nJolene: I adopted her two years ago when I was feeling lonely and wanted some company.", "model_answer": "Seraphim", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "correct", "gpt_failure_mode": null, "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
{"instance_id": "locomo_conv-26_q128", "cell": "no-context", "question": "What did Caroline find in her neighborhood during her walk?", "ground_truth": "a rainbow sidewalk", "context": "Session 14 (1:33 pm on 25 August, 2023):\nCaroline: I was out walking in my neighborhood when I came across this cool rainbow sidewalk for Pride Month. It was so vibrant and welcoming, I had to take a picture! It reminds us that love and acceptance are everywhere—even where we least expect it.", "model_answer": "A butterfly.", "opus_verdict": "incorrect", "opus_failure_mode": "F4", "gpt_verdict": "incorrect", "gpt_failure_mode": "F4", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F4"}
{"instance_id": "locomo_conv-50_q010", "cell": "no-context", "question": "When did Calvin's place get flooded in Tokyo?", "ground_truth": "On a week before 16 May, 2023", "context": "Session 6 (11:50 am on 16 May, 2023):\nCalvin: Hey Dave, not everything has been going smoothly. I had an incident last week where my place got flooded, but thankfully, I managed to save my music gear and favorite microphone. It's been tough, but I'm staying positive and looking forward to getting everything fixed up.", "model_answer": "October 2019", "opus_verdict": "incorrect", "opus_failure_mode": "F4", "gpt_verdict": "incorrect", "gpt_failure_mode": "F4", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F4"}
{"instance_id": "locomo_conv-50_q078", "cell": "no-context", "question": "What was the necklace Calvin received meant to remind him of?", "ground_truth": "why he keeps hustling as a musician", "context": "Session 4 (6:24 pm on 1 May, 2023):\nCalvin: Thanks, Dave! I got it from another artist as a gift - it's a great reminder of why I keep hustling as a musician!", "model_answer": "Home.", "opus_verdict": "incorrect", "opus_failure_mode": "F3", "gpt_verdict": "incorrect", "gpt_failure_mode": "F3", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F4"}
{"instance_id": "locomo_conv-47_q092", "cell": "no-context", "question": "What did John create for the charitable foundation that helped generate reports for analysis?", "ground_truth": "computer application on smartphones", "context": "Session 11 (5:00 pm on 11 May, 2022):\nJohn: Previously, this foundation used paper records and all inventory was recorded manually. I made an application that structured their work, and now everything they need for inventory is in one application on their smartphone.", "model_answer": "A database", "opus_verdict": "incorrect", "opus_failure_mode": "F3", "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F4"}
{"instance_id": "locomo_conv-30_q057", "cell": "no-context", "question": "What advice does Gina give to Jon about running a successful business?", "ground_truth": "build relationships with customers, create a strong brand image, stay positive", "context": "Session 7 (7:28 pm on 23 March, 2023):\nJon: Yeah, brand identity is key. Make sure yours stands out. Also be sure to build relationships with your customers – let them know you care. And don't forget to stay positive and motivate others. Your energy will be contagious!", "model_answer": "Focus on the customer.", "opus_verdict": "incorrect", "opus_failure_mode": "F2", "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F2"}
{"instance_id": "locomo_conv-30_q029", "cell": "no-context", "question": "Which cities has Jon visited?", "ground_truth": "Paris, Rome", "context": "Session 2 (2:32 pm on 29 January, 2023):\nJon: Hey Gina! Thanks for asking. I'm on the hunt for the ideal spot for my dance studio and it's been quite a journey! I've been looking at different places and picturing how the space would look. I even found a place with great natural light! Oh, I've been to Paris yesterday! It was sooo cool. [image: a photo of a bathroom with a blue floor and a pink wall]\n\nSession 15 (10:04 am on 19 June, 2023):\nJon: Hey Gina, hope you're doing great! Still working on my biz. Took a short trip last week to Rome to clear my mind a little.", "model_answer": "Unknown.", "opus_verdict": "incorrect", "opus_failure_mode": "F1", "gpt_verdict": "incorrect", "gpt_failure_mode": "F1", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F1"}
{"instance_id": "locomo_conv-44_q066", "cell": "no-context", "question": "What type of dog was Andrew looking to adopt based on his living space?", "ground_truth": "smaller dog", "context": "Session 5 (10:47 am on 6 May, 2023):\nAndrew: Meeting all these adorable pups has been awesome! For those considering getting a pup, the size of living space and the exercise needs of the breed are important. For me, a person living in an apartment, a smaller dog would be best, but if one is active, consider getting one that loves to play and run.", "model_answer": "A small dog", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "correct", "gpt_failure_mode": null, "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
{"instance_id": "locomo_conv-44_q002", "cell": "no-context", "question": "What kind of indoor activities has Andrew pursued with his girlfriend?", "ground_truth": "boardgames, volunteering at pet shelter, wine tasting, growing flowers", "context": "Session 13 (3:52 pm on 27 July, 2023):\nAndrew: Hey Audrey! How are you? My GF and I just had a great experience volunteering at a pet shelter on Monday - it was so rewarding! We loved spending time with those cute animals and it gave us so much joy. It was so rewarding, it reminded me just how much I love them!\n\nSession 19 (5:53 pm on 24 September, 2023):\nAndrew: Yeah! They really do bring so much into our lives - it's amazing to watch them interact. Here's something I've been taking care of lately. Look at those flowers! [image: a photo of a balcony with a bunch of flowers on it]\n\nSession 23 (4:22 pm on 13 October, 2023):\nAndrew: Hey Audrey, it's been a busy week for me. Last Tuesday, my gf, Toby, and I had a really awesome night playing board games. It was really nice. What's been up with you lately? [image: a photography of a dog sitting at a chess board with a chess set in front of him]\n\nSession 25 (10:14 am on 24 October, 2023):\nAndrew: Hi Audrey! How have you been lately? My girlfriend and I went to this awesome wine tasting last weekend. It was great! We tried so many unique wines and learned a lot. I was surprised at how much I enjoyed it. A reminder to step out of the comfort zone!", "model_answer": "Board games", "opus_verdict": "incorrect", "opus_failure_mode": "F2", "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F2"}
{"instance_id": "locomo_conv-47_q046", "cell": "no-context", "question": "When did John and James meet at McGee's bar?", "ground_truth": "August 27, 2022", "context": "Session 21 (9:18 pm on 26 August, 2022):\nJohn: Great, then I agree! See you tomorrow at McGee's Pub!", "model_answer": "Never", "opus_verdict": "incorrect", "opus_failure_mode": "F3", "gpt_verdict": "incorrect", "gpt_failure_mode": "F3", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F3"}
{"instance_id": "locomo_conv-43_q076", "cell": "no-context", "question": "What kind of picture did Tim share as part of their Harry Potter book collection?", "ground_truth": "MinaLima's creation from the Harry Potter films", "context": "Session 2 (5:08 pm on 15 June, 2023):\nTim: Thanks! That picture is from MinaLima. They created all the props for the Harry Potter films, and I love their work. It's like having a piece of the wizarding world at home!", "model_answer": "A selfie", "opus_verdict": "incorrect", "opus_failure_mode": "F3", "gpt_verdict": "incorrect", "gpt_failure_mode": "F4", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F4"}
{"instance_id": "locomo_conv-41_q065", "cell": "no-context", "question": "What is John's main focus in local politics?", "ground_truth": "Improving education and infrastructure", "context": "Session 1 (11:01 am on 17 December, 2022):\nJohn: I'm passionate about improving education and infrastructure in our community. Those are my main focuses.", "model_answer": "Unknown which John.", "opus_verdict": "incorrect", "opus_failure_mode": "F1", "gpt_verdict": "incorrect", "gpt_failure_mode": "F1", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F1"}
{"instance_id": "locomo_conv-50_q086", "cell": "oracle-context", "question": "What sports activity is Calvin planning to try after the tour with Frank Ocean?", "ground_truth": "Skiing", "context": "Session 9 (3:15 pm on 21 June, 2023):\nCalvin: Nope, not yet! I haven't been there before, but it's on my to-do list for after my tour with Frank Ocean ends. Can't wait to see them in person! Look at that snowy peak. [image: a photo of a person on skis on a snowy mountain]", "model_answer": "Skiing", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "correct", "gpt_failure_mode": null, "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
{"instance_id": "locomo_conv-26_q059", "cell": "oracle-context", "question": "Would Caroline be considered religious?", "ground_truth": "Somewhat, but not extremely religious", "context": "Session 12 (1:50 pm on 17 August, 2023):\nCaroline: Hey Mel! How're ya doin'? Recently, I had a not-so-great experience on a hike. I ran into a group of religious conservatives who said something that really upset me. It made me think how much work we still have to do for LGBTQ rights. It's been so helpful to have people around me who accept and support me, so I know I'll be ok!\n\nSession 14 (1:33 pm on 25 August, 2023):\nCaroline: Thanks! It was made for a local church and shows time changing our lives. I made it to show my own journey as a transgender woman and how we should accept growth and change. [image: a photo of a large stained glass window in a church]", "model_answer": "Yes", "opus_verdict": "incorrect", "opus_failure_mode": "F3", "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F2"}
{"instance_id": "locomo_conv-44_q000", "cell": "oracle-context", "question": "Which year did Audrey adopt the first three of her dogs?", "ground_truth": "2020", "context": "Session 1 (1:10 pm on 27 March, 2023):\nAudrey: I've had them for 3 years! Their names are Pepper, Precious and Panda. I can't live without my little ones!", "model_answer": "2020", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "correct", "gpt_failure_mode": null, "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
{"instance_id": "locomo_conv-41_q080", "cell": "oracle-context", "question": "What did Maria make for her home to remind her of a trip to England?", "ground_truth": "painting of a castle on a hill", "context": "Session 8 (6:03 pm on 6 March, 2023):\nMaria: Thanks, John! I got the idea from that trip to England a few years ago - I was mesmerized by the castles. I wanted to bring that same magic to my home, like a reminder of the world's beauty. I made a painting too! [image: a photography of a painting of a castle on a hill]", "model_answer": "A painting", "opus_verdict": "incorrect", "opus_failure_mode": "F2", "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F2"}
{"instance_id": "locomo_conv-42_q001", "cell": "oracle-context", "question": "What kind of interests do Joanna and Nate share?", "ground_truth": "Watching movies, making desserts", "context": "Session 1 (7:31 pm on 21 January, 2022):\nJoanna: Yeah! Besides writing, I also enjoy reading, watching movies, and exploring nature. Anything else you enjoy doing, Nate?\nNate: Playing video games and watching movies are my main hobbies.\nJoanna: Cool, Nate! So we both have similar interests. What type of movies do you like best?\n\nSession 3 (9:27 am on 7 February, 2022):\nNate: Thanks, Joanna. Not much has changed for me, but I just discovered that I can make coconut milk icecream and gave it a try. It was actually pretty good, so I'm proud of myself. [image: a photo of a bowl of ice cream with a spoon in it]\n\nSession 4 (1:07 pm on 25 February, 2022):\nNate: Thanks, Joanna! It means a lot that you enjoy the desserts I bake.\n\nSession 10 (11:54 am on 2 May, 2022):\nJoanna: Not much is new other than the screenplay. Been working on some projects and testing out dairy-free dessert recipes for friends and fam. Here's a pic of a cake I made recently! [image: a photo of a cake with white frosting on a wooden table]\n\nSession 20 (6:03 pm on 5 September, 2022):\nJoanna: Hey Nate! Cute turtles! Bummer about the setback. Any positive vibes comin' your way? I just revised on of my old recipes and made this! [image: a photo of a piece of cake with strawberries and chocolate]", "model_answer": "Movies", "opus_verdict": "incorrect", "opus_failure_mode": "F2", "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F2"}
{"instance_id": "locomo_conv-43_q155", "cell": "oracle-context", "question": "How did John feel after being able to jog without pain?", "ground_truth": "It was a huge success.", "context": "Session 24 (3:37 pm on 16 December, 2023):\nJohn: It was great! After being out for so long, jogging without any pain was a huge success. My wife and I hosted a small get-together with friends and family to celebrate.", "model_answer": "Great", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
{"instance_id": "locomo_conv-50_q015", "cell": "oracle-context", "question": "Why does Dave regularly visit parks?", "ground_truth": "because it relaxes and calms him", "context": "Session 1 (11:53 am on 23 March, 2023):\nDave: That's great, can't wait for you to return from your trip. I've been spending lots of time at this beautiful park lately - so calming. Have you been there before? [image: a photography of a park with a lake and a few boats]\n\nSession 8 (2:31 pm on 9 June, 2023):\nDave: Hey, nice photo of the studio! Working in a team yields incredible results, well done! Keep pushing it! I've been doing good - thanks for asking. I've been exploring some parks on the weekends to relax - it's so peaceful being surrounded by nature. Are there any chill spots you enjoy in Boston?", "model_answer": "To relax", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
{"instance_id": "locomo_conv-48_q020", "cell": "oracle-context", "question": "Which games have Jolene and her partner played together?", "ground_truth": "Detroit, Walking Dead, Battlefield 1, It Takes Two, Overcooked 2", "context": "Session 2 (9:49 am on 27 January, 2023):\nJolene: They are very unusual pets! Here's me and my partner gaming last week - it's so fun. We played the game \"Detroit\" on the console. We are both crazy about this activity! [image: a photo of a person laying in bed with a dog watching tv]\nJolene: We are planning to play \"Walking Dead\" next Saturday.\n\nSession 15 (7:37 pm on 9 July, 2023):\nJolene: Yeah, we love playing \"It takes two\" together! It's a fun team-strategy game and it's competitive. Plus, it's a great way for us to bond. Do you have any activities you like doing with people?\n\nSession 19 (12:52 am on 19 August, 2023):\nJolene: Oh, I forgot to mention Overcooked 2 - this is a good co-op game if you're into hilarious and chaotic cooking. My partner and I often play for bets! I once won three large pizzas!\n\nSession 20 (9:11 am on 21 August, 2023):\nJolene: Long time no talk! We were given a new game for the console last week, it is Battlefield 1. What's been up with you?", "model_answer": "Detroit, It Takes Two, Overcooked 2", "opus_verdict": "incorrect", "opus_failure_mode": "F2", "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F2"}
{"instance_id": "locomo_conv-48_q055", "cell": "oracle-context", "question": "Which pet did Jolene adopt more recently - Susie or Seraphim?", "ground_truth": "Seraphim", "context": "Session 2 (9:49 am on 27 January, 2023):\nJolene: I bought it a year ago in Paris.\nJolene: Even as a child I learned to play on my own.\n\nSession 16 (9:26 am on 1 August, 2023):\nJolene: I adopted her two years ago when I was feeling lonely and wanted some company.", "model_answer": "Susie", "opus_verdict": "incorrect", "opus_failure_mode": "F3", "gpt_verdict": "incorrect", "gpt_failure_mode": "F3", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F3"}
{"instance_id": "locomo_conv-26_q128", "cell": "oracle-context", "question": "What did Caroline find in her neighborhood during her walk?", "ground_truth": "a rainbow sidewalk", "context": "Session 14 (1:33 pm on 25 August, 2023):\nCaroline: I was out walking in my neighborhood when I came across this cool rainbow sidewalk for Pride Month. It was so vibrant and welcoming, I had to take a picture! It reminds us that love and acceptance are everywhere—even where we least expect it.", "model_answer": "A rainbow sidewalk", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "correct", "gpt_failure_mode": null, "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
{"instance_id": "locomo_conv-50_q010", "cell": "oracle-context", "question": "When did Calvin's place get flooded in Tokyo?", "ground_truth": "On a week before 16 May, 2023", "context": "Session 6 (11:50 am on 16 May, 2023):\nCalvin: Hey Dave, not everything has been going smoothly. I had an incident last week where my place got flooded, but thankfully, I managed to save my music gear and favorite microphone. It's been tough, but I'm staying positive and looking forward to getting everything fixed up.", "model_answer": "Last week", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "correct", "gpt_failure_mode": null, "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F2"}
{"instance_id": "locomo_conv-50_q078", "cell": "oracle-context", "question": "What was the necklace Calvin received meant to remind him of?", "ground_truth": "why he keeps hustling as a musician", "context": "Session 4 (6:24 pm on 1 May, 2023):\nCalvin: Thanks, Dave! I got it from another artist as a gift - it's a great reminder of why I keep hustling as a musician!", "model_answer": "Why he keeps hustling as a musician", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "correct", "gpt_failure_mode": null, "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
{"instance_id": "locomo_conv-47_q092", "cell": "oracle-context", "question": "What did John create for the charitable foundation that helped generate reports for analysis?", "ground_truth": "computer application on smartphones", "context": "Session 11 (5:00 pm on 11 May, 2022):\nJohn: Previously, this foundation used paper records and all inventory was recorded manually. I made an application that structured their work, and now everything they need for inventory is in one application on their smartphone.", "model_answer": "An application", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F2"}
{"instance_id": "locomo_conv-30_q057", "cell": "oracle-context", "question": "What advice does Gina give to Jon about running a successful business?", "ground_truth": "build relationships with customers, create a strong brand image, stay positive", "context": "Session 7 (7:28 pm on 23 March, 2023):\nJon: Yeah, brand identity is key. Make sure yours stands out. Also be sure to build relationships with your customers – let them know you care. And don't forget to stay positive and motivate others. Your energy will be contagious!", "model_answer": "Not mentioned.", "opus_verdict": "incorrect", "opus_failure_mode": "F1", "gpt_verdict": "incorrect", "gpt_failure_mode": "F1", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F1"}
{"instance_id": "locomo_conv-30_q029", "cell": "oracle-context", "question": "Which cities has Jon visited?", "ground_truth": "Paris, Rome", "context": "Session 2 (2:32 pm on 29 January, 2023):\nJon: Hey Gina! Thanks for asking. I'm on the hunt for the ideal spot for my dance studio and it's been quite a journey! I've been looking at different places and picturing how the space would look. I even found a place with great natural light! Oh, I've been to Paris yesterday! It was sooo cool. [image: a photo of a bathroom with a blue floor and a pink wall]\n\nSession 15 (10:04 am on 19 June, 2023):\nJon: Hey Gina, hope you're doing great! Still working on my biz. Took a short trip last week to Rome to clear my mind a little.", "model_answer": "Paris and Rome", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "correct", "gpt_failure_mode": null, "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
{"instance_id": "locomo_conv-44_q066", "cell": "oracle-context", "question": "What type of dog was Andrew looking to adopt based on his living space?", "ground_truth": "smaller dog", "context": "Session 5 (10:47 am on 6 May, 2023):\nAndrew: Meeting all these adorable pups has been awesome! For those considering getting a pup, the size of living space and the exercise needs of the breed are important. For me, a person living in an apartment, a smaller dog would be best, but if one is active, consider getting one that loves to play and run.", "model_answer": "A smaller dog", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "correct", "gpt_failure_mode": null, "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
{"instance_id": "locomo_conv-44_q002", "cell": "oracle-context", "question": "What kind of indoor activities has Andrew pursued with his girlfriend?", "ground_truth": "boardgames, volunteering at pet shelter, wine tasting, growing flowers", "context": "Session 13 (3:52 pm on 27 July, 2023):\nAndrew: Hey Audrey! How are you? My GF and I just had a great experience volunteering at a pet shelter on Monday - it was so rewarding! We loved spending time with those cute animals and it gave us so much joy. It was so rewarding, it reminded me just how much I love them!\n\nSession 19 (5:53 pm on 24 September, 2023):\nAndrew: Yeah! They really do bring so much into our lives - it's amazing to watch them interact. Here's something I've been taking care of lately. Look at those flowers! [image: a photo of a balcony with a bunch of flowers on it]\n\nSession 23 (4:22 pm on 13 October, 2023):\nAndrew: Hey Audrey, it's been a busy week for me. Last Tuesday, my gf, Toby, and I had a really awesome night playing board games. It was really nice. What's been up with you lately? [image: a photography of a dog sitting at a chess board with a chess set in front of him]\n\nSession 25 (10:14 am on 24 October, 2023):\nAndrew: Hi Audrey! How have you been lately? My girlfriend and I went to this awesome wine tasting last weekend. It was great! We tried so many unique wines and learned a lot. I was surprised at how much I enjoyed it. A reminder to step out of the comfort zone!", "model_answer": "Board games.", "opus_verdict": "incorrect", "opus_failure_mode": "F2", "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F2"}
{"instance_id": "locomo_conv-47_q046", "cell": "oracle-context", "question": "When did John and James meet at McGee's bar?", "ground_truth": "August 27, 2022", "context": "Session 21 (9:18 pm on 26 August, 2022):\nJohn: Great, then I agree! See you tomorrow at McGee's Pub!", "model_answer": "27 August, 2022", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "correct", "gpt_failure_mode": null, "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
{"instance_id": "locomo_conv-43_q076", "cell": "oracle-context", "question": "What kind of picture did Tim share as part of their Harry Potter book collection?", "ground_truth": "MinaLima's creation from the Harry Potter films", "context": "Session 2 (5:08 pm on 15 June, 2023):\nTim: Thanks! That picture is from MinaLima. They created all the props for the Harry Potter films, and I love their work. It's like having a piece of the wizarding world at home!", "model_answer": "MinaLima", "opus_verdict": "incorrect", "opus_failure_mode": "F2", "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F2"}
{"instance_id": "locomo_conv-41_q065", "cell": "oracle-context", "question": "What is John's main focus in local politics?", "ground_truth": "Improving education and infrastructure", "context": "Session 1 (11:01 am on 17 December, 2022):\nJohn: I'm passionate about improving education and infrastructure in our community. Those are my main focuses.", "model_answer": "Education and infrastructure", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "correct", "gpt_failure_mode": null, "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
{"instance_id": "locomo_conv-50_q086", "cell": "full-context", "question": "What sports activity is Calvin planning to try after the tour with Frank Ocean?", "ground_truth": "Skiing", "context": "Session 9 (3:15 pm on 21 June, 2023):\nCalvin: Nope, not yet! I haven't been there before, but it's on my to-do list for after my tour with Frank Ocean ends. Can't wait to see them in person! Look at that snowy peak. [image: a photo of a person on skis on a snowy mountain]", "model_answer": "skiing", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "correct", "gpt_failure_mode": null, "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
{"instance_id": "locomo_conv-26_q059", "cell": "full-context", "question": "Would Caroline be considered religious?", "ground_truth": "Somewhat, but not extremely religious", "context": "Session 12 (1:50 pm on 17 August, 2023):\nCaroline: Hey Mel! How're ya doin'? Recently, I had a not-so-great experience on a hike. I ran into a group of religious conservatives who said something that really upset me. It made me think how much work we still have to do for LGBTQ rights. It's been so helpful to have people around me who accept and support me, so I know I'll be ok!\n\nSession 14 (1:33 pm on 25 August, 2023):\nCaroline: Thanks! It was made for a local church and shows time changing our lives. I made it to show my own journey as a transgender woman and how we should accept growth and change. [image: a photo of a large stained glass window in a church]", "model_answer": "unknown", "opus_verdict": "incorrect", "opus_failure_mode": "F1", "gpt_verdict": "incorrect", "gpt_failure_mode": "F1", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F1"}
{"instance_id": "locomo_conv-44_q000", "cell": "full-context", "question": "Which year did Audrey adopt the first three of her dogs?", "ground_truth": "2020", "context": "Session 1 (1:10 pm on 27 March, 2023):\nAudrey: I've had them for 3 years! Their names are Pepper, Precious and Panda. I can't live without my little ones!", "model_answer": "2020", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "correct", "gpt_failure_mode": null, "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
{"instance_id": "locomo_conv-41_q080", "cell": "full-context", "question": "What did Maria make for her home to remind her of a trip to England?", "ground_truth": "painting of a castle on a hill", "context": "Session 8 (6:03 pm on 6 March, 2023):\nMaria: Thanks, John! I got the idea from that trip to England a few years ago - I was mesmerized by the castles. I wanted to bring that same magic to my home, like a reminder of the world's beauty. I made a painting too! [image: a photography of a painting of a castle on a hill]", "model_answer": "a painting", "opus_verdict": "incorrect", "opus_failure_mode": "F2", "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F2"}
{"instance_id": "locomo_conv-42_q001", "cell": "full-context", "question": "What kind of interests do Joanna and Nate share?", "ground_truth": "Watching movies, making desserts", "context": "Session 1 (7:31 pm on 21 January, 2022):\nJoanna: Yeah! Besides writing, I also enjoy reading, watching movies, and exploring nature. Anything else you enjoy doing, Nate?\nNate: Playing video games and watching movies are my main hobbies.\nJoanna: Cool, Nate! So we both have similar interests. What type of movies do you like best?\n\nSession 3 (9:27 am on 7 February, 2022):\nNate: Thanks, Joanna. Not much has changed for me, but I just discovered that I can make coconut milk icecream and gave it a try. It was actually pretty good, so I'm proud of myself. [image: a photo of a bowl of ice cream with a spoon in it]\n\nSession 4 (1:07 pm on 25 February, 2022):\nNate: Thanks, Joanna! It means a lot that you enjoy the desserts I bake.\n\nSession 10 (11:54 am on 2 May, 2022):\nJoanna: Not much is new other than the screenplay. Been working on some projects and testing out dairy-free dessert recipes for friends and fam. Here's a pic of a cake I made recently! [image: a photo of a cake with white frosting on a wooden table]\n\nSession 20 (6:03 pm on 5 September, 2022):\nJoanna: Hey Nate! Cute turtles! Bummer about the setback. Any positive vibes comin' your way? I just revised on of my old recipes and made this! [image: a photo of a piece of cake with strawberries and chocolate]", "model_answer": "watching movies", "opus_verdict": "incorrect", "opus_failure_mode": "F2", "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F2"}
{"instance_id": "locomo_conv-43_q155", "cell": "full-context", "question": "How did John feel after being able to jog without pain?", "ground_truth": "It was a huge success.", "context": "Session 24 (3:37 pm on 16 December, 2023):\nJohn: It was great! After being out for so long, jogging without any pain was a huge success. My wife and I hosted a small get-together with friends and family to celebrate.", "model_answer": "great", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
{"instance_id": "locomo_conv-50_q015", "cell": "full-context", "question": "Why does Dave regularly visit parks?", "ground_truth": "because it relaxes and calms him", "context": "Session 1 (11:53 am on 23 March, 2023):\nDave: That's great, can't wait for you to return from your trip. I've been spending lots of time at this beautiful park lately - so calming. Have you been there before? [image: a photography of a park with a lake and a few boats]\n\nSession 8 (2:31 pm on 9 June, 2023):\nDave: Hey, nice photo of the studio! Working in a team yields incredible results, well done! Keep pushing it! I've been doing good - thanks for asking. I've been exploring some parks on the weekends to relax - it's so peaceful being surrounded by nature. Are there any chill spots you enjoy in Boston?", "model_answer": "to relax", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
{"instance_id": "locomo_conv-48_q020", "cell": "full-context", "question": "Which games have Jolene and her partner played together?", "ground_truth": "Detroit, Walking Dead, Battlefield 1, It Takes Two, Overcooked 2", "context": "Session 2 (9:49 am on 27 January, 2023):\nJolene: They are very unusual pets! Here's me and my partner gaming last week - it's so fun. We played the game \"Detroit\" on the console. We are both crazy about this activity! [image: a photo of a person laying in bed with a dog watching tv]\nJolene: We are planning to play \"Walking Dead\" next Saturday.\n\nSession 15 (7:37 pm on 9 July, 2023):\nJolene: Yeah, we love playing \"It takes two\" together! It's a fun team-strategy game and it's competitive. Plus, it's a great way for us to bond. Do you have any activities you like doing with people?\n\nSession 19 (12:52 am on 19 August, 2023):\nJolene: Oh, I forgot to mention Overcooked 2 - this is a good co-op game if you're into hilarious and chaotic cooking. My partner and I often play for bets! I once won three large pizzas!\n\nSession 20 (9:11 am on 21 August, 2023):\nJolene: Long time no talk! We were given a new game for the console last week, it is Battlefield 1. What's been up with you?", "model_answer": "Detroit, It takes two, Overcooked 2", "opus_verdict": "incorrect", "opus_failure_mode": "F2", "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F2"}
{"instance_id": "locomo_conv-48_q055", "cell": "full-context", "question": "Which pet did Jolene adopt more recently - Susie or Seraphim?", "ground_truth": "Seraphim", "context": "Session 2 (9:49 am on 27 January, 2023):\nJolene: I bought it a year ago in Paris.\nJolene: Even as a child I learned to play on my own.\n\nSession 16 (9:26 am on 1 August, 2023):\nJolene: I adopted her two years ago when I was feeling lonely and wanted some company.", "model_answer": "unknown", "opus_verdict": "incorrect", "opus_failure_mode": "F1", "gpt_verdict": "incorrect", "gpt_failure_mode": "F1", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F1"}
{"instance_id": "locomo_conv-26_q128", "cell": "full-context", "question": "What did Caroline find in her neighborhood during her walk?", "ground_truth": "a rainbow sidewalk", "context": "Session 14 (1:33 pm on 25 August, 2023):\nCaroline: I was out walking in my neighborhood when I came across this cool rainbow sidewalk for Pride Month. It was so vibrant and welcoming, I had to take a picture! It reminds us that love and acceptance are everywhere—even where we least expect it.", "model_answer": "rainbow sidewalk", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "correct", "gpt_failure_mode": null, "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
{"instance_id": "locomo_conv-50_q010", "cell": "full-context", "question": "When did Calvin's place get flooded in Tokyo?", "ground_truth": "On a week before 16 May, 2023", "context": "Session 6 (11:50 am on 16 May, 2023):\nCalvin: Hey Dave, not everything has been going smoothly. I had an incident last week where my place got flooded, but thankfully, I managed to save my music gear and favorite microphone. It's been tough, but I'm staying positive and looking forward to getting everything fixed up.", "model_answer": "last week", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "correct", "gpt_failure_mode": null, "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F2"}
{"instance_id": "locomo_conv-50_q078", "cell": "full-context", "question": "What was the necklace Calvin received meant to remind him of?", "ground_truth": "why he keeps hustling as a musician", "context": "Session 4 (6:24 pm on 1 May, 2023):\nCalvin: Thanks, Dave! I got it from another artist as a gift - it's a great reminder of why I keep hustling as a musician!", "model_answer": "why I keep hustling as a musician", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "correct", "gpt_failure_mode": null, "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
{"instance_id": "locomo_conv-47_q092", "cell": "full-context", "question": "What did John create for the charitable foundation that helped generate reports for analysis?", "ground_truth": "computer application on smartphones", "context": "Session 11 (5:00 pm on 11 May, 2022):\nJohn: Previously, this foundation used paper records and all inventory was recorded manually. I made an application that structured their work, and now everything they need for inventory is in one application on their smartphone.", "model_answer": "an application", "opus_verdict": "incorrect", "opus_failure_mode": "F2", "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F2"}
{"instance_id": "locomo_conv-30_q057", "cell": "full-context", "question": "What advice does Gina give to Jon about running a successful business?", "ground_truth": "build relationships with customers, create a strong brand image, stay positive", "context": "Session 7 (7:28 pm on 23 March, 2023):\nJon: Yeah, brand identity is key. Make sure yours stands out. Also be sure to build relationships with your customers – let them know you care. And don't forget to stay positive and motivate others. Your energy will be contagious!", "model_answer": "unknown", "opus_verdict": "incorrect", "opus_failure_mode": "F1", "gpt_verdict": "incorrect", "gpt_failure_mode": "F1", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F1"}
{"instance_id": "locomo_conv-30_q029", "cell": "full-context", "question": "Which cities has Jon visited?", "ground_truth": "Paris, Rome", "context": "Session 2 (2:32 pm on 29 January, 2023):\nJon: Hey Gina! Thanks for asking. I'm on the hunt for the ideal spot for my dance studio and it's been quite a journey! I've been looking at different places and picturing how the space would look. I even found a place with great natural light! Oh, I've been to Paris yesterday! It was sooo cool. [image: a photo of a bathroom with a blue floor and a pink wall]\n\nSession 15 (10:04 am on 19 June, 2023):\nJon: Hey Gina, hope you're doing great! Still working on my biz. Took a short trip last week to Rome to clear my mind a little.", "model_answer": "Paris Rome", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "correct", "gpt_failure_mode": null, "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
{"instance_id": "locomo_conv-44_q066", "cell": "full-context", "question": "What type of dog was Andrew looking to adopt based on his living space?", "ground_truth": "smaller dog", "context": "Session 5 (10:47 am on 6 May, 2023):\nAndrew: Meeting all these adorable pups has been awesome! For those considering getting a pup, the size of living space and the exercise needs of the breed are important. For me, a person living in an apartment, a smaller dog would be best, but if one is active, consider getting one that loves to play and run.", "model_answer": "smaller dog", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "correct", "gpt_failure_mode": null, "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
{"instance_id": "locomo_conv-44_q002", "cell": "full-context", "question": "What kind of indoor activities has Andrew pursued with his girlfriend?", "ground_truth": "boardgames, volunteering at pet shelter, wine tasting, growing flowers", "context": "Session 13 (3:52 pm on 27 July, 2023):\nAndrew: Hey Audrey! How are you? My GF and I just had a great experience volunteering at a pet shelter on Monday - it was so rewarding! We loved spending time with those cute animals and it gave us so much joy. It was so rewarding, it reminded me just how much I love them!\n\nSession 19 (5:53 pm on 24 September, 2023):\nAndrew: Yeah! They really do bring so much into our lives - it's amazing to watch them interact. Here's something I've been taking care of lately. Look at those flowers! [image: a photo of a balcony with a bunch of flowers on it]\n\nSession 23 (4:22 pm on 13 October, 2023):\nAndrew: Hey Audrey, it's been a busy week for me. Last Tuesday, my gf, Toby, and I had a really awesome night playing board games. It was really nice. What's been up with you lately? [image: a photography of a dog sitting at a chess board with a chess set in front of him]\n\nSession 25 (10:14 am on 24 October, 2023):\nAndrew: Hi Audrey! How have you been lately? My girlfriend and I went to this awesome wine tasting last weekend. It was great! We tried so many unique wines and learned a lot. I was surprised at how much I enjoyed it. A reminder to step out of the comfort zone!", "model_answer": "playing board games", "opus_verdict": "incorrect", "opus_failure_mode": "F2", "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F2"}
{"instance_id": "locomo_conv-47_q046", "cell": "full-context", "question": "When did John and James meet at McGee's bar?", "ground_truth": "August 27, 2022", "context": "Session 21 (9:18 pm on 26 August, 2022):\nJohn: Great, then I agree! See you tomorrow at McGee's Pub!", "model_answer": "tomorrow", "opus_verdict": "incorrect", "opus_failure_mode": "F2", "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F3"}
{"instance_id": "locomo_conv-43_q076", "cell": "full-context", "question": "What kind of picture did Tim share as part of their Harry Potter book collection?", "ground_truth": "MinaLima's creation from the Harry Potter films", "context": "Session 2 (5:08 pm on 15 June, 2023):\nTim: Thanks! That picture is from MinaLima. They created all the props for the Harry Potter films, and I love their work. It's like having a piece of the wizarding world at home!", "model_answer": "from MinaLima", "opus_verdict": "incorrect", "opus_failure_mode": "F2", "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
{"instance_id": "locomo_conv-41_q065", "cell": "full-context", "question": "What is John's main focus in local politics?", "ground_truth": "Improving education and infrastructure", "context": "Session 1 (11:01 am on 17 December, 2022):\nJohn: I'm passionate about improving education and infrastructure in our community. Those are my main focuses.", "model_answer": "improving education and infrastructure", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "correct", "gpt_failure_mode": null, "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
{"instance_id": "locomo_conv-50_q086", "cell": "retrieval", "question": "What sports activity is Calvin planning to try after the tour with Frank Ocean?", "ground_truth": "Skiing", "context": "Session 9 (3:15 pm on 21 June, 2023):\nCalvin: Nope, not yet! I haven't been there before, but it's on my to-do list for after my tour with Frank Ocean ends. Can't wait to see them in person! Look at that snowy peak. [image: a photo of a person on skis on a snowy mountain]", "model_answer": "Not mentioned", "opus_verdict": "incorrect", "opus_failure_mode": "F1", "gpt_verdict": "incorrect", "gpt_failure_mode": "F1", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F1"}
{"instance_id": "locomo_conv-26_q059", "cell": "retrieval", "question": "Would Caroline be considered religious?", "ground_truth": "Somewhat, but not extremely religious", "context": "Session 12 (1:50 pm on 17 August, 2023):\nCaroline: Hey Mel! How're ya doin'? Recently, I had a not-so-great experience on a hike. I ran into a group of religious conservatives who said something that really upset me. It made me think how much work we still have to do for LGBTQ rights. It's been so helpful to have people around me who accept and support me, so I know I'll be ok!\n\nSession 14 (1:33 pm on 25 August, 2023):\nCaroline: Thanks! It was made for a local church and shows time changing our lives. I made it to show my own journey as a transgender woman and how we should accept growth and change. [image: a photo of a large stained glass window in a church]", "model_answer": "No.", "opus_verdict": "incorrect", "opus_failure_mode": "F3", "gpt_verdict": "incorrect", "gpt_failure_mode": "F3", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F3"}
{"instance_id": "locomo_conv-44_q000", "cell": "retrieval", "question": "Which year did Audrey adopt the first three of her dogs?", "ground_truth": "2020", "context": "Session 1 (1:10 pm on 27 March, 2023):\nAudrey: I've had them for 3 years! Their names are Pepper, Precious and Panda. I can't live without my little ones!", "model_answer": "Not mentioned.", "opus_verdict": "incorrect", "opus_failure_mode": "F1", "gpt_verdict": "incorrect", "gpt_failure_mode": "F1", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F1"}
{"instance_id": "locomo_conv-41_q080", "cell": "retrieval", "question": "What did Maria make for her home to remind her of a trip to England?", "ground_truth": "painting of a castle on a hill", "context": "Session 8 (6:03 pm on 6 March, 2023):\nMaria: Thanks, John! I got the idea from that trip to England a few years ago - I was mesmerized by the castles. I wanted to bring that same magic to my home, like a reminder of the world's beauty. I made a painting too! [image: a photography of a painting of a castle on a hill]", "model_answer": "A painting.", "opus_verdict": "incorrect", "opus_failure_mode": "F2", "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F2"}
{"instance_id": "locomo_conv-42_q001", "cell": "retrieval", "question": "What kind of interests do Joanna and Nate share?", "ground_truth": "Watching movies, making desserts", "context": "Session 1 (7:31 pm on 21 January, 2022):\nJoanna: Yeah! Besides writing, I also enjoy reading, watching movies, and exploring nature. Anything else you enjoy doing, Nate?\nNate: Playing video games and watching movies are my main hobbies.\nJoanna: Cool, Nate! So we both have similar interests. What type of movies do you like best?\n\nSession 3 (9:27 am on 7 February, 2022):\nNate: Thanks, Joanna. Not much has changed for me, but I just discovered that I can make coconut milk icecream and gave it a try. It was actually pretty good, so I'm proud of myself. [image: a photo of a bowl of ice cream with a spoon in it]\n\nSession 4 (1:07 pm on 25 February, 2022):\nNate: Thanks, Joanna! It means a lot that you enjoy the desserts I bake.\n\nSession 10 (11:54 am on 2 May, 2022):\nJoanna: Not much is new other than the screenplay. Been working on some projects and testing out dairy-free dessert recipes for friends and fam. Here's a pic of a cake I made recently! [image: a photo of a cake with white frosting on a wooden table]\n\nSession 20 (6:03 pm on 5 September, 2022):\nJoanna: Hey Nate! Cute turtles! Bummer about the setback. Any positive vibes comin' your way? I just revised on of my old recipes and made this! [image: a photo of a piece of cake with strawberries and chocolate]", "model_answer": "Movies and nature.", "opus_verdict": "incorrect", "opus_failure_mode": "F3", "gpt_verdict": "incorrect", "gpt_failure_mode": "F3", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F3"}
{"instance_id": "locomo_conv-43_q155", "cell": "retrieval", "question": "How did John feel after being able to jog without pain?", "ground_truth": "It was a huge success.", "context": "Session 24 (3:37 pm on 16 December, 2023):\nJohn: It was great! After being out for so long, jogging without any pain was a huge success. My wife and I hosted a small get-together with friends and family to celebrate.", "model_answer": "Relieved", "opus_verdict": "incorrect", "opus_failure_mode": "F3", "gpt_verdict": "incorrect", "gpt_failure_mode": "F3", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F4"}
{"instance_id": "locomo_conv-50_q015", "cell": "retrieval", "question": "Why does Dave regularly visit parks?", "ground_truth": "because it relaxes and calms him", "context": "Session 1 (11:53 am on 23 March, 2023):\nDave: That's great, can't wait for you to return from your trip. I've been spending lots of time at this beautiful park lately - so calming. Have you been there before? [image: a photography of a park with a lake and a few boats]\n\nSession 8 (2:31 pm on 9 June, 2023):\nDave: Hey, nice photo of the studio! Working in a team yields incredible results, well done! Keep pushing it! I've been doing good - thanks for asking. I've been exploring some parks on the weekends to relax - it's so peaceful being surrounded by nature. Are there any chill spots you enjoy in Boston?", "model_answer": "To relax, recharge, and hang out with friends.", "opus_verdict": "incorrect", "opus_failure_mode": "F4", "gpt_verdict": "incorrect", "gpt_failure_mode": "F4", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F4"}
{"instance_id": "locomo_conv-48_q020", "cell": "retrieval", "question": "Which games have Jolene and her partner played together?", "ground_truth": "Detroit, Walking Dead, Battlefield 1, It Takes Two, Overcooked 2", "context": "Session 2 (9:49 am on 27 January, 2023):\nJolene: They are very unusual pets! Here's me and my partner gaming last week - it's so fun. We played the game \"Detroit\" on the console. We are both crazy about this activity! [image: a photo of a person laying in bed with a dog watching tv]\nJolene: We are planning to play \"Walking Dead\" next Saturday.\n\nSession 15 (7:37 pm on 9 July, 2023):\nJolene: Yeah, we love playing \"It takes two\" together! It's a fun team-strategy game and it's competitive. Plus, it's a great way for us to bond. Do you have any activities you like doing with people?\n\nSession 19 (12:52 am on 19 August, 2023):\nJolene: Oh, I forgot to mention Overcooked 2 - this is a good co-op game if you're into hilarious and chaotic cooking. My partner and I often play for bets! I once won three large pizzas!\n\nSession 20 (9:11 am on 21 August, 2023):\nJolene: Long time no talk! We were given a new game for the console last week, it is Battlefield 1. What's been up with you?", "model_answer": "Detroit, Overcooked 2, It Takes Two", "opus_verdict": "incorrect", "opus_failure_mode": "F2", "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F2"}
{"instance_id": "locomo_conv-48_q055", "cell": "retrieval", "question": "Which pet did Jolene adopt more recently - Susie or Seraphim?", "ground_truth": "Seraphim", "context": "Session 2 (9:49 am on 27 January, 2023):\nJolene: I bought it a year ago in Paris.\nJolene: Even as a child I learned to play on my own.\n\nSession 16 (9:26 am on 1 August, 2023):\nJolene: I adopted her two years ago when I was feeling lonely and wanted some company.", "model_answer": "Seraphim", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "correct", "gpt_failure_mode": null, "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
{"instance_id": "locomo_conv-26_q128", "cell": "retrieval", "question": "What did Caroline find in her neighborhood during her walk?", "ground_truth": "a rainbow sidewalk", "context": "Session 14 (1:33 pm on 25 August, 2023):\nCaroline: I was out walking in my neighborhood when I came across this cool rainbow sidewalk for Pride Month. It was so vibrant and welcoming, I had to take a picture! It reminds us that love and acceptance are everywhere—even where we least expect it.", "model_answer": "A rainbow sidewalk.", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "correct", "gpt_failure_mode": null, "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
{"instance_id": "locomo_conv-50_q010", "cell": "retrieval", "question": "When did Calvin's place get flooded in Tokyo?", "ground_truth": "On a week before 16 May, 2023", "context": "Session 6 (11:50 am on 16 May, 2023):\nCalvin: Hey Dave, not everything has been going smoothly. I had an incident last week where my place got flooded, but thankfully, I managed to save my music gear and favorite microphone. It's been tough, but I'm staying positive and looking forward to getting everything fixed up.", "model_answer": "Last week.", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "correct", "gpt_failure_mode": null, "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F2"}
{"instance_id": "locomo_conv-50_q078", "cell": "retrieval", "question": "What was the necklace Calvin received meant to remind him of?", "ground_truth": "why he keeps hustling as a musician", "context": "Session 4 (6:24 pm on 1 May, 2023):\nCalvin: Thanks, Dave! I got it from another artist as a gift - it's a great reminder of why I keep hustling as a musician!", "model_answer": "Why he keeps hustling as a musician.", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "correct", "gpt_failure_mode": null, "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
{"instance_id": "locomo_conv-47_q092", "cell": "retrieval", "question": "What did John create for the charitable foundation that helped generate reports for analysis?", "ground_truth": "computer application on smartphones", "context": "Session 11 (5:00 pm on 11 May, 2022):\nJohn: Previously, this foundation used paper records and all inventory was recorded manually. I made an application that structured their work, and now everything they need for inventory is in one application on their smartphone.", "model_answer": "A software tool", "opus_verdict": "incorrect", "opus_failure_mode": "F2", "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F2"}
{"instance_id": "locomo_conv-30_q057", "cell": "retrieval", "question": "What advice does Gina give to Jon about running a successful business?", "ground_truth": "build relationships with customers, create a strong brand image, stay positive", "context": "Session 7 (7:28 pm on 23 March, 2023):\nJon: Yeah, brand identity is key. Make sure yours stands out. Also be sure to build relationships with your customers – let them know you care. And don't forget to stay positive and motivate others. Your energy will be contagious!", "model_answer": "Stay passionate, focused, and resilient; believe in yourself, keep going, and stay open to learning.", "opus_verdict": "incorrect", "opus_failure_mode": "F4", "gpt_verdict": "incorrect", "gpt_failure_mode": "F4", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F4"}
{"instance_id": "locomo_conv-30_q029", "cell": "retrieval", "question": "Which cities has Jon visited?", "ground_truth": "Paris, Rome", "context": "Session 2 (2:32 pm on 29 January, 2023):\nJon: Hey Gina! Thanks for asking. I'm on the hunt for the ideal spot for my dance studio and it's been quite a journey! I've been looking at different places and picturing how the space would look. I even found a place with great natural light! Oh, I've been to Paris yesterday! It was sooo cool. [image: a photo of a bathroom with a blue floor and a pink wall]\n\nSession 15 (10:04 am on 19 June, 2023):\nJon: Hey Gina, hope you're doing great! Still working on my biz. Took a short trip last week to Rome to clear my mind a little.", "model_answer": "Paris", "opus_verdict": "incorrect", "opus_failure_mode": "F2", "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F2"}
{"instance_id": "locomo_conv-44_q066", "cell": "retrieval", "question": "What type of dog was Andrew looking to adopt based on his living space?", "ground_truth": "smaller dog", "context": "Session 5 (10:47 am on 6 May, 2023):\nAndrew: Meeting all these adorable pups has been awesome! For those considering getting a pup, the size of living space and the exercise needs of the breed are important. For me, a person living in an apartment, a smaller dog would be best, but if one is active, consider getting one that loves to play and run.", "model_answer": "A smaller dog.", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "correct", "gpt_failure_mode": null, "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
{"instance_id": "locomo_conv-44_q002", "cell": "retrieval", "question": "What kind of indoor activities has Andrew pursued with his girlfriend?", "ground_truth": "boardgames, volunteering at pet shelter, wine tasting, growing flowers", "context": "Session 13 (3:52 pm on 27 July, 2023):\nAndrew: Hey Audrey! How are you? My GF and I just had a great experience volunteering at a pet shelter on Monday - it was so rewarding! We loved spending time with those cute animals and it gave us so much joy. It was so rewarding, it reminded me just how much I love them!\n\nSession 19 (5:53 pm on 24 September, 2023):\nAndrew: Yeah! They really do bring so much into our lives - it's amazing to watch them interact. Here's something I've been taking care of lately. Look at those flowers! [image: a photo of a balcony with a bunch of flowers on it]\n\nSession 23 (4:22 pm on 13 October, 2023):\nAndrew: Hey Audrey, it's been a busy week for me. Last Tuesday, my gf, Toby, and I had a really awesome night playing board games. It was really nice. What's been up with you lately? [image: a photography of a dog sitting at a chess board with a chess set in front of him]\n\nSession 25 (10:14 am on 24 October, 2023):\nAndrew: Hi Audrey! How have you been lately? My girlfriend and I went to this awesome wine tasting last weekend. It was great! We tried so many unique wines and learned a lot. I was surprised at how much I enjoyed it. A reminder to step out of the comfort zone!", "model_answer": "Wine tasting", "opus_verdict": "incorrect", "opus_failure_mode": "F2", "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F2"}
{"instance_id": "locomo_conv-47_q046", "cell": "retrieval", "question": "When did John and James meet at McGee's bar?", "ground_truth": "August 27, 2022", "context": "Session 21 (9:18 pm on 26 August, 2022):\nJohn: Great, then I agree! See you tomorrow at McGee's Pub!", "model_answer": "Tomorrow", "opus_verdict": "incorrect", "opus_failure_mode": "F2", "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F3"}
{"instance_id": "locomo_conv-43_q076", "cell": "retrieval", "question": "What kind of picture did Tim share as part of their Harry Potter book collection?", "ground_truth": "MinaLima's creation from the Harry Potter films", "context": "Session 2 (5:08 pm on 15 June, 2023):\nTim: Thanks! That picture is from MinaLima. They created all the props for the Harry Potter films, and I love their work. It's like having a piece of the wizarding world at home!", "model_answer": "MinaLima props", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "correct", "gpt_failure_mode": null, "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
{"instance_id": "locomo_conv-41_q065", "cell": "retrieval", "question": "What is John's main focus in local politics?", "ground_truth": "Improving education and infrastructure", "context": "Session 1 (11:01 am on 17 December, 2022):\nJohn: I'm passionate about improving education and infrastructure in our community. Those are my main focuses.", "model_answer": "Education and infrastructure", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "correct", "gpt_failure_mode": null, "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
{"instance_id": "locomo_conv-50_q086", "cell": "agentic", "question": "What sports activity is Calvin planning to try after the tour with Frank Ocean?", "ground_truth": "Skiing", "context": "Session 9 (3:15 pm on 21 June, 2023):\nCalvin: Nope, not yet! I haven't been there before, but it's on my to-do list for after my tour with Frank Ocean ends. Can't wait to see them in person! Look at that snowy peak. [image: a photo of a person on skis on a snowy mountain]", "model_answer": "Max tool turns reached (3 turns, 3 tools used).", "opus_verdict": "incorrect", "opus_failure_mode": "F1", "gpt_verdict": "incorrect", "gpt_failure_mode": "F1", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F1"}
{"instance_id": "locomo_conv-26_q059", "cell": "agentic", "question": "Would Caroline be considered religious?", "ground_truth": "Somewhat, but not extremely religious", "context": "Session 12 (1:50 pm on 17 August, 2023):\nCaroline: Hey Mel! How're ya doin'? Recently, I had a not-so-great experience on a hike. I ran into a group of religious conservatives who said something that really upset me. It made me think how much work we still have to do for LGBTQ rights. It's been so helpful to have people around me who accept and support me, so I know I'll be ok!\n\nSession 14 (1:33 pm on 25 August, 2023):\nCaroline: Thanks! It was made for a local church and shows time changing our lives. I made it to show my own journey as a transgender woman and how we should accept growth and change. [image: a photo of a large stained glass window in a church]", "model_answer": "Yes, Caroline has a connection to faith through her heritage (a necklace from Sweden representing \"love, faith and strength\") and by having made a painting for a local church.", "opus_verdict": "incorrect", "opus_failure_mode": "F3", "gpt_verdict": "incorrect", "gpt_failure_mode": "F4", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F4"}
{"instance_id": "locomo_conv-44_q000", "cell": "agentic", "question": "Which year did Audrey adopt the first three of her dogs?", "ground_truth": "2020", "context": "Session 1 (1:10 pm on 27 March, 2023):\nAudrey: I've had them for 3 years! Their names are Pepper, Precious and Panda. I can't live without my little ones!", "model_answer": "Max tool turns reached (3 turns, 3 tools used).", "opus_verdict": "incorrect", "opus_failure_mode": "F1", "gpt_verdict": "incorrect", "gpt_failure_mode": "F1", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F1"}
{"instance_id": "locomo_conv-41_q080", "cell": "agentic", "question": "What did Maria make for her home to remind her of a trip to England?", "ground_truth": "painting of a castle on a hill", "context": "Session 8 (6:03 pm on 6 March, 2023):\nMaria: Thanks, John! I got the idea from that trip to England a few years ago - I was mesmerized by the castles. I wanted to bring that same magic to my home, like a reminder of the world's beauty. I made a painting too! [image: a photography of a painting of a castle on a hill]", "model_answer": "a painting", "opus_verdict": "incorrect", "opus_failure_mode": "F2", "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F2"}
{"instance_id": "locomo_conv-42_q001", "cell": "agentic", "question": "What kind of interests do Joanna and Nate share?", "ground_truth": "Watching movies, making desserts", "context": "Session 1 (7:31 pm on 21 January, 2022):\nJoanna: Yeah! Besides writing, I also enjoy reading, watching movies, and exploring nature. Anything else you enjoy doing, Nate?\nNate: Playing video games and watching movies are my main hobbies.\nJoanna: Cool, Nate! So we both have similar interests. What type of movies do you like best?\n\nSession 3 (9:27 am on 7 February, 2022):\nNate: Thanks, Joanna. Not much has changed for me, but I just discovered that I can make coconut milk icecream and gave it a try. It was actually pretty good, so I'm proud of myself. [image: a photo of a bowl of ice cream with a spoon in it]\n\nSession 4 (1:07 pm on 25 February, 2022):\nNate: Thanks, Joanna! It means a lot that you enjoy the desserts I bake.\n\nSession 10 (11:54 am on 2 May, 2022):\nJoanna: Not much is new other than the screenplay. Been working on some projects and testing out dairy-free dessert recipes for friends and fam. Here's a pic of a cake I made recently! [image: a photo of a cake with white frosting on a wooden table]\n\nSession 20 (6:03 pm on 5 September, 2022):\nJoanna: Hey Nate! Cute turtles! Bummer about the setback. Any positive vibes comin' your way? I just revised on of my old recipes and made this! [image: a photo of a piece of cake with strawberries and chocolate]", "model_answer": "movies", "opus_verdict": "incorrect", "opus_failure_mode": "F2", "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F2"}
{"instance_id": "locomo_conv-43_q155", "cell": "agentic", "question": "How did John feel after being able to jog without pain?", "ground_truth": "It was a huge success.", "context": "Session 24 (3:37 pm on 16 December, 2023):\nJohn: It was great! After being out for so long, jogging without any pain was a huge success. My wife and I hosted a small get-together with friends and family to celebrate.", "model_answer": "great", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
{"instance_id": "locomo_conv-50_q015", "cell": "agentic", "question": "Why does Dave regularly visit parks?", "ground_truth": "because it relaxes and calms him", "context": "Session 1 (11:53 am on 23 March, 2023):\nDave: That's great, can't wait for you to return from your trip. I've been spending lots of time at this beautiful park lately - so calming. Have you been there before? [image: a photography of a park with a lake and a few boats]\n\nSession 8 (2:31 pm on 9 June, 2023):\nDave: Hey, nice photo of the studio! Working in a team yields incredible results, well done! Keep pushing it! I've been doing good - thanks for asking. I've been exploring some parks on the weekends to relax - it's so peaceful being surrounded by nature. Are there any chill spots you enjoy in Boston?", "model_answer": "to relax and recharge", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
{"instance_id": "locomo_conv-48_q020", "cell": "agentic", "question": "Which games have Jolene and her partner played together?", "ground_truth": "Detroit, Walking Dead, Battlefield 1, It Takes Two, Overcooked 2", "context": "Session 2 (9:49 am on 27 January, 2023):\nJolene: They are very unusual pets! Here's me and my partner gaming last week - it's so fun. We played the game \"Detroit\" on the console. We are both crazy about this activity! [image: a photo of a person laying in bed with a dog watching tv]\nJolene: We are planning to play \"Walking Dead\" next Saturday.\n\nSession 15 (7:37 pm on 9 July, 2023):\nJolene: Yeah, we love playing \"It takes two\" together! It's a fun team-strategy game and it's competitive. Plus, it's a great way for us to bond. Do you have any activities you like doing with people?\n\nSession 19 (12:52 am on 19 August, 2023):\nJolene: Oh, I forgot to mention Overcooked 2 - this is a good co-op game if you're into hilarious and chaotic cooking. My partner and I often play for bets! I once won three large pizzas!\n\nSession 20 (9:11 am on 21 August, 2023):\nJolene: Long time no talk! We were given a new game for the console last week, it is Battlefield 1. What's been up with you?", "model_answer": "Detroit, Overcooked 2, It Takes Two", "opus_verdict": "incorrect", "opus_failure_mode": "F2", "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F2"}
{"instance_id": "locomo_conv-48_q055", "cell": "agentic", "question": "Which pet did Jolene adopt more recently - Susie or Seraphim?", "ground_truth": "Seraphim", "context": "Session 2 (9:49 am on 27 January, 2023):\nJolene: I bought it a year ago in Paris.\nJolene: Even as a child I learned to play on my own.\n\nSession 16 (9:26 am on 1 August, 2023):\nJolene: I adopted her two years ago when I was feeling lonely and wanted some company.", "model_answer": "Susie", "opus_verdict": "incorrect", "opus_failure_mode": "F3", "gpt_verdict": "incorrect", "gpt_failure_mode": "F3", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F3"}
{"instance_id": "locomo_conv-26_q128", "cell": "agentic", "question": "What did Caroline find in her neighborhood during her walk?", "ground_truth": "a rainbow sidewalk", "context": "Session 14 (1:33 pm on 25 August, 2023):\nCaroline: I was out walking in my neighborhood when I came across this cool rainbow sidewalk for Pride Month. It was so vibrant and welcoming, I had to take a picture! It reminds us that love and acceptance are everywhere—even where we least expect it.", "model_answer": "a rainbow sidewalk", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "correct", "gpt_failure_mode": null, "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
{"instance_id": "locomo_conv-50_q010", "cell": "agentic", "question": "When did Calvin's place get flooded in Tokyo?", "ground_truth": "On a week before 16 May, 2023", "context": "Session 6 (11:50 am on 16 May, 2023):\nCalvin: Hey Dave, not everything has been going smoothly. I had an incident last week where my place got flooded, but thankfully, I managed to save my music gear and favorite microphone. It's been tough, but I'm staying positive and looking forward to getting everything fixed up.", "model_answer": "last week", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "correct", "gpt_failure_mode": null, "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F2"}
{"instance_id": "locomo_conv-50_q078", "cell": "agentic", "question": "What was the necklace Calvin received meant to remind him of?", "ground_truth": "why he keeps hustling as a musician", "context": "Session 4 (6:24 pm on 1 May, 2023):\nCalvin: Thanks, Dave! I got it from another artist as a gift - it's a great reminder of why I keep hustling as a musician!", "model_answer": "why he keeps hustling as a musician", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "correct", "gpt_failure_mode": null, "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
{"instance_id": "locomo_conv-47_q092", "cell": "agentic", "question": "What did John create for the charitable foundation that helped generate reports for analysis?", "ground_truth": "computer application on smartphones", "context": "Session 11 (5:00 pm on 11 May, 2022):\nJohn: Previously, this foundation used paper records and all inventory was recorded manually. I made an application that structured their work, and now everything they need for inventory is in one application on their smartphone.", "model_answer": "a software tool", "opus_verdict": "incorrect", "opus_failure_mode": "F2", "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F2"}
{"instance_id": "locomo_conv-30_q057", "cell": "agentic", "question": "What advice does Gina give to Jon about running a successful business?", "ground_truth": "build relationships with customers, create a strong brand image, stay positive", "context": "Session 7 (7:28 pm on 23 March, 2023):\nJon: Yeah, brand identity is key. Make sure yours stands out. Also be sure to build relationships with your customers – let them know you care. And don't forget to stay positive and motivate others. Your energy will be contagious!", "model_answer": "Focus on why you started, have faith in yourself, remind yourself of successes and progress, and have a good support system", "opus_verdict": "incorrect", "opus_failure_mode": "F3", "gpt_verdict": "incorrect", "gpt_failure_mode": "F5", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F4"}
{"instance_id": "locomo_conv-30_q029", "cell": "agentic", "question": "Which cities has Jon visited?", "ground_truth": "Paris, Rome", "context": "Session 2 (2:32 pm on 29 January, 2023):\nJon: Hey Gina! Thanks for asking. I'm on the hunt for the ideal spot for my dance studio and it's been quite a journey! I've been looking at different places and picturing how the space would look. I even found a place with great natural light! Oh, I've been to Paris yesterday! It was sooo cool. [image: a photo of a bathroom with a blue floor and a pink wall]\n\nSession 15 (10:04 am on 19 June, 2023):\nJon: Hey Gina, hope you're doing great! Still working on my biz. Took a short trip last week to Rome to clear my mind a little.", "model_answer": "unknown", "opus_verdict": "incorrect", "opus_failure_mode": "F1", "gpt_verdict": "incorrect", "gpt_failure_mode": "F1", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F1"}
{"instance_id": "locomo_conv-44_q066", "cell": "agentic", "question": "What type of dog was Andrew looking to adopt based on his living space?", "ground_truth": "smaller dog", "context": "Session 5 (10:47 am on 6 May, 2023):\nAndrew: Meeting all these adorable pups has been awesome! For those considering getting a pup, the size of living space and the exercise needs of the breed are important. For me, a person living in an apartment, a smaller dog would be best, but if one is active, consider getting one that loves to play and run.", "model_answer": "a smaller dog", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "correct", "gpt_failure_mode": null, "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
{"instance_id": "locomo_conv-44_q002", "cell": "agentic", "question": "What kind of indoor activities has Andrew pursued with his girlfriend?", "ground_truth": "boardgames, volunteering at pet shelter, wine tasting, growing flowers", "context": "Session 13 (3:52 pm on 27 July, 2023):\nAndrew: Hey Audrey! How are you? My GF and I just had a great experience volunteering at a pet shelter on Monday - it was so rewarding! We loved spending time with those cute animals and it gave us so much joy. It was so rewarding, it reminded me just how much I love them!\n\nSession 19 (5:53 pm on 24 September, 2023):\nAndrew: Yeah! They really do bring so much into our lives - it's amazing to watch them interact. Here's something I've been taking care of lately. Look at those flowers! [image: a photo of a balcony with a bunch of flowers on it]\n\nSession 23 (4:22 pm on 13 October, 2023):\nAndrew: Hey Audrey, it's been a busy week for me. Last Tuesday, my gf, Toby, and I had a really awesome night playing board games. It was really nice. What's been up with you lately? [image: a photography of a dog sitting at a chess board with a chess set in front of him]\n\nSession 25 (10:14 am on 24 October, 2023):\nAndrew: Hi Audrey! How have you been lately? My girlfriend and I went to this awesome wine tasting last weekend. It was great! We tried so many unique wines and learned a lot. I was surprised at how much I enjoyed it. A reminder to step out of the comfort zone!", "model_answer": "board games", "opus_verdict": "incorrect", "opus_failure_mode": "F2", "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F2"}
{"instance_id": "locomo_conv-47_q046", "cell": "agentic", "question": "When did John and James meet at McGee's bar?", "ground_truth": "August 27, 2022", "context": "Session 21 (9:18 pm on 26 August, 2022):\nJohn: Great, then I agree! See you tomorrow at McGee's Pub!", "model_answer": "Tomorrow", "opus_verdict": "incorrect", "opus_failure_mode": "F2", "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F3"}
{"instance_id": "locomo_conv-43_q076", "cell": "agentic", "question": "What kind of picture did Tim share as part of their Harry Potter book collection?", "ground_truth": "MinaLima's creation from the Harry Potter films", "context": "Session 2 (5:08 pm on 15 June, 2023):\nTim: Thanks! That picture is from MinaLima. They created all the props for the Harry Potter films, and I love their work. It's like having a piece of the wizarding world at home!", "model_answer": "A picture from MinaLima", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "correct", "gpt_failure_mode": null, "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
{"instance_id": "locomo_conv-41_q065", "cell": "agentic", "question": "What is John's main focus in local politics?", "ground_truth": "Improving education and infrastructure", "context": "Session 1 (11:01 am on 17 December, 2022):\nJohn: I'm passionate about improving education and infrastructure in our community. Those are my main focuses.", "model_answer": "Education and infrastructure", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "correct", "gpt_failure_mode": null, "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}

View File

@@ -1,91 +0,0 @@
# Manifest v6 κ Re-Calibration Analysis
**Date:** 2026-04-24 **Parent:** `38a830e` (v6 Phase 1 Commit 2) **v6 anchor:** `60d061e`
**Sample:** 100 instances from `benchmarks/results/locomo-mini-n20-retry-2026-04-24T00-02-12Z.jsonl` (authoritative v5 κ set; zero new Opus/GPT calls).
**MiniMax verdicts:** 100 calls via OpenRouter `minimax/minimax-m2.7` (v6 alias: `minimax-m27-via-openrouter`); direct HTTP probe (LiteLLM proxy not in loop — isolates model behavior from middleware).
**Prompt:** verbatim `failure-mode-judge.ts:245-258` (same as §1.3g / §1.3h / §1.3h-C).
**Parameters:** `temperature=0.0`, `max_tokens=4096`.
---
## §1 Three pairwise Cohen's κ
| Pair | n | Agree | Raw % | κ |
|------|---|-------|-------|-----|
| Opus vs GPT | 100 | 93 | 93.00% | **0.8480** |
| Opus vs MiniMax | 100 | 93 | 93.00% | **0.8549** |
| GPT vs MiniMax | 100 | 90 | 90.00% | **0.7878** |
**Conservative trio κ = min = 0.7878**
## §2 Verdict: **PASS**
Per v6 §5.4 gate criteria:
- `κ_trio ≥ 0.70` → PASS, halt with PM-RATIFY-V6-KAPPA
- `0.60 ≤ κ_trio < 0.70` → BORDERLINE, halt with PM adjudication
- `κ_trio < 0.60` → FAIL, halt with swap-path-re-evaluation
---
## §3 Confusion matrices
### Opus vs GPT
| | GPT=correct | GPT=incorrect |
|---|---|---|
| **Opus=correct** | 32 | 7 |
| **Opus=incorrect** | 0 | 61 |
### Opus vs MiniMax
| | MiniMax=correct | MiniMax=incorrect |
|---|---|---|
| **Opus=correct** | 37 | 2 |
| **Opus=incorrect** | 5 | 56 |
### GPT vs MiniMax
| | MiniMax=correct | MiniMax=incorrect |
|---|---|---|
| **GPT=correct** | 32 | 0 |
| **GPT=incorrect** | 10 | 58 |
---
## §4 Per-cell κ breakdown (n=20 per cell)
| Cell | n | MiniMax parsed | κ(Opus,GPT) | κ(Opus,MiniMax) | κ(GPT,MiniMax) |
|------|---|-----------------|----------------|-------------------|------------------|
| no-context | 20 | 20 | 1.0000 | 1.0000 | 1.0000 |
| oracle-context | 20 | 20 | 0.7059 | 0.7917 | 0.7059 |
| full-context | 20 | 20 | 0.8000 | 0.7000 | 0.7059 |
| retrieval | 20 | 20 | 1.0000 | 0.8936 | 0.8936 |
| agentic | 20 | 20 | 0.7826 | 0.8980 | 0.6875 |
---
## §5 MiniMax operational metrics
- Calls: 100 total, parsed OK: **100/100 (100.0%)**
- Routing errors (non-200 HTTP): **0/100** (0.0%)
- Total retries: 0
- Latency p50: **11.9 s** | p95: **31.4 s**
- Token usage: prompt = 53,855, completion = 48,920
- Cost actual (OR MiniMax pricing $0.30/$1.20 per 1M): **~$0.0749**
Per brief §3.5 operational hedge thresholds:
- parse ≥95/100 target: **MET** — actual 100/100
- parse ≥90/100 halt: **MET** — actual 100/100
- latency p50 ≤25s: **MET** — actual 11.9s
- OR routing errors <5%: **MET** — actual 0.0%
---
## §6 Comparison to v5 historical baseline
v5 κ baseline reference: Fleiss' κ=0.7458 on three-way Opus+GPT+Gemini ensemble.
v6 κ(Opus, GPT) pairwise: **0.8480** — sanity check. If significantly different from v5 baseline range (~0.74-0.82 for a high-agreement pair), investigate.
v6 conservative trio κ (Opus+GPT+MiniMax): **0.7878**.

View File

@@ -1,330 +0,0 @@
"""
Manifest v6 Phase 1 — κ re-calibration computation
===================================================
Computes three pairwise Cohen's κ on 100-instance sample:
κ(Opus, GPT) — should match v5 historical baseline ~0.74-0.82
κ(Opus, MiniMax) — new measurement
κ(GPT, MiniMax) — new measurement
Conservative trio κ = min of the three.
Also reports:
- Raw agreement % per pair
- Confusion matrix per pair
- Per-cell breakdown (no-context / oracle-context / full-context /
retrieval / agentic)
- MiniMax operational metrics: parse rate, latency p50/p95, routing
errors, token usage
Writes:
kappa-v6-analysis.md — detailed matrix + per-cell breakdown
_summary-v6-kappa.json — machine-readable for halt ping
"""
from __future__ import annotations
import json
import statistics
from collections import Counter
from pathlib import Path
OUT_DIR = Path("D:/Projects/waggle-os/benchmarks/calibration/v6-kappa-recal")
SAMPLE_PATH = OUT_DIR / "kappa-sample-instances.jsonl"
RESPONSES_PATH = OUT_DIR / "minimax-kappa-responses.jsonl"
ANALYSIS_PATH = OUT_DIR / "kappa-v6-analysis.md"
SUMMARY_JSON = OUT_DIR / "_summary-v6-kappa.json"
def load_jsonl(path: Path) -> list[dict]:
out = []
with path.open("r", encoding="utf-8") as f:
for line in f:
line = line.strip()
if line:
out.append(json.loads(line))
return out
def cohen_kappa(pairs: list[tuple[str, str]]) -> tuple[float, dict]:
"""Cohen's κ on 2-class (correct/incorrect) pairs. Returns (κ, detail)."""
if not pairs:
return (float("nan"), {"n": 0, "agree": 0, "po": 0.0, "pe": 0.0})
n = len(pairs)
agree = sum(1 for a, b in pairs if a == b)
po = agree / n
a_counts = {"correct": 0, "incorrect": 0}
b_counts = {"correct": 0, "incorrect": 0}
for a, b in pairs:
a_counts[a] = a_counts.get(a, 0) + 1
b_counts[b] = b_counts.get(b, 0) + 1
pe = sum(
(a_counts.get(v, 0) / n) * (b_counts.get(v, 0) / n)
for v in ("correct", "incorrect")
)
if pe >= 1.0:
return (1.0 if po == 1.0 else float("nan"),
{"n": n, "agree": agree, "po": po, "pe": pe,
"a_counts": a_counts, "b_counts": b_counts})
kappa = (po - pe) / (1.0 - pe)
return (kappa, {"n": n, "agree": agree, "po": po, "pe": pe,
"a_counts": a_counts, "b_counts": b_counts})
def confusion_matrix(pairs: list[tuple[str, str]]) -> dict:
"""2x2 confusion (rows = judge A, cols = judge B)."""
cm = {"correct_correct": 0, "correct_incorrect": 0,
"incorrect_correct": 0, "incorrect_incorrect": 0}
for a, b in pairs:
key = f"{a}_{b}"
cm[key] = cm.get(key, 0) + 1
return cm
def classify_verdict(trio_kappa: float) -> str:
if trio_kappa != trio_kappa: # NaN
return "INCONCLUSIVE"
if trio_kappa >= 0.70:
return "PASS"
if trio_kappa >= 0.60:
return "BORDERLINE"
return "FAIL"
def fmt_k(x: float) -> str:
if x != x:
return "NaN"
return f"{x:.4f}"
def pct_str(num: int, denom: int) -> str:
if denom == 0:
return "—"
return f"{num}/{denom} ({num * 100.0 / denom:.1f}%)"
def main() -> int:
sample = load_jsonl(SAMPLE_PATH)
mm = load_jsonl(RESPONSES_PATH)
# Index MiniMax responses
mm_by_key = {(r["instance_id"], r["cell"]): r for r in mm}
# Build pair lists for 3 pairwise κ
pairs_og = [] # (Opus, GPT)
pairs_om = [] # (Opus, MiniMax)
pairs_gm = [] # (GPT, MiniMax)
per_cell = {"no-context": [], "oracle-context": [], "full-context": [],
"retrieval": [], "agentic": []}
mm_parse_ok = 0
mm_lat = []
mm_retries = 0
mm_routing_errors = 0
mm_prompt_tok = []
mm_comp_tok = []
for s in sample:
op_v = s.get("opus_verdict")
gp_v = s.get("gpt_verdict")
if op_v not in ("correct", "incorrect") or gp_v not in ("correct", "incorrect"):
continue
pairs_og.append((op_v, gp_v))
per_cell.setdefault(s["cell"], []).append(("og", op_v, gp_v))
key = (s["instance_id"], s["cell"])
m = mm_by_key.get(key)
if m is None:
continue
mm_v = m.get("parsed_verdict")
mm_lat.append(m.get("latency_ms") or 0)
mm_retries += m.get("retries") or 0
if m.get("http_status") != 200:
mm_routing_errors += 1
if m.get("prompt_tokens"):
mm_prompt_tok.append(m["prompt_tokens"])
if m.get("completion_tokens"):
mm_comp_tok.append(m["completion_tokens"])
if mm_v in ("correct", "incorrect"):
mm_parse_ok += 1
pairs_om.append((op_v, mm_v))
pairs_gm.append((gp_v, mm_v))
per_cell.setdefault(s["cell"], []).append(("om", op_v, mm_v))
per_cell.setdefault(s["cell"], []).append(("gm", gp_v, mm_v))
k_og, k_og_det = cohen_kappa(pairs_og)
k_om, k_om_det = cohen_kappa(pairs_om)
k_gm, k_gm_det = cohen_kappa(pairs_gm)
cm_og = confusion_matrix(pairs_og)
cm_om = confusion_matrix(pairs_om)
cm_gm = confusion_matrix(pairs_gm)
kappas = [k for k in (k_og, k_om, k_gm) if k == k]
k_trio = min(kappas) if kappas else float("nan")
verdict = classify_verdict(k_trio)
# Per-cell pairwise
per_cell_rows = []
cells_order = ["no-context", "oracle-context", "full-context", "retrieval", "agentic"]
for cell in cells_order:
triples = per_cell.get(cell, [])
pog = [(a, b) for t, a, b in triples if t == "og"]
pom = [(a, b) for t, a, b in triples if t == "om"]
pgm = [(a, b) for t, a, b in triples if t == "gm"]
k_c_og, _ = cohen_kappa(pog) if pog else (float("nan"), {})
k_c_om, _ = cohen_kappa(pom) if pom else (float("nan"), {})
k_c_gm, _ = cohen_kappa(pgm) if pgm else (float("nan"), {})
n_cell = len(pog)
mm_cell_parsed = len(pom)
per_cell_rows.append({
"cell": cell,
"n": n_cell,
"mm_parsed": mm_cell_parsed,
"k_og": k_c_og,
"k_om": k_c_om,
"k_gm": k_c_gm,
})
# Operational metrics
lat_p50 = int(statistics.median(mm_lat)) if mm_lat else 0
lat_p95 = int(sorted(mm_lat)[max(0, int(len(mm_lat) * 0.95) - 1)]) if mm_lat else 0
prompt_tok_total = sum(mm_prompt_tok)
comp_tok_total = sum(mm_comp_tok)
# OR MiniMax M2.7 pricing: $0.30/M prompt, $1.20/M completion
cost_actual = round(
(prompt_tok_total / 1_000_000) * 0.30 + (comp_tok_total / 1_000_000) * 1.20, 4
)
# ── Write kappa-v6-analysis.md ─────────────────────────────────────
lines = []
lines.append("# Manifest v6 κ Re-Calibration Analysis")
lines.append("")
lines.append("**Date:** 2026-04-24 **Parent:** `38a830e` (v6 Phase 1 Commit 2) **v6 anchor:** `60d061e`")
lines.append("")
lines.append(f"**Sample:** {len(sample)} instances from `benchmarks/results/locomo-mini-n20-retry-2026-04-24T00-02-12Z.jsonl` (authoritative v5 κ set; zero new Opus/GPT calls).")
lines.append("")
lines.append(f"**MiniMax verdicts:** {len(mm)} calls via OpenRouter `minimax/minimax-m2.7` (v6 alias: `minimax-m27-via-openrouter`); direct HTTP probe (LiteLLM proxy not in loop — isolates model behavior from middleware).")
lines.append("")
lines.append(f"**Prompt:** verbatim `failure-mode-judge.ts:245-258` (same as §1.3g / §1.3h / §1.3h-C).")
lines.append(f"**Parameters:** `temperature=0.0`, `max_tokens=4096`.")
lines.append("")
lines.append("---")
lines.append("")
lines.append("## §1 Three pairwise Cohen's κ")
lines.append("")
lines.append("| Pair | n | Agree | Raw % | κ |")
lines.append("|------|---|-------|-------|-----|")
lines.append(f"| Opus vs GPT | {k_og_det['n']} | {k_og_det['agree']} | {k_og_det['po']*100:.2f}% | **{fmt_k(k_og)}** |")
lines.append(f"| Opus vs MiniMax | {k_om_det['n']} | {k_om_det['agree']} | {k_om_det['po']*100:.2f}% | **{fmt_k(k_om)}** |")
lines.append(f"| GPT vs MiniMax | {k_gm_det['n']} | {k_gm_det['agree']} | {k_gm_det['po']*100:.2f}% | **{fmt_k(k_gm)}** |")
lines.append("")
lines.append(f"**Conservative trio κ = min = {fmt_k(k_trio)}**")
lines.append("")
lines.append(f"## §2 Verdict: **{verdict}**")
lines.append("")
lines.append("Per v6 §5.4 gate criteria:")
lines.append("- `κ_trio ≥ 0.70` → PASS, halt with PM-RATIFY-V6-KAPPA")
lines.append("- `0.60 ≤ κ_trio < 0.70` → BORDERLINE, halt with PM adjudication")
lines.append("- `κ_trio < 0.60` → FAIL, halt with swap-path-re-evaluation")
lines.append("")
lines.append("---")
lines.append("")
lines.append("## §3 Confusion matrices")
lines.append("")
lines.append("### Opus vs GPT")
lines.append("")
lines.append("| | GPT=correct | GPT=incorrect |")
lines.append("|---|---|---|")
lines.append(f"| **Opus=correct** | {cm_og.get('correct_correct', 0)} | {cm_og.get('correct_incorrect', 0)} |")
lines.append(f"| **Opus=incorrect** | {cm_og.get('incorrect_correct', 0)} | {cm_og.get('incorrect_incorrect', 0)} |")
lines.append("")
lines.append("### Opus vs MiniMax")
lines.append("")
lines.append("| | MiniMax=correct | MiniMax=incorrect |")
lines.append("|---|---|---|")
lines.append(f"| **Opus=correct** | {cm_om.get('correct_correct', 0)} | {cm_om.get('correct_incorrect', 0)} |")
lines.append(f"| **Opus=incorrect** | {cm_om.get('incorrect_correct', 0)} | {cm_om.get('incorrect_incorrect', 0)} |")
lines.append("")
lines.append("### GPT vs MiniMax")
lines.append("")
lines.append("| | MiniMax=correct | MiniMax=incorrect |")
lines.append("|---|---|---|")
lines.append(f"| **GPT=correct** | {cm_gm.get('correct_correct', 0)} | {cm_gm.get('correct_incorrect', 0)} |")
lines.append(f"| **GPT=incorrect** | {cm_gm.get('incorrect_correct', 0)} | {cm_gm.get('incorrect_incorrect', 0)} |")
lines.append("")
lines.append("---")
lines.append("")
lines.append("## §4 Per-cell κ breakdown (n=20 per cell)")
lines.append("")
lines.append("| Cell | n | MiniMax parsed | κ(Opus,GPT) | κ(Opus,MiniMax) | κ(GPT,MiniMax) |")
lines.append("|------|---|-----------------|----------------|-------------------|------------------|")
for row in per_cell_rows:
lines.append(
f"| {row['cell']} | {row['n']} | {row['mm_parsed']} "
f"| {fmt_k(row['k_og'])} | {fmt_k(row['k_om'])} | {fmt_k(row['k_gm'])} |"
)
lines.append("")
lines.append("---")
lines.append("")
lines.append("## §5 MiniMax operational metrics")
lines.append("")
lines.append(f"- Calls: {len(mm)} total, parsed OK: **{pct_str(mm_parse_ok, len(mm))}**")
lines.append(f"- Routing errors (non-200 HTTP): **{mm_routing_errors}/{len(mm)}** ({mm_routing_errors*100/len(mm):.1f}%)")
lines.append(f"- Total retries: {mm_retries}")
lines.append(f"- Latency p50: **{lat_p50/1000:.1f} s** | p95: **{lat_p95/1000:.1f} s**")
lines.append(f"- Token usage: prompt = {prompt_tok_total:,}, completion = {comp_tok_total:,}")
lines.append(f"- Cost actual (OR MiniMax pricing $0.30/$1.20 per 1M): **~${cost_actual}**")
lines.append("")
lines.append(f"Per brief §3.5 operational hedge thresholds:")
lines.append(f"- parse ≥95/100 target: **{'MET' if mm_parse_ok >= 95 else 'MISS (below target)'}** — actual {mm_parse_ok}/100")
lines.append(f"- parse ≥90/100 halt: **{'MET' if mm_parse_ok >= 90 else 'FAIL (halt)'}** — actual {mm_parse_ok}/100")
lines.append(f"- latency p50 ≤25s: **{'MET' if lat_p50 <= 25000 else 'MISS'}** — actual {lat_p50/1000:.1f}s")
lines.append(f"- OR routing errors <5%: **{'MET' if mm_routing_errors < 5 else 'FLAG'}** — actual {mm_routing_errors/len(mm)*100:.1f}%")
lines.append("")
lines.append("---")
lines.append("")
lines.append(f"## §6 Comparison to v5 historical baseline")
lines.append("")
lines.append(f"v5 κ baseline reference: Fleiss' κ=0.7458 on three-way Opus+GPT+Gemini ensemble.")
lines.append(f"v6 κ(Opus, GPT) pairwise: **{fmt_k(k_og)}** — sanity check. If significantly different from v5 baseline range (~0.74-0.82 for a high-agreement pair), investigate.")
lines.append(f"v6 conservative trio κ (Opus+GPT+MiniMax): **{fmt_k(k_trio)}**.")
ANALYSIS_PATH.write_text("\n".join(lines), encoding="utf-8")
print(f"Wrote {ANALYSIS_PATH}")
# Machine-readable summary
summary = {
"verdict": verdict,
"k_opus_gpt": k_og,
"k_opus_minimax": k_om,
"k_gpt_minimax": k_gm,
"k_conservative_trio": k_trio,
"minimax_parse_success": mm_parse_ok,
"minimax_n_total": len(mm),
"minimax_lat_p50_ms": lat_p50,
"minimax_lat_p95_ms": lat_p95,
"minimax_routing_errors": mm_routing_errors,
"minimax_retries_total": mm_retries,
"minimax_prompt_tokens_total": prompt_tok_total,
"minimax_completion_tokens_total": comp_tok_total,
"cost_actual_usd": cost_actual,
"per_cell": per_cell_rows,
"confusion_opus_gpt": cm_og,
"confusion_opus_minimax": cm_om,
"confusion_gpt_minimax": cm_gm,
}
SUMMARY_JSON.write_text(json.dumps(summary, indent=2, default=str), encoding="utf-8")
print(f"Wrote {SUMMARY_JSON}")
print(f"\nVerdict: {verdict}")
print(f"k(Opus, GPT) = {fmt_k(k_og)}")
print(f"k(Opus, MiniMax) = {fmt_k(k_om)}")
print(f"k(GPT, MiniMax) = {fmt_k(k_gm)}")
print(f"k_trio (min) = {fmt_k(k_trio)}")
print(f"MiniMax parse = {mm_parse_ok}/100")
print(f"Cost = ${cost_actual}")
return 0
if __name__ == "__main__":
import sys
sys.exit(main())

View File

@@ -1,367 +0,0 @@
"""
Manifest v6 Phase 1 Commit 3 — MiniMax κ re-calibration probe
==============================================================
Executes 100 MiniMax M2.7 verdicts (via OpenRouter) on the authoritative
v5 κ calibration set:
benchmarks/results/locomo-mini-n20-retry-2026-04-24T00-02-12Z.jsonl
Reuses existing Opus + GPT verdicts from judge_ensemble field (zero new
calls for those). Reuses LoCoMo fixtures for question/ground_truth/
context lookup by instance_id (same pattern as §1.3g/h probes).
Routing: same OR endpoint as §1.3h (direct HTTP, bypasses LiteLLM proxy
for probe speed). The v6 LiteLLM alias wiring (minimax-m27-via-openrouter)
will be validated end-to-end in Phase 2 N=400 execution; κ re-cal
isolates model behavior from middleware.
Prompt: verbatim from failure-mode-judge.ts:245-258 (identical to §1.3g
and §1.3h probes).
Operational hedge per brief §3.5:
- Log parse rate (target ≥95/100, halt <90/100)
- Log latency p50 (target ≤25s) + p95
- Log OR routing errors (>5% rate raises PM flag pre-κ compute)
Scope guards:
- Parent HEAD = 38a830e (v6 Phase 1 Commit 2 anchor)
- v6 manifest anchor = 60d061e (Commit 1)
- §11 frozen paths except litellm-config.yaml (already amended in Commit 2)
- No runner/judge-runner/failure-mode-judge edits
Budget: ~$2.50 expected (100 calls × $0.02 avg per §1.3h MiniMax pricing).
Cap: $30 Phase 1 total.
Usage:
python minimax-kappa-probe.py
"""
from __future__ import annotations
import json
import sys
import time
import urllib.error
import urllib.request
from datetime import datetime, timezone
from pathlib import Path
try:
sys.stdout.reconfigure(encoding="utf-8", errors="replace")
sys.stderr.reconfigure(encoding="utf-8", errors="replace")
except Exception:
pass
# ── Paths ───────────────────────────────────────────────────────────────
OUT_DIR = Path("D:/Projects/waggle-os/benchmarks/calibration/v6-kappa-recal")
CAL_SRC = Path("D:/Projects/waggle-os/benchmarks/results/locomo-mini-n20-retry-2026-04-24T00-02-12Z.jsonl")
LOCOMO = Path("D:/Projects/waggle-os/benchmarks/data/locomo/locomo-1540.jsonl")
RESPONSES_PATH = OUT_DIR / "minimax-kappa-responses.jsonl"
SAMPLE_PATH = OUT_DIR / "kappa-sample-instances.jsonl"
# ── Verbatim judge prompt (same as §1.3g and §1.3h) ─────────────────────
JUDGE_PROMPT_TEMPLATE = "\n".join([
"You are evaluating whether an LLM's answer is correct against ground truth.",
"",
"## Question",
"{question}",
"",
"## Ground-truth answer",
"{ground_truth}",
"",
"## Ground-truth supporting context (excerpt shown to the model)",
"{context}",
"",
"## Model's answer",
"{model_answer}",
"",
"## Your task",
"",
"Step 1: Determine if the model's answer is correct.",
"- \"correct\" means the model's answer contains all required facts from ground truth, with no additional incorrect claims.",
"- Minor phrasing differences, synonyms, or alternative but equivalent formulations are acceptable.",
"- Extra detail is acceptable ONLY if it is factually correct.",
"",
"Step 2: If incorrect, assign exactly one failure mode using this decision tree:",
"",
"1. Does the model explicitly refuse or say it does not know? -> F1 (ABSTAIN)",
"2. Does the model answer a DIFFERENT question than was asked (coherent but off-topic)? -> F5 (OFF-TOPIC)",
"3. Does the model rely on entities, names, dates, or claims that do NOT appear in the ground-truth context (fabrication)? -> F4 (HALLUCINATED)",
"4. Does the model correctly state SOME required facts but miss others, without stating any incorrect facts? -> F2 (PARTIAL)",
"5. Otherwise (model states facts derived from the context but gets them wrong): -> F3 (INCORRECT)",
"",
"Step 3: Return JSON only, no prose, in this exact schema:",
"",
"{{",
" \"verdict\": \"correct\" | \"incorrect\",",
" \"failure_mode\": null | \"F1\" | \"F2\" | \"F3\" | \"F4\" | \"F5\",",
" \"rationale\": \"one sentence explaining the verdict\"",
"}}",
"",
"If verdict is \"correct\", failure_mode MUST be null.",
"If verdict is \"incorrect\", failure_mode MUST be one of F1-F5.",
])
def ts() -> str:
return datetime.now(timezone.utc).isoformat()
def logmsg(msg: str) -> None:
print(f"{ts()} {msg}", flush=True)
def load_env() -> dict[str, str]:
env_path = Path("D:/Projects/waggle-os/.env")
out: dict[str, str] = {}
for line in env_path.read_text(encoding="utf-8").splitlines():
line = line.strip()
if not line or line.startswith("#") or "=" not in line:
continue
k, _, v = line.partition("=")
out[k.strip()] = v.strip().strip('"').strip("'")
return out
import re
def extract_json_body(raw: str) -> dict | None:
if not raw:
return None
trimmed = raw.strip()
if trimmed.startswith("```"):
m = re.match(r"^```(?:json)?\s*\n?(.*?)```\s*$", trimmed, re.DOTALL)
if m:
trimmed = m.group(1).strip()
try:
return json.loads(trimmed)
except Exception:
pass
first = trimmed.find("{")
last = trimmed.rfind("}")
if first != -1 and last != -1 and last > first:
try:
return json.loads(trimmed[first:last + 1])
except Exception:
return None
return None
def parse_verdict(raw: str) -> tuple[str | None, str | None, str | None]:
body = extract_json_body(raw)
if not isinstance(body, dict):
return (None, None, None)
v = body.get("verdict")
fm = body.get("failure_mode")
rat = body.get("rationale")
if v not in ("correct", "incorrect"):
return (None, None, None)
if fm is not None and fm not in ("F1", "F2", "F3", "F4", "F5"):
fm = None
return (v, fm, rat if isinstance(rat, str) else None)
def http_post_json(url: str, headers: dict, body: dict, timeout_s: int = 60) -> tuple[int, dict | str]:
req = urllib.request.Request(
url, data=json.dumps(body).encode("utf-8"), method="POST",
headers={"Content-Type": "application/json", **headers},
)
try:
with urllib.request.urlopen(req, timeout=timeout_s) as resp:
raw = resp.read().decode("utf-8", errors="replace")
try:
return resp.status, json.loads(raw)
except Exception:
return resp.status, raw
except urllib.error.HTTPError as e:
try:
return e.code, e.read().decode("utf-8", errors="replace")[:2000]
except Exception:
return e.code, ""
except Exception as e:
return 0, f"{type(e).__name__}: {e}"
def call_minimax_via_openrouter(prompt: str, or_key: str, max_attempts: int = 3) -> dict:
url = "https://openrouter.ai/api/v1/chat/completions"
headers = {"Authorization": f"Bearer {or_key}"}
body = {
"model": "minimax/minimax-m2.7",
"messages": [{"role": "user", "content": prompt}],
"temperature": 0.0,
"max_tokens": 4096,
}
started = time.time()
retries = 0
last_err = None
for attempt in range(max_attempts):
status, resp = http_post_json(url, headers, body, timeout_s=90)
if status == 200 and isinstance(resp, dict):
choices = resp.get("choices") or []
if choices:
msg = choices[0].get("message") or {}
content = msg.get("content") or msg.get("reasoning_content") or ""
usage = resp.get("usage", {})
return {
"raw_text": content,
"status": 200,
"error": None,
"retries": retries,
"latency_ms": int((time.time() - started) * 1000),
"prompt_tokens": usage.get("prompt_tokens"),
"completion_tokens": usage.get("completion_tokens"),
}
last_err = f"status={status} resp={str(resp)[:400]}"
retries += 1
if attempt < max_attempts - 1:
time.sleep(2 ** attempt)
return {
"raw_text": "",
"status": 0,
"error": last_err,
"retries": retries,
"latency_ms": int((time.time() - started) * 1000),
"prompt_tokens": None,
"completion_tokens": None,
}
def build_sample() -> list[dict]:
"""Load all 100 κ calibration instances with enriched LoCoMo fixture."""
rows = []
with CAL_SRC.open("r", encoding="utf-8") as f:
for line in f:
line = line.strip()
if not line:
continue
rows.append(json.loads(line))
locomo_by_id = {}
with LOCOMO.open("r", encoding="utf-8") as f:
for line in f:
line = line.strip()
if not line:
continue
r = json.loads(line)
locomo_by_id[r.get("instance_id")] = r
enriched = []
for r in rows:
iid = r.get("instance_id")
ensemble = r.get("judge_ensemble") or []
opus = next((j for j in ensemble if "opus" in j.get("model", "").lower()), None)
gpt = next((j for j in ensemble if "gpt" in j.get("model", "").lower()), None)
gemini = next((j for j in ensemble if "gemini" in j.get("model", "").lower()), None)
loc = locomo_by_id.get(iid) or {}
gt = ((loc.get("expected") or [loc.get("gold_answer", "")])[0]) if loc else ""
enriched.append({
"instance_id": iid,
"cell": r.get("cell"),
"question": loc.get("question") or "",
"ground_truth": gt,
"context": loc.get("context") or "",
"model_answer": r.get("model_answer") or "",
"opus_verdict": (opus or {}).get("verdict"),
"opus_failure_mode": (opus or {}).get("failure_mode"),
"gpt_verdict": (gpt or {}).get("verdict"),
"gpt_failure_mode": (gpt or {}).get("failure_mode"),
"gemini_verdict_v5": (gemini or {}).get("verdict"),
"gemini_failure_mode_v5": (gemini or {}).get("failure_mode"),
})
return enriched
def main() -> int:
logmsg("[v6-kappa] MiniMax 100-instance re-calibration START")
env = load_env()
or_key = env.get("OPENROUTER_API_KEY", "").strip()
if not or_key:
logmsg("[v6-kappa] FATAL OPENROUTER_API_KEY missing")
return 2
OUT_DIR.mkdir(parents=True, exist_ok=True)
sample = build_sample()
logmsg(f"[v6-kappa] loaded {len(sample)} κ instances from {CAL_SRC.name}")
# Persist enriched sample for kappa compute
with SAMPLE_PATH.open("w", encoding="utf-8") as f:
for s in sample:
f.write(json.dumps(s, ensure_ascii=False) + "\n")
logmsg(f"[v6-kappa] wrote enriched sample to {SAMPLE_PATH.name}")
# Execute 100 MiniMax calls (sequential, with per-call retries)
rows = []
started_run = time.time()
routing_errors = 0
for i, s in enumerate(sample):
prompt = JUDGE_PROMPT_TEMPLATE.format(
question=s["question"],
ground_truth=s["ground_truth"],
context=s["context"],
model_answer=s["model_answer"],
)
resp = call_minimax_via_openrouter(prompt, or_key)
verdict, fm, rat = parse_verdict(resp["raw_text"])
if resp["status"] != 200:
routing_errors += 1
rows.append({
"instance_id": s["instance_id"],
"cell": s["cell"],
"provider": "minimax",
"model_id": "minimax/minimax-m2.7",
"routing": "openrouter_direct_http",
"litellm_alias_registered": "minimax-m27-via-openrouter",
"http_status": resp["status"],
"error": resp.get("error"),
"retries": resp["retries"],
"latency_ms": resp["latency_ms"],
"prompt_tokens": resp.get("prompt_tokens"),
"completion_tokens": resp.get("completion_tokens"),
"raw_text": resp["raw_text"],
"parsed_verdict": verdict,
"parsed_failure_mode": fm,
"parsed_rationale": rat,
"opus_verdict_ref": s["opus_verdict"],
"gpt_verdict_ref": s["gpt_verdict"],
})
if (i + 1) % 10 == 0 or i == 0:
elapsed = time.time() - started_run
parsed_so_far = sum(1 for r in rows if r.get("parsed_verdict") is not None)
logmsg(
f"[v6-kappa] {i+1:>3}/{len(sample)} {s['instance_id']:30} cell={s['cell']:14} "
f"status={resp['status']} verdict={verdict} parse_ok={parsed_so_far}/{i+1} "
f"routing_err={routing_errors} elapsed={elapsed:.0f}s"
)
# Halt-before-compute check if parse < 90/100
parsed = sum(1 for r in rows if r.get("parsed_verdict") is not None)
logmsg(f"[v6-kappa] completed {len(rows)} calls; parsed={parsed}/100; routing_errors={routing_errors}")
# Write responses regardless of halt status
with RESPONSES_PATH.open("w", encoding="utf-8") as f:
for r in rows:
f.write(json.dumps(r, ensure_ascii=False) + "\n")
logmsg(f"[v6-kappa] wrote {RESPONSES_PATH.name}")
if parsed < 90:
logmsg(f"[v6-kappa] HALT_BEFORE_COMPUTE: parse rate {parsed}/100 < 90 threshold (per brief §3.5)")
return 3
if routing_errors > 5:
logmsg(f"[v6-kappa] ROUTING_ERROR_RATE_FLAG: {routing_errors}/100 > 5% threshold (per brief §3.5)")
return 4
logmsg("[v6-kappa] probe step complete; κ computation handled by kappa-v6-compute.py")
logmsg("[v6-kappa] END")
return 0
if __name__ == "__main__":
import traceback
try:
sys.exit(main())
except Exception as e:
logmsg(f"[FATAL] {type(e).__name__}: {e}")
logmsg(traceback.format_exc()[:2000])
sys.exit(99)

View File

@@ -1,100 +0,0 @@
{"instance_id": "locomo_conv-50_q086", "cell": "no-context", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 17733, "prompt_tokens": 505, "completion_tokens": 571, "raw_text": "```json\n{\n \"verdict\": \"incorrect\",\n \"failure_mode\": \"F3\",\n \"rationale\": \"The model answered 'Surfing' when the ground truth clearly indicates Calvin is planning to go skiing (evidenced by the snowy peak description and image of a person on skis).\"\n}\n```", "parsed_verdict": "incorrect", "parsed_failure_mode": "F3", "parsed_rationale": "The model answered 'Surfing' when the ground truth clearly indicates Calvin is planning to go skiing (evidenced by the snowy peak description and image of a person on skis).", "opus_verdict_ref": "incorrect", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-26_q059", "cell": "no-context", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 16903, "prompt_tokens": 593, "completion_tokens": 623, "raw_text": "{\n \"verdict\": \"incorrect\",\n \"failure_mode\": \"F1\",\n \"rationale\": \"The model refused to answer the question by stating it cannot be determined without context, even though the ground-truth context was provided and contains sufficient information to conclude that Caroline is somewhat, but not extremely religious based on her making art for a church while not exhibiting strongly religious behavior.\"\n}", "parsed_verdict": "incorrect", "parsed_failure_mode": "F1", "parsed_rationale": "The model refused to answer the question by stating it cannot be determined without context, even though the ground-truth context was provided and contains sufficient information to conclude that Caroline is somewhat, but not extremely religious based on her making art for a church while not exhibiting strongly religious behavior.", "opus_verdict_ref": "incorrect", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-44_q000", "cell": "no-context", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 15866, "prompt_tokens": 450, "completion_tokens": 610, "raw_text": "{\n \"verdict\": \"incorrect\",\n \"failure_mode\": \"F4\",\n \"rationale\": \"The model answered 1954, which is a date that does not appear in the ground-truth context and cannot be derived from the provided information (session date of March 2023 and '3 years' tenure would imply ~2020, not 1954), indicating the model fabricated an answer.\"\n}", "parsed_verdict": "incorrect", "parsed_failure_mode": "F4", "parsed_rationale": "The model answered 1954, which is a date that does not appear in the ground-truth context and cannot be derived from the provided information (session date of March 2023 and '3 years' tenure would imply ~2020, not 1954), indicating the model fabricated an answer.", "opus_verdict_ref": "incorrect", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-41_q080", "cell": "no-context", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 11086, "prompt_tokens": 521, "completion_tokens": 554, "raw_text": "{\n \"verdict\": \"incorrect\",\n \"failure_mode\": \"F4\",\n \"rationale\": \"The model provided 'A wind chime' which does not appear anywhere in the ground-truth context; the correct answer is 'painting of a castle on a hill' as explicitly stated by Maria in the supporting context.\"\n}", "parsed_verdict": "incorrect", "parsed_failure_mode": "F4", "parsed_rationale": "The model provided 'A wind chime' which does not appear anywhere in the ground-truth context; the correct answer is 'painting of a castle on a hill' as explicitly stated by Maria in the supporting context.", "opus_verdict_ref": "incorrect", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-42_q001", "cell": "no-context", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 27267, "prompt_tokens": 742, "completion_tokens": 942, "raw_text": "{\n \"verdict\": \"incorrect\",\n \"failure_mode\": \"F3\",\n \"rationale\": \"The model's answer 'Reading' does not appear in the ground truth answer, and while reading appears in the context as only Joanna's interest (not Nate's), it is not a shared interest; the actual shared interests are watching movies and making desserts.\"\n}", "parsed_verdict": "incorrect", "parsed_failure_mode": "F3", "parsed_rationale": "The model's answer 'Reading' does not appear in the ground truth answer, and while reading appears in the context as only Joanna's interest (not Nate's), it is not a shared interest; the actual shared interests are watching movies and making desserts.", "opus_verdict_ref": "incorrect", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-43_q155", "cell": "no-context", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 21117, "prompt_tokens": 464, "completion_tokens": 551, "raw_text": "{\"verdict\": \"incorrect\", \"failure_mode\": \"F3\", \"rationale\": \"The model states John felt 'relieved', but the ground truth explicitly says John felt it was a 'huge success' - these are different emotional states.\"}", "parsed_verdict": "incorrect", "parsed_failure_mode": "F3", "parsed_rationale": "The model states John felt 'relieved', but the ground truth explicitly says John felt it was a 'huge success' - these are different emotional states.", "opus_verdict_ref": "incorrect", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-50_q015", "cell": "no-context", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 5779, "prompt_tokens": 553, "completion_tokens": 198, "raw_text": "{\n \"verdict\": \"incorrect\",\n \"failure_mode\": \"F1\",\n \"rationale\": \"The model explicitly states 'Unknown', refusing to provide an answer, which is a refusal to answer based on available context.\"\n}", "parsed_verdict": "incorrect", "parsed_failure_mode": "F1", "parsed_rationale": "The model explicitly states 'Unknown', refusing to provide an answer, which is a refusal to answer based on available context.", "opus_verdict_ref": "incorrect", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-48_q020", "cell": "no-context", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 8704, "prompt_tokens": 710, "completion_tokens": 493, "raw_text": "{\n \"verdict\": \"incorrect\",\n \"failure_mode\": \"F4\",\n \"rationale\": \"The model mentions games (Animal Crossing and Minecraft) that do not appear anywhere in the ground-truth context, which consists only of Detroit, Walking Dead, Battlefield 1, It Takes Two, and Overcooked 2.\"\n}", "parsed_verdict": "incorrect", "parsed_failure_mode": "F4", "parsed_rationale": "The model mentions games (Animal Crossing and Minecraft) that do not appear anywhere in the ground-truth context, which consists only of Detroit, Walking Dead, Battlefield 1, It Takes Two, and Overcooked 2.", "opus_verdict_ref": "incorrect", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-48_q055", "cell": "no-context", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 9606, "prompt_tokens": 510, "completion_tokens": 625, "raw_text": "```json\n{\n \"verdict\": \"correct\",\n \"failure_mode\": null,\n \"rationale\": \"The model's answer 'Seraphim' matches the ground-truth answer exactly, and while the context excerpts don't explicitly name the pets, the model's answer is confirmed correct by the ground truth.\"\n}\n```", "parsed_verdict": "correct", "parsed_failure_mode": null, "parsed_rationale": "The model's answer 'Seraphim' matches the ground-truth answer exactly, and while the context excerpts don't explicitly name the pets, the model's answer is confirmed correct by the ground truth.", "opus_verdict_ref": "correct", "gpt_verdict_ref": "correct"}
{"instance_id": "locomo_conv-26_q128", "cell": "no-context", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 14317, "prompt_tokens": 493, "completion_tokens": 555, "raw_text": "```json\n{\n \"verdict\": \"incorrect\",\n \"failure_mode\": \"F4\",\n \"rationale\": \"The model answered 'A butterfly,' which is an entity that does not appear anywhere in the ground-truth context; the correct answer is 'a rainbow sidewalk.'\"\n}\n```", "parsed_verdict": "incorrect", "parsed_failure_mode": "F4", "parsed_rationale": "The model answered 'A butterfly,' which is an entity that does not appear anywhere in the ground-truth context; the correct answer is 'a rainbow sidewalk.'", "opus_verdict_ref": "incorrect", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-50_q010", "cell": "no-context", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 23529, "prompt_tokens": 504, "completion_tokens": 606, "raw_text": "{\n \"verdict\": \"incorrect\",\n \"failure_mode\": \"F4\",\n \"rationale\": \"The model provided the date 'October 2019' which does not appear in the ground-truth context (which indicates the flooding occurred around May 9, 2023), constituting a fabrication.\"\n}", "parsed_verdict": "incorrect", "parsed_failure_mode": "F4", "parsed_rationale": "The model provided the date 'October 2019' which does not appear in the ground-truth context (which indicates the flooding occurred around May 9, 2023), constituting a fabrication.", "opus_verdict_ref": "incorrect", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-50_q078", "cell": "no-context", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 16510, "prompt_tokens": 475, "completion_tokens": 604, "raw_text": "{\n \"verdict\": \"incorrect\",\n \"failure_mode\": \"F4\",\n \"rationale\": \"The model answered 'Home,' which does not appear in the ground-truth context and is unrelated to the stated purpose of the necklace as a reminder of why Calvin keeps hustling as a musician.\"\n}", "parsed_verdict": "incorrect", "parsed_failure_mode": "F4", "parsed_rationale": "The model answered 'Home,' which does not appear in the ground-truth context and is unrelated to the stated purpose of the necklace as a reminder of why Calvin keeps hustling as a musician.", "opus_verdict_ref": "incorrect", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-47_q092", "cell": "no-context", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 18246, "prompt_tokens": 487, "completion_tokens": 539, "raw_text": "{\n \"verdict\": \"incorrect\",\n \"failure_mode\": \"F3\",\n \"rationale\": \"The model states 'a database' which is factually incorrect; the ground-truth context clearly states John created an application on smartphones, not a database.\"\n}", "parsed_verdict": "incorrect", "parsed_failure_mode": "F3", "parsed_rationale": "The model states 'a database' which is factually incorrect; the ground-truth context clearly states John created an application on smartphones, not a database.", "opus_verdict_ref": "incorrect", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-30_q057", "cell": "no-context", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 10809, "prompt_tokens": 503, "completion_tokens": 374, "raw_text": "{\n \"verdict\": \"incorrect\",\n \"failure_mode\": \"F2\",\n \"rationale\": \"The model correctly states one relevant fact about focusing on customers (related to building relationships) but omits the other two required pieces of advice: creating a strong brand image and staying positive.\"\n}", "parsed_verdict": "incorrect", "parsed_failure_mode": "F2", "parsed_rationale": "The model correctly states one relevant fact about focusing on customers (related to building relationships) but omits the other two required pieces of advice: creating a strong brand image and staying positive.", "opus_verdict_ref": "incorrect", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-30_q029", "cell": "no-context", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 5371, "prompt_tokens": 565, "completion_tokens": 288, "raw_text": "{\n \"verdict\": \"incorrect\",\n \"failure_mode\": \"F1\",\n \"rationale\": \"The model explicitly stated 'Unknown' when the answer (Paris and Rome) could be directly extracted from the provided supporting context.\"\n}", "parsed_verdict": "incorrect", "parsed_failure_mode": "F1", "parsed_rationale": "The model explicitly stated 'Unknown' when the answer (Paris and Rome) could be directly extracted from the provided supporting context.", "opus_verdict_ref": "incorrect", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-44_q066", "cell": "no-context", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 6208, "prompt_tokens": 492, "completion_tokens": 194, "raw_text": "{\n \"verdict\": \"correct\",\n \"failure_mode\": null,\n \"rationale\": \"The model's answer 'A small dog' is semantically equivalent to the ground truth 'smaller dog' and correctly captures Andrew's preference for his apartment living space.\"\n}", "parsed_verdict": "correct", "parsed_failure_mode": null, "parsed_rationale": "The model's answer 'A small dog' is semantically equivalent to the ground truth 'smaller dog' and correctly captures Andrew's preference for his apartment living space.", "opus_verdict_ref": "correct", "gpt_verdict_ref": "correct"}
{"instance_id": "locomo_conv-44_q002", "cell": "no-context", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 4165, "prompt_tokens": 739, "completion_tokens": 332, "raw_text": "{\n \"verdict\": \"incorrect\",\n \"failure_mode\": \"F2\",\n \"rationale\": \"The model only mentions board games but misses the other three indoor activities (volunteering at pet shelter, wine tasting, and growing flowers) listed in the ground truth.\"\n}", "parsed_verdict": "incorrect", "parsed_failure_mode": "F2", "parsed_rationale": "The model only mentions board games but misses the other three indoor activities (volunteering at pet shelter, wine tasting, and growing flowers) listed in the ground truth.", "opus_verdict_ref": "incorrect", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-47_q046", "cell": "no-context", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 7787, "prompt_tokens": 460, "completion_tokens": 584, "raw_text": "{\n \"verdict\": \"incorrect\",\n \"failure_mode\": \"F3\",\n \"rationale\": \"The model states they 'Never' met, but the context indicates John and James planned to meet the next day (August 27, 2022) at McGee's Pub, making the model's answer factually incorrect.\"\n}", "parsed_verdict": "incorrect", "parsed_failure_mode": "F3", "parsed_rationale": "The model states they 'Never' met, but the context indicates John and James planned to meet the next day (August 27, 2022) at McGee's Pub, making the model's answer factually incorrect.", "opus_verdict_ref": "incorrect", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-43_q076", "cell": "no-context", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 24634, "prompt_tokens": 476, "completion_tokens": 718, "raw_text": "{\n \"verdict\": \"incorrect\",\n \"failure_mode\": \"F4\",\n \"rationale\": \"The model claims the picture was 'a selfie' which does not appear in the ground-truth context; the context only states the picture is from MinaLima, a prop creator for the Harry Potter films, not a selfie.\"\n}", "parsed_verdict": "incorrect", "parsed_failure_mode": "F4", "parsed_rationale": "The model claims the picture was 'a selfie' which does not appear in the ground-truth context; the context only states the picture is from MinaLima, a prop creator for the Harry Potter films, not a selfie.", "opus_verdict_ref": "incorrect", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-41_q065", "cell": "no-context", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 8845, "prompt_tokens": 458, "completion_tokens": 627, "raw_text": "{\n \"verdict\": \"incorrect\",\n \"failure_mode\": \"F1\",\n \"rationale\": \"The model explicitly states it does not know which John is being referred to, effectively refusing to answer the question despite the ground-truth context clearly identifying a John who states his main focuses are improving education and infrastructure.\"\n}", "parsed_verdict": "incorrect", "parsed_failure_mode": "F1", "parsed_rationale": "The model explicitly states it does not know which John is being referred to, effectively refusing to answer the question despite the ground-truth context clearly identifying a John who states his main focuses are improving education and infrastructure.", "opus_verdict_ref": "incorrect", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-50_q086", "cell": "oracle-context", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 5159, "prompt_tokens": 503, "completion_tokens": 267, "raw_text": "{\n \"verdict\": \"correct\",\n \"failure_mode\": null,\n \"rationale\": \"The model's answer 'Skiing' matches the ground truth exactly and is supported by the context, which mentions Calvin's to-do list after the tour and includes an image of skiing on a snowy mountain.\"\n}", "parsed_verdict": "correct", "parsed_failure_mode": null, "parsed_rationale": "The model's answer 'Skiing' matches the ground truth exactly and is supported by the context, which mentions Calvin's to-do list after the tour and includes an image of skiing on a snowy mountain.", "opus_verdict_ref": "correct", "gpt_verdict_ref": "correct"}
{"instance_id": "locomo_conv-26_q059", "cell": "oracle-context", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 31426, "prompt_tokens": 569, "completion_tokens": 1075, "raw_text": "{\n \"verdict\": \"incorrect\",\n \"failure_mode\": \"F2\",\n \"rationale\": \"The model answers 'Yes' which is partially correct (Caroline has some religious connection via her church artwork) but fails to capture the nuance that ground truth specifies: she is 'somewhat, but not extremely religious.'\"\n}", "parsed_verdict": "incorrect", "parsed_failure_mode": "F2", "parsed_rationale": "The model answers 'Yes' which is partially correct (Caroline has some religious connection via her church artwork) but fails to capture the nuance that ground truth specifies: she is 'somewhat, but not extremely religious.'", "opus_verdict_ref": "incorrect", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-44_q000", "cell": "oracle-context", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 7232, "prompt_tokens": 470, "completion_tokens": 291, "raw_text": "{\n \"verdict\": \"correct\",\n \"failure_mode\": null,\n \"rationale\": \"The model correctly answered 2020, which matches the ground truth and is correctly derived from the context stating Audrey had her three dogs (Pepper, Precious, and Panda) for 3 years as of March 2023.\"\n}", "parsed_verdict": "correct", "parsed_failure_mode": null, "parsed_rationale": "The model correctly answered 2020, which matches the ground truth and is correctly derived from the context stating Audrey had her three dogs (Pepper, Precious, and Panda) for 3 years as of March 2023.", "opus_verdict_ref": "correct", "gpt_verdict_ref": "correct"}
{"instance_id": "locomo_conv-41_q080", "cell": "oracle-context", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 6440, "prompt_tokens": 517, "completion_tokens": 559, "raw_text": "{\n \"verdict\": \"incorrect\",\n \"failure_mode\": \"F2\",\n \"rationale\": \"The model correctly identifies that Maria made a painting, but it omits the specific detail that the painting depicts a castle on a hill, which is a required fact from the ground truth.\"\n}", "parsed_verdict": "incorrect", "parsed_failure_mode": "F2", "parsed_rationale": "The model correctly identifies that Maria made a painting, but it omits the specific detail that the painting depicts a castle on a hill, which is a required fact from the ground truth.", "opus_verdict_ref": "incorrect", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-42_q001", "cell": "oracle-context", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 4720, "prompt_tokens": 762, "completion_tokens": 254, "raw_text": "{\n \"verdict\": \"incorrect\",\n \"failure_mode\": \"F2\",\n \"rationale\": \"The model correctly identified movies as a shared interest but failed to include making desserts, which is also mentioned in the ground-truth answer.\"\n}", "parsed_verdict": "incorrect", "parsed_failure_mode": "F2", "parsed_rationale": "The model correctly identified movies as a shared interest but failed to include making desserts, which is also mentioned in the ground-truth answer.", "opus_verdict_ref": "incorrect", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-43_q155", "cell": "oracle-context", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 11079, "prompt_tokens": 480, "completion_tokens": 396, "raw_text": "{\n \"verdict\": \"correct\",\n \"failure_mode\": null,\n \"rationale\": \"The model's answer 'Great' is a synonym for 'a huge success' and correctly captures John's positive feeling about jogging without pain.\"\n}", "parsed_verdict": "correct", "parsed_failure_mode": null, "parsed_rationale": "The model's answer 'Great' is a synonym for 'a huge success' and correctly captures John's positive feeling about jogging without pain.", "opus_verdict_ref": "correct", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-50_q015", "cell": "oracle-context", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 14900, "prompt_tokens": 571, "completion_tokens": 540, "raw_text": "{\n \"verdict\": \"correct\",\n \"failure_mode\": null,\n \"rationale\": \"The model's answer 'To relax' is a correct and equivalent formulation of the ground truth 'because it relaxes and calms him,' as 'relax' and 'calms' are synonymous concepts that capture the same essential fact about why Dave visits parks.\"\n}", "parsed_verdict": "correct", "parsed_failure_mode": null, "parsed_rationale": "The model's answer 'To relax' is a correct and equivalent formulation of the ground truth 'because it relaxes and calms him,' as 'relax' and 'calms' are synonymous concepts that capture the same essential fact about why Dave visits parks.", "opus_verdict_ref": "correct", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-48_q020", "cell": "oracle-context", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 16522, "prompt_tokens": 719, "completion_tokens": 265, "raw_text": "```json\n{\n \"verdict\": \"incorrect\",\n \"failure_mode\": \"F2\",\n \"rationale\": \"The model correctly identified three games (Detroit, It Takes Two, Overcooked 2) but missed two others (Walking Dead, Battlefield 1) that appear in the ground truth.\"\n}\n```", "parsed_verdict": "incorrect", "parsed_failure_mode": "F2", "parsed_rationale": "The model correctly identified three games (Detroit, It Takes Two, Overcooked 2) but missed two others (Walking Dead, Battlefield 1) that appear in the ground truth.", "opus_verdict_ref": "incorrect", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-48_q055", "cell": "oracle-context", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 44330, "prompt_tokens": 509, "completion_tokens": 976, "raw_text": "{\n \"verdict\": \"incorrect\",\n \"failure_mode\": \"F3\",\n \"rationale\": \"The model answered Susie, but the ground truth states Seraphim was adopted more recently, indicating the model derived incorrect facts from the context.\"\n}", "parsed_verdict": "incorrect", "parsed_failure_mode": "F3", "parsed_rationale": "The model answered Susie, but the ground truth states Seraphim was adopted more recently, indicating the model derived incorrect facts from the context.", "opus_verdict_ref": "incorrect", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-26_q128", "cell": "oracle-context", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 3221, "prompt_tokens": 491, "completion_tokens": 216, "raw_text": "{\n \"verdict\": \"correct\",\n \"failure_mode\": null,\n \"rationale\": \"The model's answer 'A rainbow sidewalk' exactly matches the ground-truth answer 'a rainbow sidewalk' with only a minor capitalization difference, making it correct.\"\n}", "parsed_verdict": "correct", "parsed_failure_mode": null, "parsed_rationale": "The model's answer 'A rainbow sidewalk' exactly matches the ground-truth answer 'a rainbow sidewalk' with only a minor capitalization difference, making it correct.", "opus_verdict_ref": "correct", "gpt_verdict_ref": "correct"}
{"instance_id": "locomo_conv-50_q010", "cell": "oracle-context", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 13441, "prompt_tokens": 485, "completion_tokens": 336, "raw_text": "{\n \"verdict\": \"correct\",\n \"failure_mode\": null,\n \"rationale\": \"The model's answer 'Last week' is equivalent to the ground truth 'a week before 16 May, 2023' since the conversation occurred on 16 May 2023, making minor phrasing differences acceptable.\"\n}", "parsed_verdict": "correct", "parsed_failure_mode": null, "parsed_rationale": "The model's answer 'Last week' is equivalent to the ground truth 'a week before 16 May, 2023' since the conversation occurred on 16 May 2023, making minor phrasing differences acceptable.", "opus_verdict_ref": "correct", "gpt_verdict_ref": "correct"}
{"instance_id": "locomo_conv-50_q078", "cell": "oracle-context", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 9610, "prompt_tokens": 482, "completion_tokens": 303, "raw_text": "{\n \"verdict\": \"correct\",\n \"failure_mode\": null,\n \"rationale\": \"The model's answer matches the ground truth exactly, capturing the fact that the necklace was meant to remind Calvin of why he keeps hustling as a musician, with only a minor capitalization difference that doesn't affect meaning.\"\n}", "parsed_verdict": "correct", "parsed_failure_mode": null, "parsed_rationale": "The model's answer matches the ground truth exactly, capturing the fact that the necklace was meant to remind Calvin of why he keeps hustling as a musician, with only a minor capitalization difference that doesn't affect meaning.", "opus_verdict_ref": "correct", "gpt_verdict_ref": "correct"}
{"instance_id": "locomo_conv-47_q092", "cell": "oracle-context", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 9489, "prompt_tokens": 487, "completion_tokens": 422, "raw_text": "```json\n{\n \"verdict\": \"incorrect\",\n \"failure_mode\": \"F2\",\n \"rationale\": \"The model's answer correctly identifies that John created an application but omits the specific detail that it was on smartphones, which is a required fact from the ground truth.\"\n}\n```", "parsed_verdict": "incorrect", "parsed_failure_mode": "F2", "parsed_rationale": "The model's answer correctly identifies that John created an application but omits the specific detail that it was on smartphones, which is a required fact from the ground truth.", "opus_verdict_ref": "correct", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-30_q057", "cell": "oracle-context", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 71668, "prompt_tokens": 498, "completion_tokens": 2221, "raw_text": "{\n \"verdict\": \"incorrect\",\n \"failure_mode\": \"F1\",\n \"rationale\": \"The model explicitly states 'Not mentioned' to avoid providing an answer, even though the advice content (brand identity, building relationships, staying positive) is present in the context.\"\n}", "parsed_verdict": "incorrect", "parsed_failure_mode": "F1", "parsed_rationale": "The model explicitly states 'Not mentioned' to avoid providing an answer, even though the advice content (brand identity, building relationships, staying positive) is present in the context.", "opus_verdict_ref": "incorrect", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-30_q029", "cell": "oracle-context", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 11020, "prompt_tokens": 550, "completion_tokens": 251, "raw_text": "{\n \"verdict\": \"correct\",\n \"failure_mode\": null,\n \"rationale\": \"The model correctly identified both cities (Paris and Rome) that Jon visited according to the ground-truth context, with no additional incorrect claims.\"\n}", "parsed_verdict": "correct", "parsed_failure_mode": null, "parsed_rationale": "The model correctly identified both cities (Paris and Rome) that Jon visited according to the ground-truth context, with no additional incorrect claims.", "opus_verdict_ref": "correct", "gpt_verdict_ref": "correct"}
{"instance_id": "locomo_conv-44_q066", "cell": "oracle-context", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 5882, "prompt_tokens": 512, "completion_tokens": 214, "raw_text": "{\n \"verdict\": \"correct\",\n \"failure_mode\": null,\n \"rationale\": \"The model's answer 'A smaller dog' exactly matches the ground-truth answer 'smaller dog' and is fully supported by the context where Andrew states that for someone living in an apartment, 'a smaller dog would be best.'\"\n}", "parsed_verdict": "correct", "parsed_failure_mode": null, "parsed_rationale": "The model's answer 'A smaller dog' exactly matches the ground-truth answer 'smaller dog' and is fully supported by the context where Andrew states that for someone living in an apartment, 'a smaller dog would be best.'", "opus_verdict_ref": "correct", "gpt_verdict_ref": "correct"}
{"instance_id": "locomo_conv-44_q002", "cell": "oracle-context", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 9991, "prompt_tokens": 739, "completion_tokens": 412, "raw_text": "{\n \"verdict\": \"incorrect\",\n \"failure_mode\": \"F2\",\n \"rationale\": \"The model only mentions board games but omits three other activities (volunteering at pet shelter, wine tasting, and growing flowers) that Andrew pursued with his girlfriend according to the ground truth.\"\n}", "parsed_verdict": "incorrect", "parsed_failure_mode": "F2", "parsed_rationale": "The model only mentions board games but omits three other activities (volunteering at pet shelter, wine tasting, and growing flowers) that Andrew pursued with his girlfriend according to the ground truth.", "opus_verdict_ref": "incorrect", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-47_q046", "cell": "oracle-context", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 13063, "prompt_tokens": 448, "completion_tokens": 344, "raw_text": "{\n \"verdict\": \"correct\",\n \"failure_mode\": null,\n \"rationale\": \"The model's answer '27 August, 2022' is equivalent to the ground truth 'August 27, 2022' (same date, merely different formatting), and aligns with the context showing John said 'See you tomorrow at McGee's Pub' on August 26, 2022.\"\n}", "parsed_verdict": "correct", "parsed_failure_mode": null, "parsed_rationale": "The model's answer '27 August, 2022' is equivalent to the ground truth 'August 27, 2022' (same date, merely different formatting), and aligns with the context showing John said 'See you tomorrow at McGee's Pub' on August 26, 2022.", "opus_verdict_ref": "correct", "gpt_verdict_ref": "correct"}
{"instance_id": "locomo_conv-43_q076", "cell": "oracle-context", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 7570, "prompt_tokens": 497, "completion_tokens": 296, "raw_text": "{\n \"verdict\": \"correct\",\n \"failure_mode\": null,\n \"rationale\": \"The model's answer 'MinaLima' correctly identifies the creator of the picture Tim shared, which matches the ground truth that it was 'MinaLima's creation from the Harry Potter films'.\"\n}", "parsed_verdict": "correct", "parsed_failure_mode": null, "parsed_rationale": "The model's answer 'MinaLima' correctly identifies the creator of the picture Tim shared, which matches the ground truth that it was 'MinaLima's creation from the Harry Potter films'.", "opus_verdict_ref": "incorrect", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-41_q065", "cell": "oracle-context", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 15698, "prompt_tokens": 441, "completion_tokens": 260, "raw_text": "{\n \"verdict\": \"correct\",\n \"failure_mode\": null,\n \"rationale\": \"The model accurately identified both key topics (education and infrastructure) from the ground truth context, with only a minor phrasing difference (dropping 'improving') which is acceptable.\"\n}", "parsed_verdict": "correct", "parsed_failure_mode": null, "parsed_rationale": "The model accurately identified both key topics (education and infrastructure) from the ground truth context, with only a minor phrasing difference (dropping 'improving') which is acceptable.", "opus_verdict_ref": "correct", "gpt_verdict_ref": "correct"}
{"instance_id": "locomo_conv-50_q086", "cell": "full-context", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 4905, "prompt_tokens": 502, "completion_tokens": 168, "raw_text": "{\n \"verdict\": \"correct\",\n \"failure_mode\": null,\n \"rationale\": \"The model correctly identifies the sports activity (skiing) that Calvin plans to try after the tour with Frank Ocean, matching the ground truth.\"\n}", "parsed_verdict": "correct", "parsed_failure_mode": null, "parsed_rationale": "The model correctly identifies the sports activity (skiing) that Calvin plans to try after the tour with Frank Ocean, matching the ground truth.", "opus_verdict_ref": "correct", "gpt_verdict_ref": "correct"}
{"instance_id": "locomo_conv-26_q059", "cell": "full-context", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 11874, "prompt_tokens": 586, "completion_tokens": 350, "raw_text": "{\"verdict\": \"incorrect\", \"failure_mode\": \"F1\", \"rationale\": \"The model explicitly stated 'unknown' rather than attempting to answer the question, which constitutes an abstention when the context provides sufficient information to determine that Caroline is somewhat but not extremely religious.\"}", "parsed_verdict": "incorrect", "parsed_failure_mode": "F1", "parsed_rationale": "The model explicitly stated 'unknown' rather than attempting to answer the question, which constitutes an abstention when the context provides sufficient information to determine that Caroline is somewhat but not extremely religious.", "opus_verdict_ref": "incorrect", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-44_q000", "cell": "full-context", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 10437, "prompt_tokens": 470, "completion_tokens": 455, "raw_text": "{\n \"verdict\": \"correct\",\n \"failure_mode\": null,\n \"rationale\": \"The model's answer of 2020 matches the ground truth, which can be derived from the context stating Audrey had her three dogs for 3 years as of March 27, 2023, placing their adoption around 2020.\"\n}", "parsed_verdict": "correct", "parsed_failure_mode": null, "parsed_rationale": "The model's answer of 2020 matches the ground truth, which can be derived from the context stating Audrey had her three dogs for 3 years as of March 27, 2023, placing their adoption around 2020.", "opus_verdict_ref": "correct", "gpt_verdict_ref": "correct"}
{"instance_id": "locomo_conv-41_q080", "cell": "full-context", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 10963, "prompt_tokens": 517, "completion_tokens": 516, "raw_text": "{\n \"verdict\": \"incorrect\",\n \"failure_mode\": \"F2\",\n \"rationale\": \"The model correctly identifies that Maria made 'a painting' but omits the essential detail that it was 'of a castle on a hill,' which is a required fact from the ground truth.\"\n}", "parsed_verdict": "incorrect", "parsed_failure_mode": "F2", "parsed_rationale": "The model correctly identifies that Maria made 'a painting' but omits the essential detail that it was 'of a castle on a hill,' which is a required fact from the ground truth.", "opus_verdict_ref": "incorrect", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-42_q001", "cell": "full-context", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 5690, "prompt_tokens": 764, "completion_tokens": 305, "raw_text": "{\n \"verdict\": \"incorrect\",\n \"failure_mode\": \"F2\",\n \"rationale\": \"The model correctly identified watching movies as a shared interest but failed to include making desserts, which both Joanna and Nate discuss enjoying.\"\n}", "parsed_verdict": "incorrect", "parsed_failure_mode": "F2", "parsed_rationale": "The model correctly identified watching movies as a shared interest but failed to include making desserts, which both Joanna and Nate discuss enjoying.", "opus_verdict_ref": "incorrect", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-43_q155", "cell": "full-context", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 10228, "prompt_tokens": 480, "completion_tokens": 495, "raw_text": "{\n \"verdict\": \"correct\",\n \"failure_mode\": null,\n \"rationale\": \"The model's answer 'great' is a direct synonym from the context where John says 'It was great!' and is equivalent to the ground truth 'huge success' in describing John's emotional response to jogging without pain.\"\n}", "parsed_verdict": "correct", "parsed_failure_mode": null, "parsed_rationale": "The model's answer 'great' is a direct synonym from the context where John says 'It was great!' and is equivalent to the ground truth 'huge success' in describing John's emotional response to jogging without pain.", "opus_verdict_ref": "correct", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-50_q015", "cell": "full-context", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 16273, "prompt_tokens": 571, "completion_tokens": 367, "raw_text": "```json\n{\n \"verdict\": \"incorrect\",\n \"failure_mode\": \"F2\",\n \"rationale\": \"The model correctly identifies that Dave visits parks to relax, but omits the additional detail that it also calms him, making this a partial answer.\"\n}\n```", "parsed_verdict": "incorrect", "parsed_failure_mode": "F2", "parsed_rationale": "The model correctly identifies that Dave visits parks to relax, but omits the additional detail that it also calms him, making this a partial answer.", "opus_verdict_ref": "correct", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-48_q020", "cell": "full-context", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 7004, "prompt_tokens": 699, "completion_tokens": 294, "raw_text": "{\n \"verdict\": \"incorrect\",\n \"failure_mode\": \"F2\",\n \"rationale\": \"The model correctly identified three games (Detroit, It Takes Two, Overcooked 2) but missed two games (Walking Dead, Battlefield 1) that appear in the ground truth.\"\n}", "parsed_verdict": "incorrect", "parsed_failure_mode": "F2", "parsed_rationale": "The model correctly identified three games (Detroit, It Takes Two, Overcooked 2) but missed two games (Walking Dead, Battlefield 1) that appear in the ground truth.", "opus_verdict_ref": "incorrect", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-48_q055", "cell": "full-context", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 22814, "prompt_tokens": 508, "completion_tokens": 345, "raw_text": "{\n \"verdict\": \"incorrect\",\n \"failure_mode\": \"F1\",\n \"rationale\": \"The model explicitly states it does not know the answer, which is a refusal to provide the correct answer.\"\n}", "parsed_verdict": "incorrect", "parsed_failure_mode": "F1", "parsed_rationale": "The model explicitly states it does not know the answer, which is a refusal to provide the correct answer.", "opus_verdict_ref": "incorrect", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-26_q128", "cell": "full-context", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 7288, "prompt_tokens": 474, "completion_tokens": 196, "raw_text": "{\n \"verdict\": \"correct\",\n \"failure_mode\": null,\n \"rationale\": \"The model's answer 'rainbow sidewalk' matches the ground truth 'a rainbow sidewalk' exactly, representing the same object without the article.\"\n}", "parsed_verdict": "correct", "parsed_failure_mode": null, "parsed_rationale": "The model's answer 'rainbow sidewalk' matches the ground truth 'a rainbow sidewalk' exactly, representing the same object without the article.", "opus_verdict_ref": "correct", "gpt_verdict_ref": "correct"}
{"instance_id": "locomo_conv-50_q010", "cell": "full-context", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 14884, "prompt_tokens": 502, "completion_tokens": 452, "raw_text": "{\n \"verdict\": \"correct\",\n \"failure_mode\": null,\n \"rationale\": \"The model's answer 'last week' is equivalent to the ground truth 'a week before 16 May, 2023' as both convey the same temporal information that the flooding occurred during the week prior to May 16, 2023.\"\n}", "parsed_verdict": "correct", "parsed_failure_mode": null, "parsed_rationale": "The model's answer 'last week' is equivalent to the ground truth 'a week before 16 May, 2023' as both convey the same temporal information that the flooding occurred during the week prior to May 16, 2023.", "opus_verdict_ref": "correct", "gpt_verdict_ref": "correct"}
{"instance_id": "locomo_conv-50_q078", "cell": "full-context", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 3608, "prompt_tokens": 485, "completion_tokens": 201, "raw_text": "{\n \"verdict\": \"correct\",\n \"failure_mode\": null,\n \"rationale\": \"The model's answer matches the ground-truth answer exactly, capturing the intended reminder of why Calvin keeps hustling as a musician.\"\n}", "parsed_verdict": "correct", "parsed_failure_mode": null, "parsed_rationale": "The model's answer matches the ground-truth answer exactly, capturing the intended reminder of why Calvin keeps hustling as a musician.", "opus_verdict_ref": "correct", "gpt_verdict_ref": "correct"}
{"instance_id": "locomo_conv-47_q092", "cell": "full-context", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 37908, "prompt_tokens": 484, "completion_tokens": 600, "raw_text": "{\n \"verdict\": \"incorrect\",\n \"failure_mode\": \"F2\",\n \"rationale\": \"The model correctly identifies that John created an application but omits the specific detail that it was on smartphones, which is part of the ground truth answer.\"\n}", "parsed_verdict": "incorrect", "parsed_failure_mode": "F2", "parsed_rationale": "The model correctly identifies that John created an application but omits the specific detail that it was on smartphones, which is part of the ground truth answer.", "opus_verdict_ref": "incorrect", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-30_q057", "cell": "full-context", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 19809, "prompt_tokens": 497, "completion_tokens": 519, "raw_text": "{\n \"verdict\": \"incorrect\",\n \"failure_mode\": \"F1\",\n \"rationale\": \"The model explicitly stated 'unknown', indicating it does not know the answer, which constitutes an abstention from providing the required information.\"\n}", "parsed_verdict": "incorrect", "parsed_failure_mode": "F1", "parsed_rationale": "The model explicitly stated 'unknown', indicating it does not know the answer, which constitutes an abstention from providing the required information.", "opus_verdict_ref": "incorrect", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-30_q029", "cell": "full-context", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 4197, "prompt_tokens": 566, "completion_tokens": 135, "raw_text": "{\n \"verdict\": \"correct\",\n \"failure_mode\": null,\n \"rationale\": \"The model correctly listed both cities (Paris and Rome) that Jon visited according to the ground-truth context.\"\n}", "parsed_verdict": "correct", "parsed_failure_mode": null, "parsed_rationale": "The model correctly listed both cities (Paris and Rome) that Jon visited according to the ground-truth context.", "opus_verdict_ref": "correct", "gpt_verdict_ref": "correct"}
{"instance_id": "locomo_conv-44_q066", "cell": "full-context", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 7687, "prompt_tokens": 511, "completion_tokens": 239, "raw_text": "{\n \"verdict\": \"correct\",\n \"failure_mode\": null,\n \"rationale\": \"The model's answer 'smaller dog' exactly matches the ground-truth answer and is directly supported by the context which states that for Andrew, 'a person living in an apartment, a smaller dog would be best.'\"\n}", "parsed_verdict": "correct", "parsed_failure_mode": null, "parsed_rationale": "The model's answer 'smaller dog' exactly matches the ground-truth answer and is directly supported by the context which states that for Andrew, 'a person living in an apartment, a smaller dog would be best.'", "opus_verdict_ref": "correct", "gpt_verdict_ref": "correct"}
{"instance_id": "locomo_conv-44_q002", "cell": "full-context", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 12652, "prompt_tokens": 743, "completion_tokens": 634, "raw_text": "{\n \"verdict\": \"incorrect\",\n \"failure_mode\": \"F2\",\n \"rationale\": \"The model correctly identified playing board games as one indoor activity but missed the other three activities (volunteering at pet shelter, wine tasting, and growing flowers) mentioned in the ground truth.\"\n}", "parsed_verdict": "incorrect", "parsed_failure_mode": "F2", "parsed_rationale": "The model correctly identified playing board games as one indoor activity but missed the other three activities (volunteering at pet shelter, wine tasting, and growing flowers) mentioned in the ground truth.", "opus_verdict_ref": "incorrect", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-47_q046", "cell": "full-context", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 17719, "prompt_tokens": 464, "completion_tokens": 703, "raw_text": "```json\n{\n \"verdict\": \"correct\",\n \"failure_mode\": null,\n \"rationale\": \"The model's answer 'tomorrow' is factually correct, as John said 'See you tomorrow at McGee's Pub' on August 26, 2022, making August 27, 2022 the correct meeting date, which is equivalent to 'tomorrow' in this context.\"\n}\n```", "parsed_verdict": "correct", "parsed_failure_mode": null, "parsed_rationale": "The model's answer 'tomorrow' is factually correct, as John said 'See you tomorrow at McGee's Pub' on August 26, 2022, making August 27, 2022 the correct meeting date, which is equivalent to 'tomorrow' in this context.", "opus_verdict_ref": "incorrect", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-43_q076", "cell": "full-context", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 14900, "prompt_tokens": 497, "completion_tokens": 819, "raw_text": "{\n \"verdict\": \"correct\",\n \"failure_mode\": null,\n \"rationale\": \"The model's answer 'from MinaLima' correctly identifies the creator of the picture as MinaLima, which matches the ground truth, and the supporting context confirms this attribution.\"\n}", "parsed_verdict": "correct", "parsed_failure_mode": null, "parsed_rationale": "The model's answer 'from MinaLima' correctly identifies the creator of the picture as MinaLima, which matches the ground truth, and the supporting context confirms this attribution.", "opus_verdict_ref": "incorrect", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-41_q065", "cell": "full-context", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 6871, "prompt_tokens": 443, "completion_tokens": 241, "raw_text": "{\n \"verdict\": \"correct\",\n \"failure_mode\": null,\n \"rationale\": \"The model's answer matches the ground truth exactly, identifying improving education and infrastructure as John's main focus in local politics.\"\n}", "parsed_verdict": "correct", "parsed_failure_mode": null, "parsed_rationale": "The model's answer matches the ground truth exactly, identifying improving education and infrastructure as John's main focus in local politics.", "opus_verdict_ref": "correct", "gpt_verdict_ref": "correct"}
{"instance_id": "locomo_conv-50_q086", "cell": "retrieval", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 7062, "prompt_tokens": 502, "completion_tokens": 449, "raw_text": "{\n \"verdict\": \"incorrect\",\n \"failure_mode\": \"F3\",\n \"rationale\": \"The model answered 'Not mentioned' but the ground-truth context clearly indicates skiing is the activity Calvin plans to try after the tour, as evidenced by the snowy peak reference and the image of a person on skis.\"\n}", "parsed_verdict": "incorrect", "parsed_failure_mode": "F3", "parsed_rationale": "The model answered 'Not mentioned' but the ground-truth context clearly indicates skiing is the activity Calvin plans to try after the tour, as evidenced by the snowy peak reference and the image of a person on skis.", "opus_verdict_ref": "incorrect", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-26_q059", "cell": "retrieval", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 15512, "prompt_tokens": 569, "completion_tokens": 523, "raw_text": "{\n \"verdict\": \"incorrect\",\n \"failure_mode\": \"F3\",\n \"rationale\": \"The model answered 'No' while the ground truth indicates Caroline is 'somewhat, but not extremely religious' based on her creating artwork for a church and participating in religious spaces.\"\n}", "parsed_verdict": "incorrect", "parsed_failure_mode": "F3", "parsed_rationale": "The model answered 'No' while the ground truth indicates Caroline is 'somewhat, but not extremely religious' based on her creating artwork for a church and participating in religious spaces.", "opus_verdict_ref": "incorrect", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-44_q000", "cell": "retrieval", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 23052, "prompt_tokens": 467, "completion_tokens": 805, "raw_text": "{\n \"verdict\": \"incorrect\",\n \"failure_mode\": \"F2\",\n \"rationale\": \"The model failed to derive the year 2020 from the context (3 years before March 2023), instead stating 'Not mentioned' when the answer could be calculated from the given information.\"\n}", "parsed_verdict": "incorrect", "parsed_failure_mode": "F2", "parsed_rationale": "The model failed to derive the year 2020 from the context (3 years before March 2023), instead stating 'Not mentioned' when the answer could be calculated from the given information.", "opus_verdict_ref": "incorrect", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-41_q080", "cell": "retrieval", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 10364, "prompt_tokens": 500, "completion_tokens": 306, "raw_text": "{\n \"verdict\": \"incorrect\",\n \"failure_mode\": \"F2\",\n \"rationale\": \"The model correctly states that Maria made a painting, but fails to include the specific detail that it was a painting of a castle on a hill, which is the key distinguishing fact in the ground truth.\"\n}", "parsed_verdict": "incorrect", "parsed_failure_mode": "F2", "parsed_rationale": "The model correctly states that Maria made a painting, but fails to include the specific detail that it was a painting of a castle on a hill, which is the key distinguishing fact in the ground truth.", "opus_verdict_ref": "incorrect", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-42_q001", "cell": "retrieval", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 18171, "prompt_tokens": 744, "completion_tokens": 516, "raw_text": "{\n \"verdict\": \"incorrect\",\n \"failure_mode\": \"F3\",\n \"rationale\": \"The model correctly identified movies as a shared interest but incorrectly stated nature as shared (only Joanna enjoys nature, not Nate), and failed to mention the shared interest of making desserts.\"\n}", "parsed_verdict": "incorrect", "parsed_failure_mode": "F3", "parsed_rationale": "The model correctly identified movies as a shared interest but incorrectly stated nature as shared (only Joanna enjoys nature, not Nate), and failed to mention the shared interest of making desserts.", "opus_verdict_ref": "incorrect", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-43_q155", "cell": "retrieval", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 18019, "prompt_tokens": 484, "completion_tokens": 636, "raw_text": "```json\n{\n \"verdict\": \"incorrect\",\n \"failure_mode\": \"F2\",\n \"rationale\": \"The model's answer 'Relieved' captures a plausible emotional response but misses the ground truth's explicit statement that jogging without pain was 'a huge success' and John said 'It was great!' — the model provides an alternative valid emotion without stating the specific required fact.\"\n}\n```", "parsed_verdict": "incorrect", "parsed_failure_mode": "F2", "parsed_rationale": "The model's answer 'Relieved' captures a plausible emotional response but misses the ground truth's explicit statement that jogging without pain was 'a huge success' and John said 'It was great!' — the model provides an alternative valid emotion without stating the specific required fact.", "opus_verdict_ref": "incorrect", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-50_q015", "cell": "retrieval", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 30829, "prompt_tokens": 582, "completion_tokens": 1060, "raw_text": "```json\n{\n \"verdict\": \"incorrect\",\n \"failure_mode\": \"F4\",\n \"rationale\": \"The model adds 'hang out with friends' which does not appear in the ground-truth context where Dave visits parks alone to relax in nature.\"\n}\n```", "parsed_verdict": "incorrect", "parsed_failure_mode": "F4", "parsed_rationale": "The model adds 'hang out with friends' which does not appear in the ground-truth context where Dave visits parks alone to relax in nature.", "opus_verdict_ref": "incorrect", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-48_q020", "cell": "retrieval", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 13576, "prompt_tokens": 716, "completion_tokens": 393, "raw_text": "{\n \"verdict\": \"incorrect\",\n \"failure_mode\": \"F2\",\n \"rationale\": \"The model identified three of the five games (Detroit, Overcooked 2, It Takes Two) but omitted Walking Dead and Battlefield 1 from the ground-truth answer.\"\n}", "parsed_verdict": "incorrect", "parsed_failure_mode": "F2", "parsed_rationale": "The model identified three of the five games (Detroit, Overcooked 2, It Takes Two) but omitted Walking Dead and Battlefield 1 from the ground-truth answer.", "opus_verdict_ref": "incorrect", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-48_q055", "cell": "retrieval", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 8026, "prompt_tokens": 513, "completion_tokens": 328, "raw_text": "{\n \"verdict\": \"correct\",\n \"failure_mode\": null,\n \"rationale\": \"The model's answer 'Seraphim' matches the ground-truth answer exactly, indicating the model correctly identified which pet Jolene adopted more recently.\"\n}", "parsed_verdict": "correct", "parsed_failure_mode": null, "parsed_rationale": "The model's answer 'Seraphim' matches the ground-truth answer exactly, indicating the model correctly identified which pet Jolene adopted more recently.", "opus_verdict_ref": "correct", "gpt_verdict_ref": "correct"}
{"instance_id": "locomo_conv-26_q128", "cell": "retrieval", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 3990, "prompt_tokens": 494, "completion_tokens": 147, "raw_text": "{\n \"verdict\": \"correct\",\n \"failure_mode\": null,\n \"rationale\": \"The model's answer 'A rainbow sidewalk' exactly matches the ground truth answer, identifying what Caroline found during her walk.\"\n}", "parsed_verdict": "correct", "parsed_failure_mode": null, "parsed_rationale": "The model's answer 'A rainbow sidewalk' exactly matches the ground truth answer, identifying what Caroline found during her walk.", "opus_verdict_ref": "correct", "gpt_verdict_ref": "correct"}
{"instance_id": "locomo_conv-50_q010", "cell": "retrieval", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 12906, "prompt_tokens": 502, "completion_tokens": 509, "raw_text": "{\n \"verdict\": \"correct\",\n \"failure_mode\": null,\n \"rationale\": \"The model's answer 'Last week' correctly captures the temporal information from the ground truth, as the context confirms Calvin's place was flooded during the week before 16 May 2023.\"\n}", "parsed_verdict": "correct", "parsed_failure_mode": null, "parsed_rationale": "The model's answer 'Last week' correctly captures the temporal information from the ground truth, as the context confirms Calvin's place was flooded during the week before 16 May 2023.", "opus_verdict_ref": "correct", "gpt_verdict_ref": "correct"}
{"instance_id": "locomo_conv-50_q078", "cell": "retrieval", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 5835, "prompt_tokens": 482, "completion_tokens": 140, "raw_text": "{\n \"verdict\": \"correct\",\n \"failure_mode\": null,\n \"rationale\": \"The model's answer matches the ground‑truth answer exactly.\"\n}", "parsed_verdict": "correct", "parsed_failure_mode": null, "parsed_rationale": "The model's answer matches the ground‑truth answer exactly.", "opus_verdict_ref": "correct", "gpt_verdict_ref": "correct"}
{"instance_id": "locomo_conv-47_q092", "cell": "retrieval", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 18160, "prompt_tokens": 485, "completion_tokens": 678, "raw_text": "{\n \"verdict\": \"incorrect\",\n \"failure_mode\": \"F2\",\n \"rationale\": \"The model's answer 'A software tool' is missing the specific detail that the application was on smartphones, which is a required fact from the ground truth answer.\"\n}", "parsed_verdict": "incorrect", "parsed_failure_mode": "F2", "parsed_rationale": "The model's answer 'A software tool' is missing the specific detail that the application was on smartphones, which is a required fact from the ground truth answer.", "opus_verdict_ref": "incorrect", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-30_q057", "cell": "retrieval", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 21614, "prompt_tokens": 499, "completion_tokens": 665, "raw_text": "{\"verdict\":\"incorrect\",\"failure_mode\":\"F2\",\"rationale\":\"The model's answer contains no overlap with the required facts from the ground truth—it mentions passion, focus, and resilience instead of the three specific advice points (build relationships with customers, create a strong brand image, stay positive) that appeared in the context.\"}", "parsed_verdict": "incorrect", "parsed_failure_mode": "F2", "parsed_rationale": "The model's answer contains no overlap with the required facts from the ground truth—it mentions passion, focus, and resilience instead of the three specific advice points (build relationships with customers, create a strong brand image, stay positive) that appeared in the context.", "opus_verdict_ref": "incorrect", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-30_q029", "cell": "retrieval", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 8084, "prompt_tokens": 568, "completion_tokens": 266, "raw_text": "{\n \"verdict\": \"incorrect\",\n \"failure_mode\": \"F2\",\n \"rationale\": \"The model correctly identified Paris but omitted Rome, providing only a partial answer to the question.\"\n}", "parsed_verdict": "incorrect", "parsed_failure_mode": "F2", "parsed_rationale": "The model correctly identified Paris but omitted Rome, providing only a partial answer to the question.", "opus_verdict_ref": "incorrect", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-44_q066", "cell": "retrieval", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 4891, "prompt_tokens": 512, "completion_tokens": 264, "raw_text": "{\n \"verdict\": \"correct\",\n \"failure_mode\": null,\n \"rationale\": \"The model's answer 'A smaller dog' matches the ground truth 'smaller dog' exactly, correctly identifying that Andrew was looking for a smaller dog based on his apartment living space.\"\n}", "parsed_verdict": "correct", "parsed_failure_mode": null, "parsed_rationale": "The model's answer 'A smaller dog' matches the ground truth 'smaller dog' exactly, correctly identifying that Andrew was looking for a smaller dog based on his apartment living space.", "opus_verdict_ref": "correct", "gpt_verdict_ref": "correct"}
{"instance_id": "locomo_conv-44_q002", "cell": "retrieval", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 13170, "prompt_tokens": 739, "completion_tokens": 450, "raw_text": "{\n \"verdict\": \"incorrect\",\n \"failure_mode\": \"F2\",\n \"rationale\": \"The model only mentions wine tasting but misses three other activities (boardgames, volunteering at pet shelter, and growing flowers) that Andrew pursued with his girlfriend.\"\n}", "parsed_verdict": "incorrect", "parsed_failure_mode": "F2", "parsed_rationale": "The model only mentions wine tasting but misses three other activities (boardgames, volunteering at pet shelter, and growing flowers) that Andrew pursued with his girlfriend.", "opus_verdict_ref": "incorrect", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-47_q046", "cell": "retrieval", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 15948, "prompt_tokens": 460, "completion_tokens": 643, "raw_text": "{\n \"verdict\": \"correct\",\n \"failure_mode\": null,\n \"rationale\": \"The model's answer 'Tomorrow' is equivalent to the ground truth 'August 27, 2022' because the context establishes that on August 26 John said 'See you tomorrow at McGee's Pub,' making 'tomorrow' the correct relative reference to August 27, 2022.\"\n}", "parsed_verdict": "correct", "parsed_failure_mode": null, "parsed_rationale": "The model's answer 'Tomorrow' is equivalent to the ground truth 'August 27, 2022' because the context establishes that on August 26 John said 'See you tomorrow at McGee's Pub,' making 'tomorrow' the correct relative reference to August 27, 2022.", "opus_verdict_ref": "incorrect", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-43_q076", "cell": "retrieval", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 22719, "prompt_tokens": 495, "completion_tokens": 421, "raw_text": "```json\n{\n \"verdict\": \"correct\",\n \"failure_mode\": null,\n \"rationale\": \"The model's answer 'MinaLima props' correctly identifies the source (MinaLima) and implies their work on Harry Potter film props, which aligns with the ground truth that it is 'MinaLima's creation from the Harry Potter films' without adding any incorrect claims.\"\n}\n```", "parsed_verdict": "correct", "parsed_failure_mode": null, "parsed_rationale": "The model's answer 'MinaLima props' correctly identifies the source (MinaLima) and implies their work on Harry Potter film props, which aligns with the ground truth that it is 'MinaLima's creation from the Harry Potter films' without adding any incorrect claims.", "opus_verdict_ref": "correct", "gpt_verdict_ref": "correct"}
{"instance_id": "locomo_conv-41_q065", "cell": "retrieval", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 6626, "prompt_tokens": 461, "completion_tokens": 253, "raw_text": "{\n \"verdict\": \"correct\",\n \"failure_mode\": null,\n \"rationale\": \"The model correctly identifies education and infrastructure as John's main focuses in local politics, matching the ground truth without any incorrect additions.\"\n}", "parsed_verdict": "correct", "parsed_failure_mode": null, "parsed_rationale": "The model correctly identifies education and infrastructure as John's main focuses in local politics, matching the ground truth without any incorrect additions.", "opus_verdict_ref": "correct", "gpt_verdict_ref": "correct"}
{"instance_id": "locomo_conv-50_q086", "cell": "agentic", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 39796, "prompt_tokens": 515, "completion_tokens": 2489, "raw_text": "{\n \"verdict\": \"incorrect\",\n \"failure_mode\": \"F3\",\n \"rationale\": \"The model provided a system message about tool usage limits instead of stating the correct answer (skiing), which constitutes an incorrect response not covered by the other failure modes.\"\n}", "parsed_verdict": "incorrect", "parsed_failure_mode": "F3", "parsed_rationale": "The model provided a system message about tool usage limits instead of stating the correct answer (skiing), which constitutes an incorrect response not covered by the other failure modes.", "opus_verdict_ref": "incorrect", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-26_q059", "cell": "agentic", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 15569, "prompt_tokens": 619, "completion_tokens": 629, "raw_text": "{\n \"verdict\": \"incorrect\",\n \"failure_mode\": \"F4\",\n \"rationale\": \"The model fabricates a detail about a necklace from Sweden representing 'love, faith and strength' that does not appear in the ground-truth context, while the church painting connection is correctly derived from the context.\"\n}", "parsed_verdict": "incorrect", "parsed_failure_mode": "F4", "parsed_rationale": "The model fabricates a detail about a necklace from Sweden representing 'love, faith and strength' that does not appear in the ground-truth context, while the church painting connection is correctly derived from the context.", "opus_verdict_ref": "incorrect", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-44_q000", "cell": "agentic", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 22383, "prompt_tokens": 477, "completion_tokens": 638, "raw_text": "{\n \"verdict\": \"incorrect\",\n \"failure_mode\": \"F3\",\n \"rationale\": \"The model failed to provide an answer to the question, instead outputting a system message indicating it reached its maximum tool turns without generating a response.\"\n}", "parsed_verdict": "incorrect", "parsed_failure_mode": "F3", "parsed_rationale": "The model failed to provide an answer to the question, instead outputting a system message indicating it reached its maximum tool turns without generating a response.", "opus_verdict_ref": "incorrect", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-41_q080", "cell": "agentic", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 21678, "prompt_tokens": 500, "completion_tokens": 519, "raw_text": "{\n \"verdict\": \"incorrect\",\n \"failure_mode\": \"F2\",\n \"rationale\": \"The model correctly identifies that Maria made a painting but omits the key detail that it was specifically a painting of a castle on a hill, which is required by the ground truth answer.\"\n}", "parsed_verdict": "incorrect", "parsed_failure_mode": "F2", "parsed_rationale": "The model correctly identifies that Maria made a painting but omits the key detail that it was specifically a painting of a castle on a hill, which is required by the ground truth answer.", "opus_verdict_ref": "incorrect", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-42_q001", "cell": "agentic", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 9621, "prompt_tokens": 762, "completion_tokens": 364, "raw_text": "{\n \"verdict\": \"incorrect\",\n \"failure_mode\": \"F2\",\n \"rationale\": \"The model correctly identifies watching movies as a shared interest but fails to include making desserts, which is also a shared interest according to the ground-truth context.\"\n}", "parsed_verdict": "incorrect", "parsed_failure_mode": "F2", "parsed_rationale": "The model correctly identifies watching movies as a shared interest but fails to include making desserts, which is also a shared interest according to the ground-truth context.", "opus_verdict_ref": "incorrect", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-43_q155", "cell": "agentic", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 9115, "prompt_tokens": 483, "completion_tokens": 389, "raw_text": "{\n \"verdict\": \"correct\",\n \"failure_mode\": null,\n \"rationale\": \"The model's answer 'great' is a synonym that captures the same positive sentiment as 'huge success' in the ground truth.\"\n}", "parsed_verdict": "correct", "parsed_failure_mode": null, "parsed_rationale": "The model's answer 'great' is a synonym that captures the same positive sentiment as 'huge success' in the ground truth.", "opus_verdict_ref": "correct", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-50_q015", "cell": "agentic", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 39909, "prompt_tokens": 573, "completion_tokens": 484, "raw_text": "{\n \"verdict\": \"correct\",\n \"failure_mode\": null,\n \"rationale\": \"The model's answer 'to relax and recharge' captures the core meaning of the ground truth 'relaxes and calms him' - 'relax' directly matches 'relaxes', and 'recharge' is an acceptable synonym for the calming/restorative effect described in the context.\"\n}", "parsed_verdict": "correct", "parsed_failure_mode": null, "parsed_rationale": "The model's answer 'to relax and recharge' captures the core meaning of the ground truth 'relaxes and calms him' - 'relax' directly matches 'relaxes', and 'recharge' is an acceptable synonym for the calming/restorative effect described in the context.", "opus_verdict_ref": "correct", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-48_q020", "cell": "agentic", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 9349, "prompt_tokens": 719, "completion_tokens": 393, "raw_text": "{\n \"verdict\": \"incorrect\",\n \"failure_mode\": \"F2\",\n \"rationale\": \"The model correctly identified Detroit, Overcooked 2, and It Takes Two, but missed Walking Dead and Battlefield 1 which are also in the ground truth.\"\n}", "parsed_verdict": "incorrect", "parsed_failure_mode": "F2", "parsed_rationale": "The model correctly identified Detroit, Overcooked 2, and It Takes Two, but missed Walking Dead and Battlefield 1 which are also in the ground truth.", "opus_verdict_ref": "incorrect", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-48_q055", "cell": "agentic", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 20702, "prompt_tokens": 512, "completion_tokens": 666, "raw_text": "{\n \"verdict\": \"incorrect\",\n \"failure_mode\": \"F4\",\n \"rationale\": \"The model answered 'Susie' but neither 'Susie' nor 'Seraphim' appear in the ground-truth context, indicating the model hallucinated the pet name.\"\n}", "parsed_verdict": "incorrect", "parsed_failure_mode": "F4", "parsed_rationale": "The model answered 'Susie' but neither 'Susie' nor 'Seraphim' appear in the ground-truth context, indicating the model hallucinated the pet name.", "opus_verdict_ref": "incorrect", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-26_q128", "cell": "agentic", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 4544, "prompt_tokens": 494, "completion_tokens": 185, "raw_text": "```json\n{\n \"verdict\": \"correct\",\n \"failure_mode\": null,\n \"rationale\": \"The model's answer 'a rainbow sidewalk' exactly matches the ground-truth answer, both correctly identifying what Caroline found during her walk.\"\n}\n```", "parsed_verdict": "correct", "parsed_failure_mode": null, "parsed_rationale": "The model's answer 'a rainbow sidewalk' exactly matches the ground-truth answer, both correctly identifying what Caroline found during her walk.", "opus_verdict_ref": "correct", "gpt_verdict_ref": "correct"}
{"instance_id": "locomo_conv-50_q010", "cell": "agentic", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 28878, "prompt_tokens": 502, "completion_tokens": 501, "raw_text": "{\n \"verdict\": \"correct\",\n \"failure_mode\": null,\n \"rationale\": \"The model's answer 'last week' is factually correct and equivalent to the ground truth 'a week before 16 May, 2023', as Calvin stated his place got flooded 'last week' during the session on 16 May, 2023.\"\n}", "parsed_verdict": "correct", "parsed_failure_mode": null, "parsed_rationale": "The model's answer 'last week' is factually correct and equivalent to the ground truth 'a week before 16 May, 2023', as Calvin stated his place got flooded 'last week' during the session on 16 May, 2023.", "opus_verdict_ref": "correct", "gpt_verdict_ref": "correct"}
{"instance_id": "locomo_conv-50_q078", "cell": "agentic", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 3384, "prompt_tokens": 485, "completion_tokens": 112, "raw_text": "{\n \"verdict\": \"correct\",\n \"failure_mode\": null,\n \"rationale\": \"The model’s answer matches the ground‑truth answer exactly, stating the necklace was meant to remind Calvin of why he keeps hustling as a musician.\"\n}", "parsed_verdict": "correct", "parsed_failure_mode": null, "parsed_rationale": "The model’s answer matches the ground‑truth answer exactly, stating the necklace was meant to remind Calvin of why he keeps hustling as a musician.", "opus_verdict_ref": "correct", "gpt_verdict_ref": "correct"}
{"instance_id": "locomo_conv-47_q092", "cell": "agentic", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 11980, "prompt_tokens": 488, "completion_tokens": 539, "raw_text": "{\n \"verdict\": \"incorrect\",\n \"failure_mode\": \"F2\",\n \"rationale\": \"The model correctly identifies that John created a tool/application, but it misses the specific required detail that the application was on smartphones.\"\n}", "parsed_verdict": "incorrect", "parsed_failure_mode": "F2", "parsed_rationale": "The model correctly identifies that John created a tool/application, but it misses the specific required detail that the application was on smartphones.", "opus_verdict_ref": "incorrect", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-30_q057", "cell": "agentic", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 19907, "prompt_tokens": 520, "completion_tokens": 549, "raw_text": "{\n \"verdict\": \"incorrect\",\n \"failure_mode\": \"F5\",\n \"rationale\": \"The model's answer provides motivational self-help advice about faith in oneself and support systems, which is completely different from the ground truth's advice about building customer relationships, creating brand image, and staying positive.\"\n}", "parsed_verdict": "incorrect", "parsed_failure_mode": "F5", "parsed_rationale": "The model's answer provides motivational self-help advice about faith in oneself and support systems, which is completely different from the ground truth's advice about building customer relationships, creating brand image, and staying positive.", "opus_verdict_ref": "incorrect", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-30_q029", "cell": "agentic", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 13857, "prompt_tokens": 565, "completion_tokens": 267, "raw_text": "{\n \"verdict\": \"incorrect\",\n \"failure_mode\": \"F1\",\n \"rationale\": \"The model explicitly stated 'unknown' when the cities (Paris and Rome) could have been extracted from the provided supporting context.\"\n}", "parsed_verdict": "incorrect", "parsed_failure_mode": "F1", "parsed_rationale": "The model explicitly stated 'unknown' when the cities (Paris and Rome) could have been extracted from the provided supporting context.", "opus_verdict_ref": "incorrect", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-44_q066", "cell": "agentic", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 5477, "prompt_tokens": 512, "completion_tokens": 215, "raw_text": "{\n \"verdict\": \"correct\",\n \"failure_mode\": null,\n \"rationale\": \"The model's answer 'a smaller dog' exactly matches the ground truth, correctly identifying that Andrew was looking for a smaller dog based on his apartment living space.\"\n}", "parsed_verdict": "correct", "parsed_failure_mode": null, "parsed_rationale": "The model's answer 'a smaller dog' exactly matches the ground truth, correctly identifying that Andrew was looking for a smaller dog based on his apartment living space.", "opus_verdict_ref": "correct", "gpt_verdict_ref": "correct"}
{"instance_id": "locomo_conv-44_q002", "cell": "agentic", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 15222, "prompt_tokens": 742, "completion_tokens": 560, "raw_text": "{\n \"verdict\": \"incorrect\",\n \"failure_mode\": \"F2\",\n \"rationale\": \"The model correctly identified board games as an indoor activity Andrew pursued with his girlfriend but missed the other three activities (volunteering at pet shelter, wine tasting, and growing flowers) that appear in the ground truth, without stating any incorrect information.\"\n}", "parsed_verdict": "incorrect", "parsed_failure_mode": "F2", "parsed_rationale": "The model correctly identified board games as an indoor activity Andrew pursued with his girlfriend but missed the other three activities (volunteering at pet shelter, wine tasting, and growing flowers) that appear in the ground truth, without stating any incorrect information.", "opus_verdict_ref": "incorrect", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-47_q046", "cell": "agentic", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 9876, "prompt_tokens": 463, "completion_tokens": 599, "raw_text": "{\n \"verdict\": \"correct\",\n \"failure_mode\": null,\n \"rationale\": \"The model correctly identifies the timing as 'tomorrow', which aligns with the ground truth of August 27, 2022, since John said 'See you tomorrow' on August 26, 2022, making tomorrow August 27, 2022.\"\n}", "parsed_verdict": "correct", "parsed_failure_mode": null, "parsed_rationale": "The model correctly identifies the timing as 'tomorrow', which aligns with the ground truth of August 27, 2022, since John said 'See you tomorrow' on August 26, 2022, making tomorrow August 27, 2022.", "opus_verdict_ref": "incorrect", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-43_q076", "cell": "agentic", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 26616, "prompt_tokens": 496, "completion_tokens": 520, "raw_text": "```json\n{\n \"verdict\": \"correct\",\n \"failure_mode\": null,\n \"rationale\": \"The model's answer 'A picture from MinaLima' correctly identifies the source of the picture, which matches the ground truth that it is MinaLima's creation, and minor phrasing differences are acceptable.\"\n}\n```", "parsed_verdict": "correct", "parsed_failure_mode": null, "parsed_rationale": "The model's answer 'A picture from MinaLima' correctly identifies the source of the picture, which matches the ground truth that it is MinaLima's creation, and minor phrasing differences are acceptable.", "opus_verdict_ref": "correct", "gpt_verdict_ref": "correct"}
{"instance_id": "locomo_conv-41_q065", "cell": "agentic", "provider": "minimax", "model_id": "minimax/minimax-m2.7", "routing": "openrouter_direct_http", "litellm_alias_registered": "minimax-m27-via-openrouter", "http_status": 200, "error": null, "retries": 0, "latency_ms": 10691, "prompt_tokens": 461, "completion_tokens": 330, "raw_text": "{\n \"verdict\": \"correct\",\n \"failure_mode\": null,\n \"rationale\": \"The model correctly identified education and infrastructure as John's main focuses, matching the ground truth without any incorrect additions.\"\n}", "parsed_verdict": "correct", "parsed_failure_mode": null, "parsed_rationale": "The model correctly identified education and infrastructure as John's main focuses, matching the ground truth without any incorrect additions.", "opus_verdict_ref": "correct", "gpt_verdict_ref": "correct"}

View File

@@ -1,6 +0,0 @@
{"instance_id": "locomo_conv-43_q155", "cell": "agentic", "provider": "minimax", "alias": "minimax-m27-via-openrouter", "routing": "openrouter_direct_http", "http_status": 200, "error": null, "latency_ms": 28396, "prompt_tokens": 480, "completion_tokens": 635, "parsed_verdict": "correct", "parsed_failure_mode": null, "parsed_rationale": "The model's answer 'great' is a direct synonym for John's stated feeling ('It was great!') and captures the same positive sentiment as 'huge success' from the ground truth, making it an acceptable equivalent formulation.", "raw_text": "{\n \"verdict\": \"correct\",\n \"failure_mode\": null,\n \"rationale\": \"The model's answer 'great' is a direct synonym for John's stated feeling ('It was great!') and captures the same positive sentiment as 'huge success' from the ground truth, making it an acceptable equivalent formulation.\"\n}", "opus_verdict_ref": "correct", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-43_q155", "cell": "agentic", "provider": "kimi", "alias": "kimi-k26-direct", "routing": "moonshot_direct_http", "http_status": 0, "error": "TimeoutError: The read operation timed out", "latency_ms": 60112, "prompt_tokens": null, "completion_tokens": null, "parsed_verdict": null, "parsed_failure_mode": null, "parsed_rationale": null, "raw_text": "", "opus_verdict_ref": "correct", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-50_q015", "cell": "agentic", "provider": "minimax", "alias": "minimax-m27-via-openrouter", "routing": "openrouter_direct_http", "http_status": 200, "error": null, "latency_ms": 17311, "prompt_tokens": 573, "completion_tokens": 558, "parsed_verdict": "correct", "parsed_failure_mode": null, "parsed_rationale": "The model's answer captures the primary reason (relax) and adds a conceptually aligned detail (recharge), which is acceptable as minor elaboration that is not contradictory to the ground truth of relaxing and calming.", "raw_text": "{\"verdict\":\"correct\",\"failure_mode\":null,\"rationale\":\"The model's answer captures the primary reason (relax) and adds a conceptually aligned detail (recharge), which is acceptable as minor elaboration that is not contradictory to the ground truth of relaxing and calming.\"}", "opus_verdict_ref": "correct", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-50_q015", "cell": "agentic", "provider": "kimi", "alias": "kimi-k26-direct", "routing": "moonshot_direct_http", "http_status": 200, "error": null, "latency_ms": 15158, "prompt_tokens": 550, "completion_tokens": 351, "parsed_verdict": "correct", "parsed_failure_mode": null, "parsed_rationale": "The model's answer captures the ground truth that Dave visits parks to relax, and 'recharge' is an acceptable synonymous formulation of the calming, restorative benefit described in the context without introducing any incorrect claims.", "raw_text": "{\n \"verdict\": \"correct\",\n \"failure_mode\": null,\n \"rationale\": \"The model's answer captures the ground truth that Dave visits parks to relax, and 'recharge' is an acceptable synonymous formulation of the calming, restorative benefit described in the context without introducing any incorrect claims.\"\n}", "opus_verdict_ref": "correct", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-43_q155", "cell": "full-context", "provider": "minimax", "alias": "minimax-m27-via-openrouter", "routing": "openrouter_direct_http", "http_status": 200, "error": null, "latency_ms": 25481, "prompt_tokens": 480, "completion_tokens": 518, "parsed_verdict": "correct", "parsed_failure_mode": null, "parsed_rationale": "The model's answer 'great' is a direct synonym for 'a huge success' from the ground truth, both expressing John's positive feeling about jogging without pain, and contains no incorrect claims.", "raw_text": "{\n \"verdict\": \"correct\",\n \"failure_mode\": null,\n \"rationale\": \"The model's answer 'great' is a direct synonym for 'a huge success' from the ground truth, both expressing John's positive feeling about jogging without pain, and contains no incorrect claims.\"\n}", "opus_verdict_ref": "correct", "gpt_verdict_ref": "incorrect"}
{"instance_id": "locomo_conv-43_q155", "cell": "full-context", "provider": "kimi", "alias": "kimi-k26-direct", "routing": "moonshot_direct_http", "http_status": 200, "error": null, "latency_ms": 47213, "prompt_tokens": 456, "completion_tokens": 1617, "parsed_verdict": "correct", "parsed_failure_mode": null, "parsed_rationale": "The model's answer 'great' is explicitly stated in the ground-truth context as John's direct feeling about jogging without pain, making it factually correct and an acceptable equivalent formulation.", "raw_text": "{\n \"verdict\": \"correct\",\n \"failure_mode\": null,\n \"rationale\": \"The model's answer 'great' is explicitly stated in the ground-truth context as John's direct feeling about jogging without pain, making it factually correct and an acceptable equivalent formulation.\"\n}", "opus_verdict_ref": "correct", "gpt_verdict_ref": "incorrect"}

View File

@@ -1,48 +0,0 @@
# Manifest v6 Phase 1 κ Re-Calibration — Exit Memo
**Date:** 2026-04-24 **Target:** PM-RATIFY-V6-KAPPA **v6 anchor:** `60d061e`
## Verdict: **PASS** (κ_conservative_trio = 0.7878 ≥ 0.70)
## Three pairwise Cohen's κ (n=100)
| Pair | Agree | κ |
|------|-------|-----|
| Opus vs GPT | 93/100 | **0.8480** |
| Opus vs MiniMax | 93/100 | **0.8549** |
| GPT vs MiniMax | 90/100 | **0.7878** (min) |
**Notable:** MiniMax agrees with Opus *more* than GPT does (0.8549 > 0.8480). Swap validated empirically. v5 historical baseline (Fleiss κ=0.7458 three-way) is consistent with v6's pairwise Opus-GPT of 0.8480 — sanity check PASSES.
## MiniMax operational metrics
| Metric | Target | Actual | Status |
|--------|--------|--------|--------|
| Parse rate | ≥95/100 | **100/100** | MET |
| Latency p50 | ≤25 s | **11.9 s** | MET |
| Latency p95 | — | 31.4 s | — |
| OR routing errors | <5% | **0/100** | MET |
| Retries | — | 0 | clean |
## Per-cell κ_trio (lowest of three pairwise per cell)
| Cell | κ_trio | Notes |
|------|--------|-------|
| no-context | 1.0000 | perfect unanimity |
| retrieval | 0.8936 | strong |
| full-context | 0.7000 | acceptable |
| oracle-context | 0.7059 | acceptable |
| **agentic** | **0.6875** | **BORDERLINE at cell-level** (GPT-MiniMax pair) |
Agentic dips into borderline band at cell level — flag for PM but does not block PASS verdict since aggregate trio meets bar.
## Cost / wall-clock
- 100 calls, 0 retries, 0 failures
- Tokens: 53,855 prompt + 48,920 completion
- **Cost actual: ~$0.075** (cap $30 Phase 1)
- **Wall-clock: ~24.2 min** (14:09-14:34 UTC) (cap 90 min)
## PM next step
Phase 1 complete. Awaiting `PM-RATIFY-V6-KAPPA` for Phase 2 (N=400) authorization. `cc1_state: HALTED`.

View File

@@ -1,2 +0,0 @@
# Probe working data — contains a COPY OF THE REAL PERSONAL MIND. Never commit.
data/

View File

@@ -1,144 +0,0 @@
/**
* D1 chunk-lane eval probe — long-frame needle retrieval (2026-06-12).
*
* PRE-REGISTERED DESIGN (before any run):
* - LoCoMo was REJECTED as the ruler: its frames max at ~1000 chars, below
* the 2000-char chunk threshold — every frame yields one chunk ≡ the whole
* frame, so a LoCoMo A/B measures noise by construction.
* - Honest corpus: a COPY of the real production personal mind
* (~/.hive-mind/personal.mind: 471 frames, 129 >2k chars, max 25.7k).
* - Both cells re-embedded from scratch with the SAME local embedder
* (Ollama nomic-embed-text-8k, 1024-dim) — the original mind's vectors are
* mock-fingerprinted and unusable; equal footing by construction.
* - Needle = a verbatim sentence drawn from the 50–90% depth of a long frame
* (where whole-frame embedding signal dilutes). Verbatim is fair: vector
* lanes never touch FTS, and both cells get the identical query.
* - Cell A (control): HybridSearch.vectorSearch (whole-frame vectors)
* - Cell B (treatment): HybridSearch.vectorSearchChunks (chunk vectors,
* best-chunk-per-frame) — called directly; no env flag needed.
* - Metric: hit@1 / hit@5 / hit@10 of the source frame, paired per needle.
* - Decision rule (pre-registered): flip default-ON only if chunk hit@5
* beats whole-frame hit@5 by a clear margin (paired; report exact counts).
*
* Run: node benchmarks/chunk-probe/run-probe.mjs (from waggle-os-w4 root)
* Requires: Ollama up with nomic-embed-text-8k; hive-mind-core dist built.
* Cost: $0 (all local). The source mind is NEVER touched (sqlite backup API).
*/
import { pathToFileURL, fileURLToPath } from 'node:url';
import path from 'node:path';
import fs from 'node:fs';
const ROOT = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..', '..');
const CORE = pathToFileURL(path.join(ROOT, 'packages/hive-mind-core/dist/index.js')).href;
const SOURCE_MIND = 'C:/Users/MarkoMarkovic/.hive-mind/personal.mind';
const PROBE_DIR = path.join(ROOT, 'benchmarks/chunk-probe/data');
const PROBE_MIND = path.join(PROBE_DIR, 'probe.mind');
const MIN_FRAME_LEN = 2500; // only frames where chunking can matter
const MAX_NEEDLES = 60;
const K = 10;
const { MindDB, HybridSearch, createOllamaEmbedder, rechunkAllFrames } = await import(CORE);
// ── 1. Copy the mind (sqlite backup API — consistent snapshot, source untouched)
fs.mkdirSync(PROBE_DIR, { recursive: true });
for (const suffix of ['', '-wal', '-shm']) {
const p = PROBE_MIND + suffix;
if (fs.existsSync(p)) fs.rmSync(p);
}
{
const BetterSqlite3 = (await import(pathToFileURL(path.join(ROOT, 'node_modules/better-sqlite3/lib/index.js')).href)).default;
const src = new BetterSqlite3(SOURCE_MIND, { readonly: true });
await src.backup(PROBE_MIND);
src.close();
}
console.log('[probe] mind copied to', PROBE_MIND);
// ── 2. Open with mono substrate + real local embedder
const db = new MindDB(PROBE_MIND);
const embedder = createOllamaEmbedder({ model: 'nomic-embed-text-8k', targetDimensions: 1024 });
const search = new HybridSearch(db, embedder);
// ── 3. Wipe mock vectors, re-embed all frames (whole-frame control vectors)
db.recreateVecTables(1024);
const raw = db.getDatabase();
const frames = raw.prepare('SELECT id, content FROM memory_frames ORDER BY id').all();
console.log(`[probe] re-embedding ${frames.length} frames (whole-frame vectors)…`);
const BATCH = 16;
for (let i = 0; i < frames.length; i += BATCH) {
// Cap at 4,000 chars. LIVE FINDING: the '-8k' model's underlying nomic-bert
// architecture caps at 2048 TOKENS (ollama show: nomic-bert.context_length
// 2048) and Ollama enforces it — R5's maxEmbedCharsForModel trusts the
// model NAME (24k chars) and would 400, then per-text-fallback every long
// frame to a MOCK vector in production. Whole-frame embedding beyond the
// real cap is impossible — that blindness is what the chunk lane (≤2k
// chars ≈ 500 tokens) structurally fixes.
const batch = frames.slice(i, i + BATCH).map(f => ({ id: f.id, content: f.content.slice(0, 4_000) }));
await search.indexFramesBatch(batch);
if ((i / BATCH) % 5 === 0) process.stdout.write(` ${Math.min(i + BATCH, frames.length)}/${frames.length}\r`);
}
console.log(`\n[probe] whole-frame vectors: ${raw.prepare('SELECT COUNT(*) n FROM memory_frames_vec').get().n}`);
// ── 4. Chunk backfill (treatment vectors; flag-independent helper)
const rechunk = await rechunkAllFrames(db, search);
console.log('[probe] rechunk:', JSON.stringify(rechunk));
const chunkStats = raw.prepare(
`SELECT COUNT(*) chunks, SUM(cnt > 1) multi FROM (
SELECT frame_id, COUNT(*) cnt FROM memory_frame_chunks GROUP BY frame_id)`
).get();
console.log(`[probe] chunks: ${chunkStats.chunks} total, ${chunkStats.multi} frames with >1 chunk`);
// ── 5. Needle extraction: verbatim sentence from 50–90% depth of long frames
function extractNeedle(content) {
const start = Math.floor(content.length * 0.5);
const end = Math.floor(content.length * 0.9);
const slice = content.slice(start, end);
const sentences = slice.split(/(?<=[.!?])\s+|\n+/).map(s => s.trim());
const candidates = sentences.filter(s => s.length >= 80 && s.length <= 250 && s.split(/\s+/).length >= 10);
if (candidates.length === 0) return null;
// deterministic pick: the longest candidate (most distinctive)
return candidates.sort((a, b) => b.length - a.length)[0];
}
const longFrames = frames.filter(f => f.content.length >= MIN_FRAME_LEN);
const needles = [];
for (const f of longFrames) {
const n = extractNeedle(f.content);
// beyondCap: needle text starts past the 8k whole-frame embed cap — content
// structurally invisible to the whole-frame vector (chunking's strongest case).
if (n) needles.push({ frameId: f.id, needle: n, frameLen: f.content.length, beyondCap: f.content.indexOf(n) >= 4_000 });
if (needles.length >= MAX_NEEDLES) break;
}
console.log(`[probe] ${needles.length} needles from ${longFrames.length} long frames (>=${MIN_FRAME_LEN} chars)`);
// ── 6. Paired cells
let stats = { a1: 0, a5: 0, a10: 0, b1: 0, b5: 0, b10: 0, bOnly5: 0, aOnly5: 0 };
let strat = { withinCap: { n: 0, a5: 0, b5: 0 }, beyondCap: { n: 0, a5: 0, b5: 0 } };
const rows = [];
for (const { frameId, needle, frameLen, beyondCap } of needles) {
const a = await search.vectorSearch(needle, K);
const b = (await search.vectorSearchChunks(needle, K)) ?? [];
const ra = a.indexOf(frameId);
const rb = b.indexOf(frameId);
if (ra === 0) stats.a1++; if (ra >= 0 && ra < 5) stats.a5++; if (ra >= 0) stats.a10++;
if (rb === 0) stats.b1++; if (rb >= 0 && rb < 5) stats.b5++; if (rb >= 0) stats.b10++;
const aHit5 = ra >= 0 && ra < 5, bHit5 = rb >= 0 && rb < 5;
if (bHit5 && !aHit5) stats.bOnly5++;
if (aHit5 && !bHit5) stats.aOnly5++;
const s = beyondCap ? strat.beyondCap : strat.withinCap;
s.n++; if (aHit5) s.a5++; if (bHit5) s.b5++;
rows.push({ frameId, frameLen, beyondCap, rankWhole: ra, rankChunk: rb, needle: needle.slice(0, 80) });
}
// ── 7. Report
const n = needles.length;
console.log('\n========== D1 CHUNK-LANE PROBE RESULT ==========');
console.log(`corpus: copy of personal.mind | needles: ${n} | embedder: nomic-embed-text-8k/1024`);
console.log(` hit@1 hit@5 hit@10`);
console.log(`whole-frame ${stats.a1}/${n} ${stats.a5}/${n} ${stats.a10}/${n}`);
console.log(`chunk-lane ${stats.b1}/${n} ${stats.b5}/${n} ${stats.b10}/${n}`);
console.log(`paired hit@5: chunk-only wins ${stats.bOnly5}, whole-only wins ${stats.aOnly5} (discordant pairs)`);
console.log(`strata hit@5: within-cap n=${strat.withinCap.n} whole ${strat.withinCap.a5} vs chunk ${strat.withinCap.b5}` +
` | beyond-cap n=${strat.beyondCap.n} whole ${strat.beyondCap.a5} vs chunk ${strat.beyondCap.b5}`);
fs.writeFileSync(path.join(PROBE_DIR, 'probe-result.json'), JSON.stringify({ n, stats, rows }, null, 2));
console.log('[probe] per-needle detail → benchmarks/chunk-probe/data/probe-result.json');
db.close();

View File

@@ -1,2 +0,0 @@
# LoCoMo / LongMemEval data lands here. Contents are gitignored.
# Run the dataset download script (Week 1) to populate.

View File

@@ -1,47 +0,0 @@
{
"dataset_version": "9311bba4d5ebe22ef11fbbdfc1f837a41bed9b80b5c169af1885fc2a50b854a1",
"instance_count": 400,
"chat_size": "128K",
"built_at": "2026-06-04T19:29:05.930Z",
"source": "mohammadtavakoli78/BEAM (GitHub)",
"source_reference": "Tavakoli, Salemi, Ye, Abdalla, Zamani, Mitchell 2024, \"Beyond a Million Tokens: Benchmarking and Enhancing Long-Term Memory in LLMs\" (arXiv:2510.27246, ICLR 2026)",
"beam_chats_path": "/home/user/workspace/BEAM/chats",
"conversations_processed": 20,
"chat_size_dir_alias": "100K",
"canonicalisation": {
"sort_order": "instance_id ascending",
"field_order": [
"instance_id",
"conversation_id",
"question",
"expected",
"context",
"memory_ability",
"chat_size",
"conversation_index"
],
"line_terminator": "\\n",
"trailing_newline": true,
"encoding": "utf-8",
"no_bom": true
},
"distribution_by_memory_ability": {
"abstention": 40,
"contradiction_resolution": 40,
"event_ordering": 40,
"information_extraction": 40,
"instruction_following": 40,
"knowledge_update": 40,
"multi_session_reasoning": 40,
"preference_following": 40,
"summarization": 40,
"temporal_reasoning": 40
},
"skip_stats": {
"missingChat": 0,
"missingProbing": 0,
"missingQuestion": 0,
"missingAnswer": 0,
"noTurns": 0
}
}

View File

@@ -1,51 +0,0 @@
{
"dataset_version": "cc81ed8f7624261a2fa43a335eb46159b7665074b82ddfb4e2c0783fbf2caa46",
"schema_version": 2,
"instances_with_rubric": 700,
"total_nuggets": 2415,
"instance_count": 700,
"chat_size": "1M",
"built_at": "2026-07-06T13:34:37.017Z",
"source": "mohammadtavakoli78/BEAM (GitHub)",
"source_reference": "Tavakoli, Salemi, Ye, Abdalla, Zamani, Mitchell 2024, \"Beyond a Million Tokens: Benchmarking and Enhancing Long-Term Memory in LLMs\" (arXiv:2510.27246, ICLR 2026)",
"beam_chats_path": "D:\\Projects\\BEAM\\chats",
"conversations_processed": 35,
"chat_size_dir_alias": "1M",
"canonicalisation": {
"sort_order": "instance_id ascending",
"field_order": [
"instance_id",
"conversation_id",
"question",
"expected",
"context",
"memory_ability",
"chat_size",
"conversation_index",
"rubric"
],
"line_terminator": "\\n",
"trailing_newline": true,
"encoding": "utf-8",
"no_bom": true
},
"distribution_by_memory_ability": {
"abstention": 70,
"contradiction_resolution": 70,
"event_ordering": 70,
"information_extraction": 70,
"instruction_following": 70,
"knowledge_update": 70,
"multi_session_reasoning": 70,
"preference_following": 70,
"summarization": 70,
"temporal_reasoning": 70
},
"skip_stats": {
"missingChat": 0,
"missingProbing": 0,
"missingQuestion": 0,
"missingAnswer": 0,
"noTurns": 0
}
}

View File

@@ -1,18 +0,0 @@
# Failure-mode judge calibration set
# brief: PM-Waggle-OS/briefs/2026-04-20-cc-preflight-prep-tasks.md Task 2
# locked: decisions/2026-04-20-failure-mode-oq-resolutions-locked.md §OQ-FM-3
# source: https://raw.githubusercontent.com/snap-research/locomo/main/data/locomo10.json
# seed=43 (non-overlapping with preflight-locomo-50.json seed=42)
# distribution: single-hop=3, multi-hop=3, temporal=2, open-ended=2 (total=10)
# human_label.{verdict,failure_mode,rationale} are left null. PM labels first pass; CC validates second pass.
# Judge activates Stage 1 only after ≥8/10 match against human_label.
{"id":"locomo_conv-26_q109","category":"single-hop","context":"Session 8 (1:51 pm on 15 July, 2023):\nMelanie: Hey Caroline, it's been super busy here. So much since we talked! Last Fri I finally took my kids to a pottery workshop. We all made our own pots, it was fun and therapeutic! [image: a photography of a group of children making clay sculptures in a classroom]","question":"What did Mel and her kids make during the pottery workshop?","ground_truth_answer":"pots","locomo_metadata":{"sample_id":"conv-26","qa_index":109,"locomo_category":4,"evidence":["D8:2"],"speaker_a":"Caroline","speaker_b":"Melanie"},"human_label":{"verdict":null,"failure_mode":null,"rationale":null}}
{"id":"locomo_conv-41_q123","category":"single-hop","context":"Session 23 (6:29 pm on 7 July, 2023):\nMaria: For sure! We can keep doing great stuff and making a difference. Well, I'm off to have dinner with some friends from the gym. Talk to you later!","question":"What did Maria plan to do later on the evening of 7 July, 2023?","ground_truth_answer":"have dinner with friends from the gym","locomo_metadata":{"sample_id":"conv-41","qa_index":123,"locomo_category":4,"evidence":["D23:14"],"speaker_a":"John","speaker_b":"Maria"},"human_label":{"verdict":null,"failure_mode":null,"rationale":null}}
{"id":"locomo_conv-50_q141","category":"single-hop","context":"Session 27 (10:49 am on 29 October, 2023):\nDave: That's Boston, Cal! Check this out, I took this picture last month, and got a great shot - it was stunning! [image: a photography of a clock tower in a city with buildings]","question":"Which city is featured in the photograph Dave showed Calvin?","ground_truth_answer":"Boston","locomo_metadata":{"sample_id":"conv-50","qa_index":141,"locomo_category":4,"evidence":["D27:6"],"speaker_a":"Calvin","speaker_b":"Dave"},"human_label":{"verdict":null,"failure_mode":null,"rationale":null}}
{"id":"locomo_conv-42_q030","category":"multi-hop","context":"Session 2 (2:01 pm on 23 January, 2022):\nJoanna: Woo! I finally finished my first full screenplay and printed it last Friday. I've been working on for a while, such a relief to have it all done! [image: a photography of a book with a page of text on it]\n\nSession 17 (2:34 pm on 10 July, 2022):\nJoanna: I will! I actually started on a book recently since my movie did well! [image: a photo of a person holding a notebook with a handwritten page]\n\nSession 18 (6:12 pm on 14 August, 2022):\nJoanna: Hey Nate, long time no talk! I've been busy with writing projects and really going all out with it. It's been the best thing ever - a mix of highs and lows - and my journal's pretty much my rock. Writing's such a huge part of me now. [image: a photo of a notebook with a bunch of stickers on it]\nJoanna: Yep. Last week, someone wrote me a letter after reading an online blog post I made about a hard moment in my life. Their words touched me; they said my story had brought them comfort. It was awesome to realize my words had that kind of power. It reminded me why I love writing. [image: a photo of a note written to a person on a piece of paper]","question":"What kind of writings does Joanna do?","ground_truth_answer":"Screenplays,books, online blog posts, journal","locomo_metadata":{"sample_id":"conv-42","qa_index":30,"locomo_category":1,"evidence":["D2:3","D17:14","D18:1","D18:5"],"speaker_a":"Joanna","speaker_b":"Nate"},"human_label":{"verdict":null,"failure_mode":null,"rationale":null}}
{"id":"locomo_conv-49_q015","category":"multi-hop","context":"Session 3 (3:55 pm on 6 June, 2023):\nSam: Nah, no changes for me. Still enjoying my soda and candy, although I know it's not the best habit to have.","question":"What kind of unhealthy snacks does Sam enjoy eating?","ground_truth_answer":"soda, candy","locomo_metadata":{"sample_id":"conv-49","qa_index":15,"locomo_category":1,"evidence":["D3:4"],"speaker_a":"Evan","speaker_b":"Sam"},"human_label":{"verdict":null,"failure_mode":null,"rationale":null}}
{"id":"locomo_conv-41_q036","category":"multi-hop","context":"Session 8 (6:03 pm on 6 March, 2023):\nJohn: I just try to find things that we'll have fun with, like a walk or picnic in the park, or finding events in our town and beyond. Just last week, I found a violin concert that we all enjoyed. It's all about making memories together.\n\nSession 20 (12:21 am on 27 June, 2023):\nJohn: That's great, Maria! It's such a blessing to have family who always supports us and reminds us that we're not alone. They know us like no one else and stick by us no matter what. Last week, we had a blast at a live music event. Seeing them dancing and having fun was awesome. The energy in the air was amazing.","question":"What music events has John attended?","ground_truth_answer":"Live music event, violin concert","locomo_metadata":{"sample_id":"conv-41","qa_index":36,"locomo_category":1,"evidence":["D20:4","D8:12"],"speaker_a":"John","speaker_b":"Maria"},"human_label":{"verdict":null,"failure_mode":null,"rationale":null}}
{"id":"locomo_conv-42_q038","category":"temporal","context":"Session 21 (1:43 pm on 14 September, 2022):\nJoanna: Cool, Nate! Gonna give it a go. Dairy-free is a must for me, especially for desserts. Last Friday, I made a deeeelish dessert with almond milk - it was good! Got any favs when it comes to dairy-free desserts?","question":"When did Joanna make a desert with almond milk?","ground_truth_answer":"The Friday before 14September, 2022","locomo_metadata":{"sample_id":"conv-42","qa_index":38,"locomo_category":2,"evidence":["D21:9"],"speaker_a":"Joanna","speaker_b":"Nate"},"human_label":{"verdict":null,"failure_mode":null,"rationale":null}}
{"id":"locomo_conv-41_q053","category":"temporal","context":"Session 28 (5:19 pm on 5 August, 2023):\nJohn: I haven't been able to volunteer much lately, but I definitely care about it. Last year, I helped renovate a rundown community center back home. It took a lot of work, but seeing the impact on the community was so worth it.","question":"When did John help renovate his hometown community center?","ground_truth_answer":"2022","locomo_metadata":{"sample_id":"conv-41","qa_index":53,"locomo_category":2,"evidence":["D28:11"],"speaker_a":"John","speaker_b":"Maria"},"human_label":{"verdict":null,"failure_mode":null,"rationale":null}}
{"id":"locomo_conv-50_q037","category":"open-ended","context":"Session 7 (6:06 pm on 31 May, 2023):\nCalvin: Hey Dave! Been ages since we chatted. So much has gone down. Touring with Frank Ocean last week was wild. Tokyo was unreal -- the crowd was insane. I felt so alive when performing, can't find the words to express it.\n\nSession 16 (2:55 pm on 31 August, 2023):\nCalvin: Hey Dave! The tour was amazing! I was so pumped from all the energy from the audience. This pic totally captures how I felt. Such an amazing time! [image: a photo of a crowd of people watching a concert with a large screen]\n\nSession 18 (10:56 am on 13 September, 2023):\nCalvin: Thanks Dave! Lots of cool stuff happening. Next up, a tour - so excited! After that, I'm off to explore and grow my brand. Can't wait to see what the future has in store!","question":"Does Calvin love music tours?","ground_truth_answer":"yes","locomo_metadata":{"sample_id":"conv-50","qa_index":37,"locomo_category":3,"evidence":["D18:7","D16:2","D7:1"],"speaker_a":"Calvin","speaker_b":"Dave"},"human_label":{"verdict":null,"failure_mode":null,"rationale":null}}
{"id":"locomo_conv-47_q017","category":"open-ended","context":"Session 8 (2:36 pm on 29 April, 2022):\nJohn: Yeah for sure! I've been playing one more game with friends these days. It's a game to figure out who the impostors are and it's super fun.","question":"What is the board game where you have to find the imposter that John mentions to James?","ground_truth_answer":"Mafia","locomo_metadata":{"sample_id":"conv-47","qa_index":17,"locomo_category":3,"evidence":["D8:36"],"speaker_a":"James","speaker_b":"John"},"human_label":{"verdict":null,"failure_mode":null,"rationale":null}}

File diff suppressed because one or more lines are too long

View File

@@ -1,41 +0,0 @@
{
"dataset_version": "39e415e2f3a0fa1bd3cb1804a58d0b440b50d3070b2100698437e4ec402a5b24",
"instance_count": 1531,
"built_at": "2026-04-22T09:23:30.601Z",
"source": "https://raw.githubusercontent.com/snap-research/locomo/main/data/locomo10.json",
"source_reference": "Maharana et al. 2024, ACL-2024, \"Evaluating Very Long-Term Conversational Memory of LLM Agents\"",
"canonicalisation": {
"adversarial_excluded": true,
"no_evidence_excluded": true,
"sort_order": "instance_id ascending",
"field_order": [
"instance_id",
"conversation_id",
"question",
"gold_answer",
"expected",
"category",
"context",
"locomo_metadata"
],
"line_terminator": "\\n",
"trailing_newline": true,
"encoding": "utf-8",
"no_bom": true
},
"distribution": {
"single-hop": 841,
"multi-hop": 281,
"temporal": 320,
"open-ended": 89
},
"skip_stats": {
"adversarial": 446,
"noEvidence": 4,
"unresolved": 5,
"unknownCat": 0
},
"paper_total_claim": 1540,
"actual_count": 1531,
"count_matches_paper": false
}

View File

@@ -1,42 +0,0 @@
{
"dataset_version": "a8a99545d77a236e3c7aa1f5d0ccfd94d4bcc5c2d5adbd19f938aba844586c56",
"instance_count": 500,
"variant": "s",
"built_at": "2026-06-04T17:25:36.563Z",
"source": "https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned/resolve/main/longmemeval_s_cleaned.json",
"source_reference": "Wu et al. 2024, \"LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory\" (arXiv:2410.10813)",
"hf_repo": "xiaowu0162/longmemeval-cleaned",
"canonicalisation": {
"abstention_included": true,
"no_sessions_excluded": true,
"sort_order": "instance_id ascending",
"field_order": [
"instance_id",
"conversation_id",
"question",
"expected",
"context",
"question_type",
"is_abstention"
],
"line_terminator": "\\n",
"trailing_newline": true,
"encoding": "utf-8",
"no_bom": true
},
"distribution_by_question_type": {
"single-session-user": 70,
"single-session-assistant": 56,
"single-session-preference": 30,
"temporal-reasoning": 133,
"knowledge-update": 78,
"multi-session": 133
},
"abstention_count": 30,
"skip_stats": {
"missingFields": 0,
"noSessions": 0
},
"expected_count": 500,
"count_matches_expected": true
}

View File

@@ -1,918 +0,0 @@
{
"_meta": {
"description": "Stage 2 preflight 4-cell sample lock. Istih 50 LoCoMo instanci preko sva 4 ćelije (raw / memory-only / evolve-only / full-stack).",
"brief": "PM-Waggle-OS/briefs/2026-04-20-cc-preflight-prep-tasks.md Task 1",
"locked_decision": "decisions/2026-04-20-preflight-oq-resolutions-locked.md §OQ-PF-1",
"source": "https://raw.githubusercontent.com/snap-research/locomo/main/data/locomo10.json",
"source_reference": "Maharana et al. 2024, ACL-2024, \"Evaluating Very Long-Term Conversational Memory of LLM Agents\"",
"seed": 42,
"selection_algorithm": "Bucket LoCoMo qa entries by category (mapping verified via task_eval/evaluation.py + paper §4.1); within each category sort by instance_id ascending (canonical order), apply Fisher-Yates shuffle with xorshift32(seed=42), then take first N per category. Fisher-Yates PRNG is shared across buckets — key iteration order is fixed (single-hop, multi-hop, temporal, open-ended) to keep the selection stable against re-runs. instance_id = locomo_<sample_id>_q<3-digit qa-array index>.",
"distribution": {
"single-hop": 13,
"multi-hop": 13,
"temporal": 12,
"open-ended": 12
},
"total": 50,
"locomo_category_map": {
"1": "multi-hop",
"2": "temporal",
"3": "open-ended",
"4": "single-hop",
"5": "adversarial (excluded)"
},
"context_assembly": "Evidence dia_ids are grouped by session, prefixed with session_N_date_time for temporal anchoring, and rendered as \"speaker: text\" lines. Images are preserved via blip_caption tags."
},
"instances": [
{
"id": "locomo_conv-26_q110",
"category": "single-hop",
"context": "Session 8 (1:51 pm on 15 July, 2023):\nMelanie: The kids loved it! They were so excited to get their hands dirty and make something with clay. It was special to watch their creativity and imagination come to life, they made this! [image: a photo of a cup with a dog face on it]",
"question": "What kind of pot did Mel and her kids make with clay?",
"ground_truth_answer": "a cup with a dog face on it",
"locomo_metadata": {
"sample_id": "conv-26",
"qa_index": 110,
"locomo_category": 4,
"evidence": [
"D8:4"
],
"speaker_a": "Caroline",
"speaker_b": "Melanie"
}
},
{
"id": "locomo_conv-41_q104",
"category": "single-hop",
"context": "Session 17 (11:51 am on 3 June, 2023):\nJohn: Hey Maria, long time no talk! Life's been pretty wild lately. The toughest thing to deal with is that we had to say goodbye to Max. He was such an important part of our family for 10 years and it's so hard to think he's not here wagging that tail anymore.",
"question": "How long was Max a part of John's family?",
"ground_truth_answer": "10 years",
"locomo_metadata": {
"sample_id": "conv-41",
"qa_index": 104,
"locomo_category": 4,
"evidence": [
"D17:1"
],
"speaker_a": "John",
"speaker_b": "Maria"
}
},
{
"id": "locomo_conv-44_q073",
"category": "single-hop",
"context": "Session 7 (11:27 am on 2 June, 2023):\nAndrew: I'm still on the hunt, but it's tough finding a pet-friendly spot in the city. Been checking out some places, but no luck so far. A bit discouraged but I'm determined to find the right place and dog.",
"question": "What challenge is Andrew facing in their search for a pet?",
"ground_truth_answer": "Finding a pet-friendly spot in the city",
"locomo_metadata": {
"sample_id": "conv-44",
"qa_index": 73,
"locomo_category": 4,
"evidence": [
"D7:8"
],
"speaker_a": "Audrey",
"speaker_b": "Andrew"
}
},
{
"id": "locomo_conv-26_q132",
"category": "single-hop",
"context": "Session 16 (12:09 am on 13 September, 2023):\nCaroline: Since I was 17 or so. I find it soempowering and cathartic. It's amazing how art can show things that are hard to put into words. How long have you been into art?",
"question": "How long has Melanie been creating art?",
"ground_truth_answer": "7 years",
"locomo_metadata": {
"sample_id": "conv-26",
"qa_index": 132,
"locomo_category": 4,
"evidence": [
"D16:7"
],
"speaker_a": "Caroline",
"speaker_b": "Melanie"
}
},
{
"id": "locomo_conv-42_q108",
"category": "single-hop",
"context": "Session 5 (6:59 pm on 18 March, 2022):\nNate: No, not really. Just keep their area clean, feed them properly, and make sure they get enough light. It's actually kind of fun.",
"question": "How does Nate describe the process of taking care of turtles?",
"ground_truth_answer": "Not tough; keep their area clean, feed them properly, give them enough light.",
"locomo_metadata": {
"sample_id": "conv-42",
"qa_index": 108,
"locomo_category": 4,
"evidence": [
"D5:8"
],
"speaker_a": "Joanna",
"speaker_b": "Nate"
}
},
{
"id": "locomo_conv-48_q151",
"category": "single-hop",
"context": "Session 19 (12:52 am on 19 August, 2023):\nDeborah: Reminds me of when I used to play games with my husband. We'd take turns and it was a great way to bond and make memories. Gaming really can bring people closer, right?",
"question": "What did Deborah and her husband use to play to bond and make memories?",
"ground_truth_answer": "video games",
"locomo_metadata": {
"sample_id": "conv-48",
"qa_index": 151,
"locomo_category": 4,
"evidence": [
"D19:11"
],
"speaker_a": "Deborah",
"speaker_b": "Jolene"
}
},
{
"id": "locomo_conv-47_q123",
"category": "single-hop",
"context": "Session 23 (9:23 pm on 4 September, 2022):\nJames: At only $10 per class, it's very cheap! Also, I made meringue there and they taught us how to make the dough.",
"question": "What did James learn to make in the cooking class besides omelette and meringue?",
"ground_truth_answer": "Dough",
"locomo_metadata": {
"sample_id": "conv-47",
"qa_index": 123,
"locomo_category": 4,
"evidence": [
"D23:15"
],
"speaker_a": "James",
"speaker_b": "John"
}
},
{
"id": "locomo_conv-44_q067",
"category": "single-hop",
"context": "Session 5 (10:47 am on 6 May, 2023):\nAndrew: I'm looking for a place near a park or woods, so I can stay close to nature and give the dog a large open space to run around",
"question": "Where does Andrew want to live to give their dog a large, open space to run around?",
"ground_truth_answer": "near a park or woods",
"locomo_metadata": {
"sample_id": "conv-44",
"qa_index": 67,
"locomo_category": 4,
"evidence": [
"D5:7"
],
"speaker_a": "Audrey",
"speaker_b": "Andrew"
}
},
{
"id": "locomo_conv-48_q129",
"category": "single-hop",
"context": "Session 12 (4:30 pm on 9 April, 2023):\nDeborah: Hey Jolene! Great to see you! Had a blast biking nearby with my neighbor last week - was so freeing and beautiful. Checked out an art show with a friend today - really cool and inspiring stuff. Reminded me of my mom. [image: a photo of a large brown and white photo of a person]",
"question": "Which show did Deborah go to with a friend on 9 April, 2023?",
"ground_truth_answer": "an art show",
"locomo_metadata": {
"sample_id": "conv-48",
"qa_index": 129,
"locomo_category": 4,
"evidence": [
"D12:1"
],
"speaker_a": "Deborah",
"speaker_b": "Jolene"
}
},
{
"id": "locomo_conv-50_q109",
"category": "single-hop",
"context": "Session 18 (10:56 am on 13 September, 2023):\nCalvin: Thanks Dave! Lots of cool stuff happening. Next up, a tour - so excited! After that, I'm off to explore and grow my brand. Can't wait to see what the future has in store!",
"question": "What is Calvin excited about after the tour?",
"ground_truth_answer": "exploring and growing his brand",
"locomo_metadata": {
"sample_id": "conv-50",
"qa_index": 109,
"locomo_category": 4,
"evidence": [
"D18:7"
],
"speaker_a": "Calvin",
"speaker_b": "Dave"
}
},
{
"id": "locomo_conv-47_q103",
"category": "single-hop",
"context": "Session 16 (5:13 pm on 9 July, 2022):\nJames: Yeah, staying balanced can be tough. I'm trying to take breaks from my hobbies and do other things. Lately I've become interested in extreme sports. Yesterday, for example, I was doing rope jumping. The highest height I jumped from was 150 meters!",
"question": "What new hobby did James become interested in on 9 July, 2022?",
"ground_truth_answer": "Extreme sports",
"locomo_metadata": {
"sample_id": "conv-47",
"qa_index": 103,
"locomo_category": 4,
"evidence": [
"D16:5"
],
"speaker_a": "James",
"speaker_b": "John"
}
},
{
"id": "locomo_conv-42_q193",
"category": "single-hop",
"context": "Session 28 (5:54 pm on 9 November, 2022):\nJoanna: Thanks, Nate! I took that pic on a hike last summer near Fort Wayne. The sunset and the surrounding beauty were just incredible. It was an awesome reminder of nature's beauty.",
"question": "What inspired Joanna to take a picture of the sunset in the field near Fort Wayne?",
"ground_truth_answer": "The incredible sunset and surrounding beauty",
"locomo_metadata": {
"sample_id": "conv-42",
"qa_index": 193,
"locomo_category": 4,
"evidence": [
"D28:22"
],
"speaker_a": "Joanna",
"speaker_b": "Nate"
}
},
{
"id": "locomo_conv-49_q139",
"category": "single-hop",
"context": "Session 21 (4:25 pm on 26 December, 2023):\nSam: Ooh, Evan, I'd put a GPS sensor on your keys!",
"question": "What did Sam suggest Evan should do with his keys?",
"ground_truth_answer": "put a GPS sensor on them",
"locomo_metadata": {
"sample_id": "conv-49",
"qa_index": 139,
"locomo_category": 4,
"evidence": [
"D21:21"
],
"speaker_a": "Evan",
"speaker_b": "Sam"
}
},
{
"id": "locomo_conv-26_q039",
"category": "multi-hop",
"context": "Session 5 (1:36 pm on 3 July, 2023):\nCaroline: Since we last spoke, some big things have happened. Last week I went to an LGBTQ+ pride parade. Everyone was so happy and it made me feel like I belonged. It showed me how much our community has grown, it was amazing!\n\nSession 9 (2:31 pm on 17 July, 2023):\nCaroline: Yay! Next month I'm having an LGBTQ art show with my paintings - can't wait!\nCaroline: Hey Melanie! That sounds great! Last weekend I joined a mentorship program for LGBTQ youth - it's really rewarding to help the community.\n\nSession 10 (8:56 pm on 20 July, 2023):\nCaroline: Hey Mel! A lot's happened since we last chatted - I just joined a new LGBTQ activist group last Tues. I'm meeting so many cool people who are as passionate as I am about rights and community support. I'm giving my voice and making a real difference, plus it's fulfilling in so many ways. It's just great, you know?",
"question": "In what ways is Caroline participating in the LGBTQ community?",
"ground_truth_answer": "Joining activist group, going to pride parades, participating in an art show, mentoring program",
"locomo_metadata": {
"sample_id": "conv-26",
"qa_index": 39,
"locomo_category": 1,
"evidence": [
"D10:3",
"D5:1",
"D9:12",
"D9:2"
],
"speaker_a": "Caroline",
"speaker_b": "Melanie"
}
},
{
"id": "locomo_conv-30_q031",
"category": "multi-hop",
"context": "Session 1 (4:04 pm on 20 January, 2023):\nJon: Hey Gina! Good to see you too. Lost my job as a banker yesterday, so I'm gonna take a shot at starting my own business.\n\nSession 15 (10:04 am on 19 June, 2023):\nJon: Yeah! Let's make some awesome memories tomorrow at the grand opening! [image: a photo of a man in a native costume is giving another man a high five]",
"question": "How long did it take for Jon to open his studio?",
"ground_truth_answer": "six months",
"locomo_metadata": {
"sample_id": "conv-30",
"qa_index": 31,
"locomo_category": 1,
"evidence": [
"D1:2",
"D15:13"
],
"speaker_a": "Jon",
"speaker_b": "Gina"
}
},
{
"id": "locomo_conv-50_q005",
"category": "multi-hop",
"context": "Session 4 (6:24 pm on 1 May, 2023):\nDave: Thanks! Appreciate the support. My dream was to open a shop and it's a step towards my other dream of working on classic cars. I love their design and engineering.\nDave: Thanks! Appreciate the support. My dream was to open a shop and it's a step towards my other dream of working on classic cars. I love their design and engineering.\n\nSession 5 (1:16 pm on 3 May, 2023):\nDave: Thanks Calvin! Appreciate the support. I'm gonna keep learning more about auto engineering, maybe even build a custom car from scratch someday - that's the dream! For now, just gonna keep working on this project and assisting customers.",
"question": "What are Dave's dreams?",
"ground_truth_answer": "open a car maintenance shop, work on classic cars, build a custom car from scratch",
"locomo_metadata": {
"sample_id": "conv-50",
"qa_index": 5,
"locomo_category": 1,
"evidence": [
"D4:5",
"D4:5",
"D5:5"
],
"speaker_a": "Calvin",
"speaker_b": "Dave"
}
},
{
"id": "locomo_conv-50_q018",
"category": "multi-hop",
"context": "Session 6 (11:50 am on 16 May, 2023):\nCalvin: Thanks, Dave. Appreciate the support. Waiting on insurance to kick in so I can start repairs - fingers crossed it won't take too long. Take a look at my creative haven, where I pour my heart into the music. [image: a photo of a music studio with a keyboard, synthesizer, and other musical equipment]\n\nSession 9 (3:15 pm on 21 June, 2023):\nCalvin: Hi Dave! Nice to hear from you. Since last chat, some things have happened. I had a lil car accident last Friday, kinda upsetting but no one was hurt. I've been dealing with insurance and repairs - taking up a lot of time and energy. Look at this photo! [image: a photo of a red car with a black rim parked on a sidewalk]",
"question": "How many times has Calvin had to deal with insurance paperwork?",
"ground_truth_answer": "two times",
"locomo_metadata": {
"sample_id": "conv-50",
"qa_index": 18,
"locomo_category": 1,
"evidence": [
"D6:5",
"D9:1"
],
"speaker_a": "Calvin",
"speaker_b": "Dave"
}
},
{
"id": "locomo_conv-48_q080",
"category": "multi-hop",
"context": "Session 10 (5:35 pm on 22 March, 2023):\nJolene: Definitely! I've been gathering information, watching videos, and I even got a beginners' guide to surfing. Just need to find the right time and place to get a lesson.\n\nSession 29 (1:24 pm on 17 September, 2023):\nJolene: How cool! But I never decided to try it.\nJolene: Just started learning, but haven't gone yet. Want to come with me sometime?",
"question": "Has Jolene tried surfing?",
"ground_truth_answer": "no",
"locomo_metadata": {
"sample_id": "conv-48",
"qa_index": 80,
"locomo_category": 1,
"evidence": [
"D10:20",
"D29:26",
"D29:30"
],
"speaker_a": "Deborah",
"speaker_b": "Jolene"
}
},
{
"id": "locomo_conv-47_q009",
"category": "multi-hop",
"context": "Session 1 (3:47 pm on 17 March, 2022):\nJames: Max and Daisy. Will be actually cool to build an app for dog walking and pet care. The goal is to connect pet owners with reliable dog walkers and provide helpful information on pet care.\n\nSession 5 (9:52 am on 12 April, 2022):\nJames: Hey John! Long time no chat - I adopted a pup from a shelter in Stamford last week and my days have been so much happier with him in the fam. I named it Ned. Any progress on your gaming goals? [image: a photo of a dog and a cat sitting on a dog bed]",
"question": "What are the names of James's dogs?",
"ground_truth_answer": "Ned, Daisy, Max",
"locomo_metadata": {
"sample_id": "conv-47",
"qa_index": 9,
"locomo_category": 1,
"evidence": [
"D1:14",
"D5:1"
],
"speaker_a": "James",
"speaker_b": "John"
}
},
{
"id": "locomo_conv-50_q019",
"category": "multi-hop",
"context": "Session 3 (4:15 pm on 20 April, 2023):\nCalvin: Hey Dave! Long time no see. I just went to an awesome music thingy in Tokyo - so cool!\n\nSession 12 (1:12 pm on 3 August, 2023):\nCalvin: Aww, that's cool, Dave. Reminiscing is always fun! That pic you shared takes me back to my trip to the Ferrari dealership. I saw a lot of amazing cars, but as for me, my car is the best and I'm pretty proud of this. Sure, it's just material, but it reminds me of my hard work and dedication. It really inspires me. Take a look at this beauty! [image: a photography of a red car is lifted on a lift in a garage]\n\nSession 24 (10:11 am on 19 October, 2023):\nCalvin: Yeah definitely! Shibuya Crossing is like Tokyo's Times Square, and I was excited to explore Shinjuku. Plus, there's amazing food there, can't wait to try it again! Look at the photo of Shibuya Crossing at night, that's amazing! [image: a photo of a crowd of people with umbrellas in the rain]",
"question": "Which places or events has Calvin visited in Tokyo?",
"ground_truth_answer": "music festival, car museum, Shibuya crossing, Shinjuku",
"locomo_metadata": {
"sample_id": "conv-50",
"qa_index": 19,
"locomo_category": 1,
"evidence": [
"D3:1",
"D12:7",
"D24:19"
],
"speaker_a": "Calvin",
"speaker_b": "Dave"
}
},
{
"id": "locomo_conv-48_q001",
"category": "multi-hop",
"context": "Session 1 (4:06 pm on 23 January, 2023):\nDeborah: It was full of memories, she passed away a few years ago. This is our last photo together. [image: a photo of a woman in a wheelchair hugging a woman in a wheelchair]\n\nSession 2 (9:49 am on 27 January, 2023):\nDeborah: Hey Jolene, sorry to tell you this but my dad passed away two days ago. It's been really tough on us all - his sudden death left us all kinda shell-shocked. I'm trying to channel my grief by spending more time with family and cherishing the memories. These moments remind me to live life fully. [image: a photo of a woman hugging a woman who is sitting on a couch]\n\nSession 6 (4:12 pm on 22 February, 2023):\nDeborah: The roses and dahlias bring me peace. I lost a friend last week, so I've been spending time in the garden to find some comfort.",
"question": "Which of Deborah`s family and friends have passed away?",
"ground_truth_answer": "mother, father, her friend Karlie",
"locomo_metadata": {
"sample_id": "conv-48",
"qa_index": 1,
"locomo_category": 1,
"evidence": [
"D1:5",
"D2:1",
"D6:4"
],
"speaker_a": "Deborah",
"speaker_b": "Jolene"
}
},
{
"id": "locomo_conv-50_q033",
"category": "multi-hop",
"context": "Session 14 (12:35 am on 14 August, 2023):\nDave: Hey Cal, how's it going? Something cool happened since last we talked - I got to go to a car workshop in San Francisco! So cool to dive into the world of car restoration and see all the different techniques. People were really passionate and dedicated - truly inspiring!\n\nSession 17 (9:19 am on 2 September, 2023):\nDave: Hey Calvin! Been a while, what's up? I'm tied up with car stuff lately, yesterday I came back from San Francsico with some great insights and knowledge on car modification that I want to share with you! Changing things around, and giving an old car a new life - so satisfying!",
"question": "What was Dave doing in San Francisco?",
"ground_truth_answer": "attending a car modification workshop",
"locomo_metadata": {
"sample_id": "conv-50",
"qa_index": 33,
"locomo_category": 1,
"evidence": [
"D17:1",
"D14:1"
],
"speaker_a": "Calvin",
"speaker_b": "Dave"
}
},
{
"id": "locomo_conv-43_q021",
"category": "multi-hop",
"context": "Session 3 (4:21 pm on 16 July, 2023):\nJohn: It's Seattle, I'm stoked for my game there next month! It's one of my favorite cities to explore - super vibrant! [image: a photo of a crowd of people watching a basketball game]\n\nSession 6 (1:08 pm on 11 August, 2023):\nJohn: I was in Chicago, it was awesome! It had so much energy and the locals were really friendly. It's great to experience other cultures and connect with new folks.\n\nSession 9 (6:59 pm on 26 August, 2023):\nJohn: Wow, Tim, that's an awesome book collection! It's cool to escape to different worlds with a hobby. By the way, I love discovering new cities - check out this pic from one of my trips to New York City! [image: a photo of a cityscape with a view of a skyscraper]",
"question": "Which US cities does John mention visiting to Tim?",
"ground_truth_answer": "Seattle, Chicago, New York",
"locomo_metadata": {
"sample_id": "conv-43",
"qa_index": 21,
"locomo_category": 1,
"evidence": [
"D3:19",
"D6:3",
"D9:6"
],
"speaker_a": "Tim",
"speaker_b": "John"
}
},
{
"id": "locomo_conv-50_q063",
"category": "multi-hop",
"context": "Session 16 (2:55 pm on 31 August, 2023):\nDave: Sure, let me know when, I'm here to lend a hand. It's great to fuel your ideas. Remember that photo you sent me once? Love how this guitar shows our different artistic styles. [image: a photo of a guitar with a octopus on it]\nCalvin: Performing on such a big stage was a dream come true! The energy was incredible and I felt on top of the world. It was seriously surreal.\nCalvin: Yeah, Dave! It's like every mark and strum holds a story. Take a look. [image: a photo of a guitar with a purple glow on it]\nDave: That guitar has a gorgeous purple hue. Why did you make it so shiny?",
"question": "What style of guitars does Calvin own?",
"ground_truth_answer": "custom-made yellow guitar with an octopus on it, shiny purple guitar",
"locomo_metadata": {
"sample_id": "conv-50",
"qa_index": 63,
"locomo_category": 1,
"evidence": [
"D16:13",
"D16:4",
"D16:18",
"D16:19"
],
"speaker_a": "Calvin",
"speaker_b": "Dave"
}
},
{
"id": "locomo_conv-41_q023",
"category": "multi-hop",
"context": "Session 2 (6:10 pm on 22 December, 2022):\nMaria: I can picture you all laughing and having a blast making your own pizzas - a great way to bond! I made some peach cobbler recently, it was great.\n\nSession 13 (3:18 pm on 4 May, 2023):\nMaria: Thanks, John! It had lots of great things like salads, sandwiches, and homemade desserts. My favorite is the amazing banana split sundae - I enjoy it after a day of volunteering. It's a little moment of joy in life - balance is key!",
"question": "What desserts has Maria made?",
"ground_truth_answer": "Banana split sundae, Peach cobbler",
"locomo_metadata": {
"sample_id": "conv-41",
"qa_index": 23,
"locomo_category": 1,
"evidence": [
"D2:25",
"D13:18"
],
"speaker_a": "John",
"speaker_b": "Maria"
}
},
{
"id": "locomo_conv-48_q013",
"category": "multi-hop",
"context": "Session 2 (9:49 am on 27 January, 2023):\nDeborah: That's my old home. I go there now and then for my mom, who passed away. Sitting in that spot by the window gives me peace.\n\nSession 4 (9:48 am on 4 February, 2023):\nDeborah: That sounds great, Jolene. Nature's calming for sure. Guess it helps us forget the daily craziness and find inner peace. No wonder you're a fan! I like to come to this spot by the water near my mom's old house. It's where I reflect on her life and find some peace. Being surrounded by nature helps a lot. [image: a photo of a small island with a lone boat in the water]\n\nSession 6 (4:12 pm on 22 February, 2023):\nDeborah: I've been blessed to travel to a few places and Bali last year was one of my favs. It was a gorgeous island that gave me peace, great for yoga. [image: a photo of a swing on a beach with a blue sky]\n\nSession 19 (12:52 am on 19 August, 2023):\nDeborah: I love going to this park near my house - it has a nice forest trail and a beach. It's a peaceful spot where I can do some yoga and reflect. There's also a special bench that holds special meaning to me.",
"question": "What places give Deborah peace?",
"ground_truth_answer": "sitting in a spot by the window in her Mom's house, sitting by the beach, Bali, forest trail in a nearby park",
"locomo_metadata": {
"sample_id": "conv-48",
"qa_index": 13,
"locomo_category": 1,
"evidence": [
"D2:13",
"D4:34",
"D6:10",
"D19:17"
],
"speaker_a": "Deborah",
"speaker_b": "Jolene"
}
},
{
"id": "locomo_conv-44_q047",
"category": "temporal",
"context": "Session 24 (6:12 pm on 19 October, 2023):\nAndrew: Hi Audrey! Pets really can make our lives better, huh? Speaking of which, I've got some awesome news -- I recently adopted another pup from a shelter. He's the best. [image: a photo of a dog sitting on a couch with a bowl of food]\n\nSession 28 (9:02 am on 22 November, 2023):\nAndrew: No, we haven't got the chance to take them to the groomer yet. But will do that soon! So guess what, I can't help myself but to adpot another dog the other day. Here's a photo of the doggo! [image: a photo of a woman feeding a dog hay in a barn]",
"question": "How many months passed between Andrew adopting Buddy and Scout",
"ground_truth_answer": "one month",
"locomo_metadata": {
"sample_id": "conv-44",
"qa_index": 47,
"locomo_category": 2,
"evidence": [
"D24:2",
"D28:6"
],
"speaker_a": "Audrey",
"speaker_b": "Andrew"
}
},
{
"id": "locomo_conv-26_q021",
"category": "temporal",
"context": "Session 6 (8:18 pm on 6 July, 2023):\nCaroline: Wow, that's great! It sure shows how important friendship and compassion are. It's made me appreciate how lucky I am to have my friends and family helping with my transition. They make all the difference. We even had a picnic last week! [image: a photo of a group of women sitting on a blanket in a park]",
"question": "When did Caroline have a picnic?",
"ground_truth_answer": "The week before 6 July 2023",
"locomo_metadata": {
"sample_id": "conv-26",
"qa_index": 21,
"locomo_category": 2,
"evidence": [
"D6:11"
],
"speaker_a": "Caroline",
"speaker_b": "Melanie"
}
},
{
"id": "locomo_conv-48_q021",
"category": "temporal",
"context": "Session 2 (9:49 am on 27 January, 2023):\nJolene: We are planning to play \"Walking Dead\" next Saturday.",
"question": "When do Jolene and her partner plan to complete the game \"Walking Dead\"?",
"ground_truth_answer": "Saturday after 27 January, 2023",
"locomo_metadata": {
"sample_id": "conv-48",
"qa_index": 21,
"locomo_category": 2,
"evidence": [
"D2:30"
],
"speaker_a": "Deborah",
"speaker_b": "Jolene"
}
},
{
"id": "locomo_conv-30_q010",
"category": "temporal",
"context": "Session 5 (9:32 am on 8 February, 2023):\nGina: That's awesome! I'm sure you feel great knowing your students are doing so well with dance. It's amazing what it can do for people! Oh, and btw - I've been working hard on my online store and just teamed up with a local artist for some cool designs. Check 'em out! [image: a photo of a rack of clothes with a coat and a dress]",
"question": "When did Gina team up with a local artist for some cool designs?",
"ground_truth_answer": "February, 2023",
"locomo_metadata": {
"sample_id": "conv-30",
"qa_index": 10,
"locomo_category": 2,
"evidence": [
"D5:5"
],
"speaker_a": "Jon",
"speaker_b": "Gina"
}
},
{
"id": "locomo_conv-26_q073",
"category": "temporal",
"context": "Session 17 (10:31 am on 13 October, 2023):\nMelanie: Thanks for the tip, Caroline. Doing research and readying myself emotionally makes sense. I'll do that. BTW, recently I had a setback. Last month I got hurt and had to take a break from pottery, which I use for self-expression and peace.",
"question": "When did Melanie get hurt?",
"ground_truth_answer": "September 2023",
"locomo_metadata": {
"sample_id": "conv-26",
"qa_index": 73,
"locomo_category": 2,
"evidence": [
"D17:8"
],
"speaker_a": "Caroline",
"speaker_b": "Melanie"
}
},
{
"id": "locomo_conv-30_q033",
"category": "temporal",
"context": "Session 16 (2:15 pm on 21 June, 2023):\nJon: That's awesome, Gina! Yesterday I chose to go to networking events to make things happen. It's been tough but I'm staying determined and focused.",
"question": "When did Jon visit networking events for his store?",
"ground_truth_answer": "20 June, 2023",
"locomo_metadata": {
"sample_id": "conv-30",
"qa_index": 33,
"locomo_category": 2,
"evidence": [
"D16:6"
],
"speaker_a": "Jon",
"speaker_b": "Gina"
}
},
{
"id": "locomo_conv-44_q001",
"category": "temporal",
"context": "Session 1 (1:10 pm on 27 March, 2023):\nAndrew: Hey Audrey! So, I started a new job as a Financial Analyst last week - it's been quite a change from my previous job. How about you? Anything interesting happening?",
"question": "When did Andrew start his new job as a financial analyst?",
"ground_truth_answer": "The week before March 27, 2023",
"locomo_metadata": {
"sample_id": "conv-44",
"qa_index": 1,
"locomo_category": 2,
"evidence": [
"D1:2"
],
"speaker_a": "Audrey",
"speaker_b": "Andrew"
}
},
{
"id": "locomo_conv-49_q079",
"category": "temporal",
"context": "Session 23 (1:32 pm on 6 January, 2024):\nEvan: Thanks Sam! We're off to Canada next month for our honeymoon. So excited to create some awesome memories. Looking forward to exploring the beautiful snowy landscapes there. [image: a photo of a stream running through a snowy forest filled with snow]",
"question": "When will Evan and his partner have their honeymoon in Canada?",
"ground_truth_answer": "February 2024",
"locomo_metadata": {
"sample_id": "conv-49",
"qa_index": 79,
"locomo_category": 2,
"evidence": [
"D23:23"
],
"speaker_a": "Evan",
"speaker_b": "Sam"
}
},
{
"id": "locomo_conv-26_q010",
"category": "temporal",
"context": "Session 3 (7:55 pm on 9 June, 2023):\nCaroline: Yeah, I'm really lucky to have them. They've been there through everything, I've known these friends for 4 years, since I moved from my home country. Their love and help have been so important especially after that tough breakup. I'm super thankful. Who supports you, Mel?",
"question": "How long has Caroline had her current group of friends for?",
"ground_truth_answer": "4 years",
"locomo_metadata": {
"sample_id": "conv-26",
"qa_index": 10,
"locomo_category": 2,
"evidence": [
"D3:13"
],
"speaker_a": "Caroline",
"speaker_b": "Melanie"
}
},
{
"id": "locomo_conv-49_q058",
"category": "temporal",
"context": "Session 12 (3:09 pm on 8 October, 2023):\nEvan: Hey Sam, tough news. Yeah, our health can really put a damper on things. I started lifting weights one year ago and it's been a journey. It was a struggle at first, but I'm seeing some gains. You interested in trying it out? [image: a photo of a man doing a squat on a machine in a gym]",
"question": "When did Evan start lifting weights?",
"ground_truth_answer": "October 2022",
"locomo_metadata": {
"sample_id": "conv-49",
"qa_index": 58,
"locomo_category": 2,
"evidence": [
"D12:2"
],
"speaker_a": "Evan",
"speaker_b": "Sam"
}
},
{
"id": "locomo_conv-43_q037",
"category": "temporal",
"context": "Session 13 (1:50 pm on 13 October, 2023):\nTim: Hey John! It's been ages since we last talked. Guess what? Last week I went to a Harry Potter conference in the UK - it was incredible! There were so many people who shared the same love of HP as me, it was like a magical family. I felt so inspired and like I got a new lease of life. I love how my passion for fantasy stuff brings me closer to people from all over the world, it's pretty cool.",
"question": "Which week did Tim visit the UK for the Harry Potter Conference?",
"ground_truth_answer": "The week before October 13th, 2023.",
"locomo_metadata": {
"sample_id": "conv-43",
"qa_index": 37,
"locomo_category": 2,
"evidence": [
"D13:1"
],
"speaker_a": "Tim",
"speaker_b": "John"
}
},
{
"id": "locomo_conv-50_q027",
"category": "temporal",
"context": "Session 16 (2:55 pm on 31 August, 2023):\nCalvin: Cool! Last weekend I started shooting a video for my new album - can't wait for you to check it out! Look how great it was! [image: a photo of a camera and a video camera on a beach]",
"question": "Which city was Calvin visiting in August 2023?",
"ground_truth_answer": "Miami",
"locomo_metadata": {
"sample_id": "conv-50",
"qa_index": 27,
"locomo_category": 2,
"evidence": [
"D16:6"
],
"speaker_a": "Calvin",
"speaker_b": "Dave"
}
},
{
"id": "locomo_conv-43_q051",
"category": "open-ended",
"context": "Session 20 (9:52 am on 1 December, 2023):\nJohn: Hi Tim! Congrats on your success! Keep it up, you're doing great! I'm also trying out yoga to get a little extra strength and flexibility. It's challenging but worth it. [image: a photo of a white wall with a black lettering that says 30 positive suites]",
"question": "What kind of yoga for building core strength might John benefit from?",
"ground_truth_answer": "Hatha Yoga",
"locomo_metadata": {
"sample_id": "conv-43",
"qa_index": 51,
"locomo_category": 3,
"evidence": [
"D20:2"
],
"speaker_a": "Tim",
"speaker_b": "John"
}
},
{
"id": "locomo_conv-42_q014",
"category": "open-ended",
"context": "Session 7 (7:37 pm on 15 April, 2022):\nNate: Hey Jo, guess what I did? Dyed my hair last week - come see!",
"question": "What nickname does Nate use for Joanna?",
"ground_truth_answer": "Jo",
"locomo_metadata": {
"sample_id": "conv-42",
"qa_index": 14,
"locomo_category": 3,
"evidence": [
"D7:1"
],
"speaker_a": "Joanna",
"speaker_b": "Nate"
}
},
{
"id": "locomo_conv-47_q016",
"category": "open-ended",
"context": "Session 8 (2:36 pm on 29 April, 2022):\nJohn: Yes, we played one game, but I forgot its name. Perhaps you know this game. There were multi-colored cards with numbers. You can only place a card with the same color or number on your opponent's card. Sometimes you trade cards, sometimes you need to draw a few extra from the deck or skip a turn.",
"question": "What is the game with different colored cards that was John talking about with James?",
"ground_truth_answer": "UNO",
"locomo_metadata": {
"sample_id": "conv-47",
"qa_index": 16,
"locomo_category": 3,
"evidence": [
"D8:34"
],
"speaker_a": "James",
"speaker_b": "John"
}
},
{
"id": "locomo_conv-26_q069",
"category": "open-ended",
"context": "Session 7 (4:33 pm on 12 July, 2023):\nMelanie: Wow, Caroline. We've come so far, but there's more to do. Your drive to help is awesome! What's your plan to pitch in?\n\nSession 13 (3:31 pm on 23 August, 2023):\nMelanie: Wow, Caroline! That's amazing. You really care about being real and helping others. Wishing you the best on your adoption journey!\n\nSession 16 (12:09 am on 13 September, 2023):\nMelanie: The sign was just a precaution, I had a great time. But thank you for your concern, you're so thoughtful!",
"question": "What personality traits might Melanie say Caroline has?",
"ground_truth_answer": "Thoughtful, authentic, driven",
"locomo_metadata": {
"sample_id": "conv-26",
"qa_index": 69,
"locomo_category": 3,
"evidence": [
"D16:18",
"D13:16",
"D7:4"
],
"speaker_a": "Caroline",
"speaker_b": "Melanie"
}
},
{
"id": "locomo_conv-42_q073",
"category": "open-ended",
"context": "Session 28 (5:54 pm on 9 November, 2022):\nJoanna: Thanks, Nate! I took that pic on a hike last summer near Fort Wayne. The sunset and the surrounding beauty were just incredible. It was an awesome reminder of nature's beauty.",
"question": "What state did Joanna visit in summer 2021?",
"ground_truth_answer": "Indiana",
"locomo_metadata": {
"sample_id": "conv-42",
"qa_index": 73,
"locomo_category": 3,
"evidence": [
"D28:22"
],
"speaker_a": "Joanna",
"speaker_b": "Nate"
}
},
{
"id": "locomo_conv-48_q005",
"category": "open-ended",
"context": "Session 1 (4:06 pm on 23 January, 2023):\nJolene: Staying connected is super important. Do you have something to remember her by? This pendant reminds me of my mother, she gave it to me in 2010 in Paris. [image: a photo of a heart shaped pendant with a bird on it]",
"question": "In what country did Jolene's mother buy her the pendant?",
"ground_truth_answer": "In France",
"locomo_metadata": {
"sample_id": "conv-48",
"qa_index": 5,
"locomo_category": 3,
"evidence": [
"D1:8"
],
"speaker_a": "Deborah",
"speaker_b": "Jolene"
}
},
{
"id": "locomo_conv-49_q019",
"category": "open-ended",
"context": "Session 3 (3:55 pm on 6 June, 2023):\nSam: Yeah, you're right. It takes time, but I'm up for keep trying and making those tiny changes.\nEvan: Sure Sam, I'd be glad to help. Let's get together and I'll show you some basic exercises. We'll reach our goals!\n\nSession 8 (6:17 pm on 19 August, 2023):\nSam: Cool, Evan! What have you been learning in those classes?\nEvan: Thanks Sam! I aim to capture the vibe of nature in my paintings, conveying the peacefulness of being outdoors. [image: a photo of a tree with pink flowers in a field]\n\nSession 9 (10:18 am on 27 August, 2023):\nEvan: Yeah, thanks for the push, Sam. Oh, hey, I came across this pic when I was tidying up. I took a road trip last month - the scenery was stunning and nature really chilled me out. [image: a photography of a car is parked near a lake with mountains in the background]\nSam: I haven't gone on a road trip in ages, but I love being surrounded by nature. It's so tranquil and refreshing. I'd love to go hiking more, but it can be a bit challenging sometimes. However, I am working on becoming healthier, so maybe a road trip and a hike will be possible soon.\n\nSession 12 (3:09 pm on 8 October, 2023):\nSam: Thanks, Evan. I appreciate your support. It really means a lot to me. I'll definitely keep you posted on my progress.\nSam: Wow, Evan, that's really inspiring. Gonna keep believing in it!\n\nSession 14 (1:50 pm on 17 October, 2023):\nSam: Sounds awesome, Evan! Can't wait to go on a hike with you. It's going to be a fun challenge and a great opportunity to appreciate the beauty of nature.\nEvan: Sure thing! Our hike is going to be awesome, I can tell. I'm always here to support you.\n\nSession 22 (11:00 am on 31 December, 2023):\nSam: Hey Evan! I’m really getting into this healthier lifestyle—just took my friends on an epic hiking trip last Friday! [image: a photography of a man standing on a rock looking out over a valley]",
"question": "Considering their conversations and personal growth, what advice might Evan and Sam give to someone facing a major life transition or challenge?",
"ground_truth_answer": "Evan and Sam would likely advise embracing small, consistent changes​​, finding stress-relieving activities like hiking​​, painting, and road trips​​, and the importance of friendship and support in navigating challenges​​.",
"locomo_metadata": {
"sample_id": "conv-49",
"qa_index": 19,
"locomo_category": 3,
"evidence": [
"D3:10",
"D3:15",
"D22:1",
"D8:17",
"D8:22",
"D9:8",
"D9:11",
"D14:7",
"D14:12",
"D12:7",
"D12:11"
],
"speaker_a": "Evan",
"speaker_b": "Sam"
}
},
{
"id": "locomo_conv-43_q015",
"category": "open-ended",
"context": "Session 4 (4:17 pm on 2 August, 2023):\nJohn: Oh yeah, I remember you telling me about Harry Potter! I've got a funny story. Anthony and I went to this charity thing and ended up in this intense Harry Potter trivia contest. We did alright, but there was this one super-nerd there that took home this as a prize.\n [image: a photo of a book with a picture of a person holding a bookmark]",
"question": "Who is Anthony?",
"ground_truth_answer": "likely John's friend, colleague or family",
"locomo_metadata": {
"sample_id": "conv-43",
"qa_index": 15,
"locomo_category": 3,
"evidence": [
"D4:8"
],
"speaker_a": "Tim",
"speaker_b": "John"
}
},
{
"id": "locomo_conv-48_q040",
"category": "open-ended",
"context": "Session 10 (5:35 pm on 22 March, 2023):\nDeborah: Don't forget to take it easy and look after yourself. Wishing you all the best! Recently, Anna and I were sitting by the sea, watching the sunset and talking about each other. And we realized that we inspire each other. What thoughts does the sea in this photo make you think of? [image: a photo of a person walking on the beach with a surfboard]",
"question": "Does Deborah live close to the beach or the mountains?",
"ground_truth_answer": "beach",
"locomo_metadata": {
"sample_id": "conv-48",
"qa_index": 40,
"locomo_category": 3,
"evidence": [
"D10:17"
],
"speaker_a": "Deborah",
"speaker_b": "Jolene"
}
},
{
"id": "locomo_conv-41_q008",
"category": "open-ended",
"context": "Session 5 (1:17 pm on 28 January, 2023):\nJohn: It's definitely isn't, Maria. My kids have so much and others don't. We really need to do something about it.",
"question": "What might John's financial status be?",
"ground_truth_answer": "Middle-class or wealthy",
"locomo_metadata": {
"sample_id": "conv-41",
"qa_index": 8,
"locomo_category": 3,
"evidence": [
"D5:5"
],
"speaker_a": "John",
"speaker_b": "Maria"
}
},
{
"id": "locomo_conv-41_q050",
"category": "open-ended",
"context": "Session 2 (6:10 pm on 22 December, 2022):\nJohn: Yeah, they are my rock in tough times and always cheer me on. I'm really thankful for their love. Family time means a lot to me. [image: a photography of a family enjoying a ride on a swing]\n\nSession 3 (8:30 pm on 1 January, 2023):\nJohn: We held some events and got to meet some people. We went to a homeless shelter to give out food and supplies. Seeing the smiles on their faces, we knew we made a real difference. We also organized a toy drive for kids in need. It was amazing seeing the community come together to spread some joy.\n\nSession 4 (7:06 pm on 9 January, 2023):\nJohn: I tried to stay calm and asked for assistance, which helped me handle the situation and make it back safely.\n\nSession 26 (1:59 pm on 31 July, 2023):\nJohn: It was chaotic when we arrived, but we pulled together. I got a surge of energy and purpose, and we were able to save a family from a burning building. It was wild, but knowing we made a difference made it worth it.",
"question": "What attributes describe John?",
"ground_truth_answer": "Selfless, family-oriented, passionate, rational",
"locomo_metadata": {
"sample_id": "conv-41",
"qa_index": 50,
"locomo_category": 3,
"evidence": [
"D26:6",
"D2:14",
"D3:5",
"D4:6"
],
"speaker_a": "John",
"speaker_b": "Maria"
}
},
{
"id": "locomo_conv-26_q022",
"category": "open-ended",
"context": "Session 6 (8:18 pm on 6 July, 2023):\nCaroline: I've got lots of kids' books- classics, stories from different cultures, educational books, all of that. What's a favorite book you remember from your childhood?",
"question": "Would Caroline likely have Dr. Seuss books on her bookshelf?",
"ground_truth_answer": "Yes, since she collects classic children's books",
"locomo_metadata": {
"sample_id": "conv-26",
"qa_index": 22,
"locomo_category": 3,
"evidence": [
"D6:9"
],
"speaker_a": "Caroline",
"speaker_b": "Melanie"
}
}
]
}

View File

@@ -1,62 +0,0 @@
# Judge-Delta — GAIA 2 search N=160 self-judge vs independent trio
**Date:** 2026-05-22 · Phase 1 (judge integrity) of the harness benchmark plan
**Purpose:** the search split is judged ~entirely by the LLM `user_message_checker`
(semantic equivalence of the agent's final answer vs the oracle answer; no app-action
oracle events to hard-match). The production run self-judged (Sonnet 4.6 judging a
Sonnet 4.6 agent), so we re-judged with an independent trio to detect inflation.
## Method (controlled — only the judge model varies)
Imported GAIA 2's **own** `user_message_checker` — same system prompt, same few-shot
examples, same `[[Success]]/[[Failure]]` parsing (`gaia2_core.judge.prompts` +
`LLMChecker`). For each of the 148 answerable scenarios extracted (task, agent final
message, oracle reference) and re-ran the checker with three independent judges. Only
the judge MODEL changed. Mirrors the C-1 LOCOMO trio-strict discipline.
Judges: **Opus 4.7 · Gemini 2.5 Pro · GPT-5** (M6 roster, all independent of the
Sonnet 4.6 agent+self-judge). Engine omits `temperature` uniformly (Opus 4.7 / GPT-5
reject it); `num_votes=1`. Script: `rejudge_user_message.py`. Raw: `runs/rejudge-search-n160.jsonl`.
Denominator = **148 answerable** (160 scenarios − 12 with no final answer / errored).
## Results
| Judge | PASS | Rate (n=148) |
|---|---:|---:|
| **Self — Sonnet 4.6** (production) | 134 | **90.5%** |
| Opus 4.7 | 134 | 90.5% |
| Gemini 2.5 Pro | 133 | 89.9% |
| GPT-5 | 132 | 89.2% |
| **Trio-strict** (all 3 independent agree PASS) | 129 | **87.2%** |
| Trio-majority (≥2/3 independent PASS) | 135 | 91.2% |
**Agreement with the self-judge:** Opus 98.6% · Gemini 98.0% · GPT-5 97.3%.
## Verdict — self-judge is NOT inflated
- Independent single judges land within **0.6–1.3pp** of the Sonnet self-judge.
- **Trio-strict is only −3.3pp** below self (90.5% → 87.2% on n=148) — well within
cross-LLM-judge norms, and far tighter than LOCOMO's +5.3pp self-judge inflation.
- Only **6/148 (4%)** scenarios are marginal (self-PASS but not unanimous across the trio):
`21_csyctc, 22_52pwi3, 23_onhtod, 26_oqrx9a, 30_69r1z7, 30_9uo633`.
**The 83.8% GAIA 2 search result was not a self-grading artifact.** The harness-quality
signal is real and independently confirmed.
## Mapping to the full split (N=160 denominator)
| Metric | Self (Sonnet) | Trio-strict |
|---|---:|---:|
| n=148 answerable | 90.5% | 87.2% |
| **N=160 full (errors/unanswerable count against)** | **83.8%** | **~80.6%** (129/160) |
**Defensible public framing:** *"GAIA 2 search split, N=160 — 83.8% self-judged,
independently confirmed at 80.6% trio-strict (Opus 4.7 + Gemini 2.5 Pro + GPT-5,
GAIA 2's own checker, judges agree 97–99%)."*
## Caveat (carried forward)
This validates the **harness-cell judging**, but the harness was **Hermes** (a third-party
reference agent), NOT Waggle's own `runAgentLoop`. Pillar-1 (Waggle harness SOTA) still
requires the `waggle_worker` build to put Waggle's loop in the same rig. See plan doc.

View File

@@ -1,201 +0,0 @@
# Phase 4 P4.2 Progress — ARE-native architecture validated; runner polling has Windows Docker quirk
**Date:** 2026-05-21
**Branch:** `feature/gaia2-are-setup`
**Cumulative spend:** ~$2 (cap $80 hard / $50 halt — Marko ratification 2026-05-21)
**Author note:** Phase 4 P4.0 (gaia2-cli base build) + P4.1 (gaia2-hermes runtime build) + P4.2 (smoke partial pass)
---
## TL;DR
The ARE-native architecture **works as predicted**. Agent makes targeted app-API calls (Calendar, RentAFlat, MessagingAppV2, Contacts) instead of bulk `flattenAppStateToCorpus` retrieval. The Phase 3 narrow-proxy economics gap (9-31× over) is resolved by switching to gaia2-cli runtime images.
What's **NOT** working yet: the gaia2-runner's host-side `/status` polling at `127.0.0.1:8090` fails with `WinError 10061` on Windows Docker because the container has no host port mapping (runner uses `docker exec curl` for health but host TCP for status — mismatch). Judge daemon IS launching + reading events correctly inside the container; runner just can't detect "agent done" reliably from the host.
This is a **runner-host networking bug specific to Windows Docker**, not an architectural problem. Linux/macOS Docker handles host networking differently; this issue likely only manifests on Windows. Worth an upstream PR.
---
## §1 Setup completed
- ✅ **P4.0 gaia2-cli base image built:** `localhost/gaia2-cli:local` (983MB)
- ✅ **P4.1 gaia2-hermes runtime built:** `localhost/gaia2-hermes:latest` (1.75GB)
- ✅ **CRLF→LF conversion** on all `.sh` files in gaia2-cli/ (Windows line endings broke container entrypoints initially)
- ✅ **TOML `[agent].runtime = "docker"`** override added (default `podman` not installed on Marko's machine)
- ✅ **Anthropic API key** wired via `.env` at `gaia2-cli/.env`
- ✅ **Single-scenario debug** with `--judge-provider/--judge-model/--judge-api-key` explicit
---
## §2 Architecture validation (the load-bearing finding)
Single scenario `scenario_universe_21_5bftlu` — "How many of my saved properties have the same number of bedrooms as the average number of Chats conversations I've had with each of the attendees in events with 'coffee break' in the title."
**Agent execution trace (21 events captured):**
| App | Function calls |
|---|---|
| MessagingAppV2 | 15 (lookup_user_id ×5, list_conversations_by_participant ×3, search ×3, regex_search ×2, read_conversation ×1, get_user_name_from_id ×1) |
| Calendar | 2 (get_calendar_events_from_to ×1, search_events ×1) |
| Contacts | 2 (search_contacts ×1, get_current_user_details ×1) |
| RentAFlat | 1 (list_saved_apartments) |
| AgentUserInterface | 1 (send_message_to_agent — the question) |
**This is canonical ARE behavior.** Each call returns only requested data, not bulk app state. Compare to Phase 3 narrow-proxy:
| Metric | Phase 3 narrow-proxy | Phase 4 ARE-native (this run) |
|---|---|---|
| Tool-use pattern | `flattenAppStateToCorpus` → search top-K | Targeted app API calls |
| Tokens per turn | ~200KB-1MB (bulk retrieval) | ~2-5KB per call (focused returns) |
| Per-invocation cost (probe-validated) | **$4.09** | **~$0.50** (extrapolated from 21 calls × Sonnet 4.6 thinking high) |
| Multiple over original estimate | 9-31× | ~1-2× (in line) |
**Projected full N=160 search-split cost:** ~$25-80 (ARE-native) vs $300-500+ (narrow-proxy extrapolation). Well within the $80 Phase 4 budget cap.
---
## §3 Residual issue: judge polling on Windows Docker
Runner's `_poll_for_response` (`runner/gaia2_runner/runner.py`) makes HTTP calls to `127.0.0.1:8090/status` from the **host**, but the runner's `launcher.py` (around line 420-510) does NOT pass `-p HOSTPORT:8090` when starting the container. Only the runner's initial health-check uses `docker exec curl` (which works); status polling uses requests from the host (which doesn't).
**Symptoms:**
```
gaia2_runner.runner: Status poll error: HTTPConnectionPool(host='127.0.0.1', port=8090): Max retries exceeded with url: /status
(Caused by NewConnectionError("HTTPConnection(host='127.0.0.1', port=8090): Failed to establish a new connection:
[WinError 10061] No connection could be made because the target machine actively refused it"))
```
`docker port <container>` confirms **no port mapping** is published.
**In-container daemon is healthy** (verified by `docker exec ... cat /tmp/gaia2-eventd.log`):
- gaia2-eventd reading events.jsonl ✅
- Faketime advancing correctly ✅
- Waiting for turn boundary (would judge if it detected agent done) ✅
The agent eventually completes (16-21 tool calls observed) but the runner timeout-fires before the judge can write `daemon_status.json` because the runner never sees the agent's "done" signal from the unreachable `/status` endpoint.
**Why it likely only affects Windows Docker:** podman on Linux uses `--network=host` semantics where `127.0.0.1` from the host can reach into rootless container namespaces; Docker Desktop on Windows isolates the container in a VM, so host→container loopback only works via published ports. The runner was likely written/tested on Linux+podman where the loopback "just works."
---
## §4 Cost actuals so far
| Phase | Spend | Notes |
|---|---|---|
| P4.0 base build | $0 | Docker build (network bandwidth only) |
| P4.1 Hermes build | $0 | Docker build (network bandwidth only) |
| P4.2 v1+v2+v3 smoke (3 errored before agent ran) | $0 | Errored at health check, no API calls |
| P4.2 v3 smoke (3 ran, judge timeout) | ~$1.50 | Each ran ~16 LLM calls × ~$0.03 = ~$0.50/scenario |
| P4.2 debug N=1 (this scenario, killed mid-run) | ~$0.50 | 21 LLM calls before kill |
| **Cumulative** | **~$2** | Well under $50 halt-trigger |
---
## §5 Three paths forward (decision gate for Marko)
### Path A — Fix the runner upstream (1-2 hr)
Submit upstream PR to `meta-agents-research-environments` adding `-p $(host_port):$adapter_port` to `launcher.py` container launch, OR switching `_poll_for_response` to use `docker exec curl` like the health check. Once merged + re-installed via `uv sync`, Windows Docker should work fully.
**Pros:** Real fix, benefits all Windows users
**Cons:** External dependency on merge timeline; upstream maintainers may want a more thorough solution
### Path B — Monkey-patch locally (~30 min)
Edit `runner/gaia2_runner/launcher.py` and `runner.py` in our checkout. Add port mapping or switch poll path. Run with patched runner.
**Pros:** Unblocks N=10 probe today
**Cons:** Local-only fix; needs maintenance against upstream
### Path C — Switch to Linux runner (Marko-side or CI)
Run gaia2-runner on WSL2 or a Linux box (CI runner, cloud VM). Native podman or Docker namespace handling avoids the Windows quirk entirely.
**Pros:** Zero code changes, runner intended host
**Cons:** Adds infra complexity (WSL2 setup or remote runner)
**Recommendation: Path B for the N=10 probe THIS session** (unblocks the architecture-validates-cost claim with statistical signal), then Path A as a clean upstream contribution post-launch.
---
## §6 What's already verified for the launch narrative
- ✅ **ARE-native architecture is real, not theoretical.** Agent makes 21 targeted API calls per scenario, not 1MB bulk retrievals.
- ✅ **Cost projection holds.** ARE-native runs at ~$0.50/scenario vs narrow-proxy $4.09 → 8× cost reduction confirmed in actuals (160× projection upper bound).
- ✅ **gaia2-cli pipeline integrates with Waggle's dev environment.** Docker + uv + Hermes runtime image + HF dataset all work on Marko's Windows hardware.
- ⏸️ **Pass@1 grading deferred** until Path A/B/C resolves the runner polling.
---
## §7 Spend headroom for next steps
- Hard cap: $80
- Halt-trigger: $50
- Spent so far: ~$2
- **Headroom: $48 to halt / $78 to hard cap**
If Path B unblocks the probe, N=10 at ~$0.50/scenario = $5. N=40 (one-quarter of search split) = $20. N=160 (full search split) = $80 — at the hard cap; would need a budget amendment or scope reduction.
---
## Authority chain
- Phase 4 plan: `project_are_native_docker_architectural_solution.md` (memory anchor 2026-04-29)
- Phase 3 closure: `dry-run-results-memo.md` §9 (this directory)
- Budget ratification: Marko 2026-05-21 ($80 hard / $50 halt)
- Judge roster: Opus 4.7 / GPT-5.5 / MiniMax M2.7 (canonical from RESULTS.md, NOT the OPEN-TASKS-listed 4-judge variant)
---
## §8 P4.3 N=10 probe — RESULTS (2026-05-21 19:24)
Fired with 3 runner patches active (port publication + drop --network=host on Windows + UTF-8 artifact write).
**Aggregate:**
| Metric | Value |
|---|---:|
| Pass rate (strict, denom=10) | **8/10 = 80%** |
| Pass rate (judged-only, denom=8) | **8/8 = 100%** |
| Errors (missing daemon_status) | 2 — likely concurrency=5 port-race on Windows Docker |
| Total events across N=10 | 213 |
| Avg events/scenario | 21.3 |
| Wall-clock | ~10 min |
| Cost actual | ~$5.50 |
| Cumulative Phase 4 spend | ~$8 of $50 halt / $80 hard cap |
**Mem0 paper baseline for Hermes + Sonnet 4.6 on the search split is ~40-55% pass@1.** Our 80% is significantly above baseline at N=10 — CI is wide (~±25pp at N=10) but the result is consistent with the substrate-and-architecture validation hypothesis.
**Per-scenario verdicts:**
| Scenario | Verdict |
|---|---|
| 5bftlu | PASS |
| 7306ow | PASS |
| 7zrdq1 | PASS |
| 95xj34 | PASS |
| a7j0iw | PASS |
| csyctc | PASS |
| eo7tr6 | PASS |
| er2clq | PASS |
| 1afh09 | MISSING JUDGE (error) |
| bnrehm | MISSING JUDGE (error) |
The 2 errors are scenarios where the in-container judge daemon didn't complete writing daemon_status.json before the runner tore down the container — same root cause as the original v3 smoke fail, but only manifesting on 2/10 with the patched runner (was 3/3 before). Likely solvable with concurrency=1 OR a larger health_timeout. Worth one more iteration before committing to N=160.
**Headline projection update:** at $0.55/scenario actual + 10% error rate worst case, **full N=160 probe ≈ $88-100** — slightly over the $80 hard cap. Would need either: (a) a budget amendment to $100-120, (b) scope reduction to N=80, or (c) tighten the concurrency-vs-error tradeoff to push error rate < 5%.
---
## §9 Recommended next step (Marko decision gate)
Three live options for P4.5:
1. **Tighten + go to full N=160** — re-run with concurrency=2 (instead of 5) to eliminate the port-race; that adds wall-clock time but should drop errors to ~0. Budget ask: $100 amendment (vs $80 current cap). Most rigorous.
2. **N=40 sample at concurrency=5** — accepts ~10% error rate, stops at quarter of the split. Budget: ~$25. Demonstrates substrate-claim validation at a more meaningful N than 10. Fits current budget.
3. **Stop here, publish N=10 result** — call P4.3 the deliverable. Headline: "ARE-native architecture validated at N=10, 80% strict / 100% judged-only pass rate, total cost $5.50." Light on statistical power but the architecture story is told. Defer full run to a fresh budget cycle.
**My recommendation:** Option 2 (N=40 at concurrency=5). Best risk-adjusted scope — meaningful statistical signal (CI ~±15pp at N=40) without budget overrun. Concurrency error rate is documented; doesn't invalidate the architecture claim.

View File

@@ -1,80 +0,0 @@
# Phase 4 P4.5 — GAIA 2 search split FULL RUN N=160 RESULTS
**Date:** 2026-05-22
**Branch:** `feature/gaia2-are-setup`
**Agent:** Hermes runtime (`localhost/gaia2-hermes:latest`) + Anthropic `claude-sonnet-4-6`, thinking=high
**Judge:** `claude-sonnet-4-6` (in-container daemon)
**Dataset:** `meta-agents-research-environments/gaia2-cli`, split `search`, N=160 (full split)
**Cost:** ~$91 extrapolated (~165 scenario-executions × ~$0.55), under the $100 hard cap (Marko amendment 2026-05-21)
---
## Headline
| Metric | Value |
|---|---:|
| **Strict pass rate** | **134/160 = 83.8%** |
| **Judged-only pass rate** | **134/155 = 86.5%** (excludes 5 harness-undecidable) |
| FAIL | 21 |
| ERROR (no turn boundary) | 5 |
| Mem0 paper baseline (Hermes + Sonnet, search split) | ~40–55% pass@1 |
**The ARE-native architecture clears the Mem0 baseline by ~30–45pp at full N=160.** CI at N=160 is
~±6pp (Wilson, p=0.84), so the result is robust — not a small-N artifact.
---
## How this run completed (provenance — it was a two-part run)
1. **First pass** (`waggle_phase4_full_n160.toml`, concurrency=2, 2026-05-21 20:02 → 2026-05-22 01:43):
reached **141/160** before the CC session crashed and killed the runner process.
Partial: 117 PASS / 19 FAIL / 5 ERROR. All 141 `result.json` survived on disk.
2. **Finish pass** (`waggle_phase4_finish_subset.toml`, 2026-05-22 03:35 → 04:45): a **deterministic
24-scenario subset** (the 5 errored + 19 never-run), written to the SAME output dir so the
results merge into a clean N=160 aggregate.
### Why a subset manifest instead of `--retry`
The runner's built-in `--retry` over-selected: it picked **48 scenarios** = the correct 24
(missing + errored) PLUS **24 already-terminal PASS/FAIL** that it re-queued needlessly. That would
have cost ~$26 (breaching the $100 cap) and re-rolled good verdicts. Root cause not fully isolated
(paths are uniform `search/<id>/result.json`, so it is not a directory-structure mismatch — the
`_select_retry_scenarios` selector misclassifies a specific 24-scenario subset). The subset-manifest
path (`[target].subset_manifest` listing exact IDs) sidesteps it entirely and is deterministic.
---
## The 5-error floor (honest framing)
Errors are all `Daemon error: no turn boundary detected` — the agent ran real work (12–41 tool calls)
but the in-container judge daemon never observed a clean turn close before idle-timeout. **Re-running
swapped *which* scenarios error rather than reducing the count** (2 of the original 5 resolved;
2 new appeared), so this is a stable ~3% runner turn-detection floor on long scenarios, **not** agent
failure and **not** flaky randomness.
Persistent errors at N=160: `27_azdwq4`, `28_ofdzab`, `29_f3f586`, `30_48xm0t`, `30_o7av0c`.
Honest reporting: **155 judged, 5 undecidable on this harness** → report both the strict (83.8%,
counts errors against us) and judged-only (86.5%, excludes them) numbers. Strict is the defensible
public figure; judged-only is the architecture-capability figure.
---
## Open follow-ups (not blocking the result)
- **4th Windows unicode gap:** `trace_viewer.py` HTML generation fails on `▸` (▸) with cp1252 —
emitted as WARNINGs, affects only the per-scenario trace HTML, NOT `result.json` grading data.
Patch 3 (runner.py artifact writes) does not cover the trace viewer. Logged in
`WINDOWS-DOCKER-RUNNER-PATCHES.md` as the 4th patch needed for the upstream PR.
- **Turn-boundary floor:** worth an upstream issue — daemon should treat a long idle after N tool
calls with no further events as a soft turn-close + judge, rather than erroring.
---
## Authority chain
- Budget: Marko $80→$100 amendment 2026-05-21
- Architecture validation: `PHASE-4-P4.2-PROGRESS-2026-05-21.md` §2 (ARE-native, targeted API calls)
- N=10 probe: `PHASE-4-P4.2-PROGRESS-2026-05-21.md` §8 (8/10 strict, 8/8 judged-only)
- Runner patches: `WINDOWS-DOCKER-RUNNER-PATCHES.md`
- Artifacts: `runs/p4-full-hermes-n160/` (160 scenario dirs + index.html + results.jsonl)

View File

@@ -1,221 +0,0 @@
# Pillar 1 — Qwen 3.6 35B-A3B follow-up runbook
**Goal:** API-served Qwen 3.6 number for the Waggle harness on GAIA 2 search split.
Repeat the **2026-05-22 Pillar-1 ON-PAR run** (Waggle harness + Sonnet 4.6 trio-strict **86.5% at N=40**) but with **Qwen 3.6 35B-A3B via LiteLLM → DashScope-intl direct** instead of Sonnet via OpenRouter.
## 2026-05-26 PIVOT — API-served, not local
This runbook originally scaffolded a **local-Ollama** path. We pivoted to **API-served via LiteLLM** on 2026-05-26 because (1) the LiteLLM proxy already holds all credentials, (2) DashScope-intl direct delivers the TRUE Qwen 3.6 (the OpenRouter route silently regresses to Qwen 3.5 per `models.json:43`), and (3) eliminates the gateway-confound caveat from the Sonnet baseline by routing both agent and judge through one proxy.
The sovereign-local variant is preserved at the bottom as an optional follow-up.
## Smoke verdict (2026-05-26 17:14)
Plumbing **PASS**. Single-scenario judge **inconclusive** on the known-flaky `21_1afh09`:
| Signal | Result |
|---|---|
| LiteLLM → DashScope-intl auth | ✅ keys resolved via `LITELLM_MASTER_KEY` |
| Worker → ARE adapter socket | ✅ scenario lifecycle clean |
| Qwen 3.6 agent engagement | ✅ **28 agent events**, coherent multi-step reasoning ("count 31 Shanghai contacts → sum ages 1402 → avg 45.23 → round 45") |
| Judge route via LiteLLM | ✅ ran, returned `inconclusive` (not "wrong") |
| Cost | ✅ ~$0.05 |
Scenario `21_1afh09` was flagged in `PILLAR1-WAGGLE-VS-HERMES-N40-2026-05-22.md` as "flipped between runs" — its `user_message_checker` is unstable on phrasing. Trio-rejudge typically resolves these. **Smoke is GREEN.**
---
## Why this is single-knob
Worker (`waggle-container/waggle_worker.mjs`) is fully env-driven. Runner `container_env.py` `_DEFAULT` profile injects from the TOML `[agent]` block into the container:
| Env var | Sonnet baseline (2026-05-22) | Qwen 3.6 (this run) |
|---|---|---|
| `MODEL` | `claude-sonnet-4-6` | `qwen3.6-35b-a3b` (LiteLLM alias) |
| `BASE_URL` | `https://openrouter.ai/api/v1` | `http://host.docker.internal:4000/v1` (LiteLLM proxy) |
| `API_KEY` | OpenRouter key | `sk-waggle-dev` (LiteLLM master key) |
LiteLLM alias `qwen3.6-35b-a3b` routes to `openai/qwen3.6-35b-a3b` at `https://dashscope-intl.aliyuncs.com/compatible-mode/v1` (`litellm-config.yaml:211` — Sprint 12 sibling of the deprecated `via-openrouter` route).
Same `AGENTS.md` renders, same single `terminal` tool, same `runAgentLoop`. **Only the model differs.** Strict fairness with the 86.5% Sonnet baseline.
---
## Prerequisites (all verified 2026-05-26)
1. **LiteLLM proxy stack up.** From repo root: `docker-compose up -d` brings up `litellm` (port 4000), `postgres`, `redis`, `minio`. Confirm: `curl http://localhost:4000/health` returns auth-required error (= healthy + listening).
2. **`localhost/gaia2-waggle:latest` built.** Verify: `docker images | grep gaia2-waggle`. Rebuild via `waggle-container/BUILD.md` if the worker or `@waggle/agent/dist` has changed since last build.
3. **`LITELLM_MASTER_KEY` set in shell.** Default `sk-waggle-dev` per `.env` and `docker-compose.yml`.
4. **gaia2-runner available.** At `external/meta-agents-research-environments/gaia2-cli/runner/.venv/Scripts/gaia2-runner.exe` (Windows) — installed via `uv sync --frozen`.
5. **GAIA 2 dataset cached.** First run pulls from HuggingFace. Cached at `~/.cache/gaia2/hf_datasets/`.
---
## TOMLs
Working copies live in `external/meta-agents-research-environments/gaia2-cli/runner/examples/` (`.gitignored` per the existing pattern; embedded verbatim below for reproduction on any fresh checkout).
### Smoke (n=1) — `waggle_qwen36_smoke_n1.toml`
```toml
[target]
dataset = "meta-agents-research-environments/gaia2-cli"
splits = ["search"]
limit = 1
[agent]
image = "localhost/gaia2-waggle:latest"
runtime = "docker"
provider = "openai-compat"
model = "qwen3.6-35b-a3b"
api_key_env = "LITELLM_MASTER_KEY"
base_url = "http://host.docker.internal:4000/v1"
thinking = "high"
[judge]
provider = "openai-compat"
model = "claude-sonnet-4-6"
api_key_env = "LITELLM_MASTER_KEY"
base_url = "http://localhost:4000/v1"
[run]
timeout = 1800
health_timeout = 180
concurrency = 1
pass_at = 1
output_dir = "D:/Projects/waggle-os-gaia2-wt/benchmarks/gaia2/runs/waggle-qwen36-smoke-n1"
log_level = "INFO"
```
### Full N=160 — `waggle_qwen36_n160.toml`
```toml
[target]
dataset = "meta-agents-research-environments/gaia2-cli"
splits = ["search"]
limit = 160
[agent]
image = "localhost/gaia2-waggle:latest"
runtime = "docker"
provider = "openai-compat"
model = "qwen3.6-35b-a3b"
api_key_env = "LITELLM_MASTER_KEY"
base_url = "http://host.docker.internal:4000/v1"
thinking = "high"
[judge]
provider = "openai-compat"
model = "claude-sonnet-4-6"
api_key_env = "LITELLM_MASTER_KEY"
base_url = "http://localhost:4000/v1"
[run]
timeout = 1800
health_timeout = 180
concurrency = 2
pass_at = 1
output_dir = "D:/Projects/waggle-os-gaia2-wt/benchmarks/gaia2/runs/waggle-qwen36-n160"
log_level = "INFO"
```
Difference: `limit` and `concurrency` only. Output dirs distinct.
---
## Run sequence
```powershell
# 0. Confirm LiteLLM stack
docker ps --format '{{.Names}} ({{.Status}})' | Select-String 'litellm|postgres|redis|minio'
# 1. Set the master key for the runner's host shell
$env:LITELLM_MASTER_KEY = "sk-waggle-dev"
# 2. Smoke first (~3-5 min wall, ~$0.05)
cd D:/Projects/waggle-os/external/meta-agents-research-environments/gaia2-cli
& "runner/.venv/Scripts/gaia2-runner.exe" run-config `
--config "runner/examples/waggle_qwen36_smoke_n1.toml"
# 3. Inspect smoke
Get-Content D:/Projects/waggle-os-gaia2-wt/benchmarks/gaia2/runs/waggle-qwen36-smoke-n1/results.jsonl
# 4. If smoke clean (plumbing + non-zero agent events) -> full run (~3-5 hrs wall, ~$5-8)
& "runner/.venv/Scripts/gaia2-runner.exe" run-config `
--config "runner/examples/waggle_qwen36_n160.toml"
# 5. Trio-strict rejudge
cd D:/Projects/waggle-os-gaia2-wt/benchmarks/gaia2
python rejudge_user_message.py `
--runs-dir runs/waggle-qwen36-n160 `
--judges opus-4-7,gpt-5-4,gemini-2.5-pro `
--output runs/rejudge-waggle-qwen36-n160.jsonl
```
---
## Gate criteria
| Gate | Pass condition |
|---|---|
| **Smoke plumbing** | LiteLLM auth resolves; container completes lifecycle; agent emits ≥1 event; judge runs (even if inconclusive). The 28-event scenario `21_1afh09` smoke met this on 2026-05-26. |
| **Full run health** | error-rate < 10% (per-scenario container exits or judge errors). Concurrency=2 means 2 docker containers + 2 simulated app daemons in parallel — `entrypoint.log` per scenario captures issues. |
| **Headline** | trio-strict score with 95% CI. **No threshold** — this is "where does API-served Qwen 3.6 land," not a hypothesis test. Sonnet ON-PAR (86.5%, N=40, trio-strict) is the comparator. A trio-strict in the **75-85% band** would be a strong sovereign-eligible result; lower than that frames Pillar 1 as model-bound and worth focused harness work. |
---
## Known risks & open assumptions
1. **`runAgentLoop` against LiteLLM-proxied DashScope** — smoke proved roundtrip works at the OpenAI-compat surface, with tool calls flowing correctly. Full-N parallelism is the next stress test: 2 containers competing for LiteLLM throughput shouldn't hit rate limits at N=160 / 2 = 80 sequential calls per stream.
2. **DashScope 16K thinking-tokens cap** (per `models.json:55`, Stage 2 Mini Retry v3 manifest §2.1) — reduced from 64K to avoid tail-latency timeouts. If the agent loop blows through this on multi-step scenarios, expect `loop_exhausted` failures similar to the 2026-04-30 dry-run-results-memo §3 finding.
3. **Concurrency=2 GPU/cost contention** — DashScope is API-served, no local GPU constraint. Cost stays linear with N (~$5-8 total at $0.20+$0.80/M).
4. **No prompt-shape change** — Waggle's GAIA 2 worker uses `AGENTS.md` + single `terminal` tool. The `qwen-thinking.ts` PromptShape lives in `packages/agent/src/prompt-shapes/` and is a LoCoMo retrieval framework, **not** the GAIA 2 shape. Do not wire it in here — that would break fairness with the 86.5% Sonnet baseline.
5. **Scenario `21_1afh09` flakiness** — known unstable per the N=40 memo. Single-instance failure carries no signal; trio-rejudge resolves most "inconclusive" verdicts.
---
## Comparator targets
| Cell | N | trio-strict | source |
|---|---:|---:|---|
| Hermes (reference) + Sonnet 4.6 | 40 (matched subset of 160) | **89.2%** | `PILLAR1-WAGGLE-VS-HERMES-N40-2026-05-22.md` |
| **Waggle + Sonnet 4.6** | 40 | **86.5%** | same memo |
| Hermes + Sonnet 4.6 | 160 (full search) | 83.8% strict / 86.5% judged-only | `PHASE-4-P4.5-RESULTS-N160-2026-05-22.md` |
| **Waggle + Qwen 3.6 35B-A3B (API via LiteLLM→DashScope)** | 160 | **TBD** | this runbook (fired 2026-05-26) |
The third row is the natural comparator for the new API-served Qwen 3.6 number. A defensible launch claim: *"Waggle harness lands within ε of the Hermes-Sonnet ceiling even with a sovereign-eligible 35B model"* — provided ε is small enough. Defining "small enough" is the next round of analysis once the trio-strict number is in.
---
## Optional follow-up: sovereign-local variant via Ollama
If a fully-local number is later needed (no cloud API surface), the variant just swaps `[agent]`:
```toml
[agent]
image = "localhost/gaia2-waggle:latest"
runtime = "docker"
provider = "openai-compat"
model = "qwen2.5:32b" # FIXME: confirm exact Ollama tag for Qwen 3.6 35B thinking
api_key_env = "OPENAI_COMPAT_API_KEY" # set to any non-empty string ("ollama" works)
base_url = "http://host.docker.internal:11434/v1"
thinking = "high"
```
Prerequisites: `ollama pull <qwen-3.6-thinking-tag>`, plus `concurrency=1` (single GPU serializes) → wall time ~3-15h instead of ~3-5h. Cost is $0 on agent side, judge stays the same (~$2 in-container Sonnet). Output dir suffix changes to `-ollama-local` to keep the API-served and sovereign-local runs distinct.
---
## Provenance
- Initial scaffold 2026-05-26 (Ollama-only): commit `b4e4354` on `feature/gaia2-are-setup`.
- 2026-05-26 PM pivot to API-served via LiteLLM → DashScope: this revision.
- Smoke fired 2026-05-26 17:14 (task ID `benccwrgl`): plumbing PASS, scenario `21_1afh09` judge inconclusive (known flaky).
- N=160 fired 2026-05-26 17:19 (task ID `blt6winb3`): in progress at the time this runbook update was written; result memo to follow under `PILLAR1-QWEN36-N160-RESULT-2026-05-26.md`.
- Worker entry: `benchmarks/gaia2/waggle-container/waggle_worker.mjs` (HEAD `6479dfa`).
- ARE container_env: `external/meta-agents-research-environments/gaia2-cli/runner/gaia2_runner/container_env.py` `_DEFAULT` profile.
- LiteLLM alias source: `litellm-config.yaml:211` (Sprint 12 Task 2 C3 Stage 2 mini).

View File

@@ -1,122 +0,0 @@
# Pillar 1 — Waggle harness + Qwen 3.6 35B-A3B on GAIA 2 search · N=160 result
**Run fired:** 2026-05-26 17:19 (task ID `blt6winb3`) · **Trio-rejudge completed:** 2026-05-27 06:35
**Setup:** Waggle harness (`waggle_worker.mjs` + `runAgentLoop`) · Qwen 3.6 35B-A3B via LiteLLM → DashScope-intl direct · thinking="high" · single `terminal` tool · same AGENTS.md as the 2026-05-22 Sonnet baseline.
---
## Headline — trio-strict
| Cell | N | trio-strict | source |
|---|---:|---:|---|
| Hermes (reference harness) + Sonnet 4.6 | 148 | **87.2%** | `rejudge-search-n160.jsonl` (P4.5) |
| **Waggle harness + Sonnet 4.6** | 39 | **84.6%** | `rejudge-waggle-n40.jsonl` (PILLAR1 N=40) |
| **Waggle harness + Qwen 3.6 35B-A3B thinking** | **156** | **67.9%** | `rejudge-waggle-qwen36-thinking-n160.jsonl` (this run) |
Same harness (Waggle), model swap Sonnet→Qwen: **−16.7pp** on the matched N=38 subset.
Sovereign-eligible vs published-SOTA frontier (Hermes+Sonnet): **−19.3pp** on matched N=145.
**Trio judges were essentially unanimous** on Qwen: 106 unanimous PASS / 49 unanimous FAIL / 1 single-judge split. trio-strict ≡ trio-majority for Qwen — failures are *decisive*, not borderline phrasing disputes.
---
## Matched-pair breakdown
### vs Hermes+Sonnet on N=145 common scenarios
| | count | note |
|---|---:|---|
| Both PASS | 88 | |
| Qwen FAIL · Hermes PASS | **39** | the gap |
| Qwen PASS · Hermes FAIL | 10 | Qwen wins |
| Both FAIL | 8 | scenario-hard |
Net gap = 29 net losses out of 145 ≈ −20pp.
### vs Waggle+Sonnet on N=38 common scenarios (pure model attribution, same harness)
| | count | note |
|---|---:|---|
| Both PASS | 25 | |
| Qwen FAIL · Sonnet PASS | **8** | pure model-attributable gap |
| Qwen PASS · Sonnet FAIL | 2 | |
| Both FAIL | 3 | |
Net same-harness gap = 6/38 ≈ −15.8pp. Within tight CIs of the full-N gap above.
---
## Where Qwen is weaker — categorical failure modes
Surveyed 7 failing scenarios (3 same-harness gap, 4 vs-Hermes gap) on disk at `runs/waggle-qwen36-thinking-n160/search/`. Same root failure mode recurs:
### Cat 1 — **Verbose multi-paragraph final answer** (≈6 of 7 sampled, dominant)
Qwen's final `send_message_to_user` ranges from 300–2000 characters: includes a "thinking preface" (*"Now let me analyze the data. I need to:"*, *"Based on my analysis:"*), bulleted analysis, headers, and the answer buried inside or at the end. The GAIA 2 deterministic `user_message_checker` returns `inconclusive` because the message doesn't crisply match the expected answer pattern. The trio LLM judges, more lenient on phrasing, *also* score these as wrong — Qwen often gets the actual answer wrong on top of being verbose (e.g., `21_er2clq`: answered "Stockholm" — the user's own city — apparently confusing contact Astrid Lindqvist with the user Astrid Lundqvist).
**Sonnet self-disciplines.** Qwen does not. Examples:
- `21_er2clq` · 1992 chars · begins "Now let me analyze the data. I need to:"
- `21_otvqov` · 1303 chars · "Based on my analysis:" + bullet list of message participants
- `22_auk06f` · 532 chars · "Based on my analysis:" + bold city/zip breakdown
- `27_9yg3xx` · 333 chars · multi-paragraph + "**Answer:** ..." suffix
- `28_y6gxdt` · 665 chars · numbered analysis list
### Cat 2 — **Tool-call JSON malformation → run crashes after 2 events**
At least one scenario (`22_1xhz8j`) crashed with a DashScope 400:
```
litellm.BadRequestError: OpenAIException - <400> InternalError.Algo.InvalidParameter:
The "function.arguments" parameter of the code model must be in JSON format.
```
`events=2`, `oracle=0`. Qwen emitted a tool call whose `function.arguments` was not valid JSON; DashScope rejected; the agent loop produced no further events and the final answer is the raw error string. Sonnet doesn't trigger this class of failure.
### Cat 3 — **Thinking-mode bleed**
Several Qwen final answers begin with first-person planning text ("Now let me…", "I need to…") that should have lived inside the `<think>` block, not in the user-facing message. Suggests the worker is forwarding the entire model output rather than parsing/stripping a `<think>…</think>` envelope, or that Qwen 3.6 35B-A3B thinking-high doesn't always emit a clean separator.
---
## Are these harness-fixable?
| Fix | Class | Expected closure | Risk |
|---|---|---|---|
| **H-1 · Final-answer extraction + reformatting shim** in `waggle_worker.mjs` | post-processing | Most of the 39-scenario gap | Need to re-baseline Sonnet with the same shim to keep fairness, OR apply Qwen-only and disclose |
| **H-2 · Tool-call JSON validator + retry** before forwarding to LiteLLM | pre-flight validation | Some unknown count of `events=2` crashes | None — pure defensive guard |
| **H-3 · System-prompt format hardening** in AGENTS.md ("final answer = single concise value, no prose") | prompt-shape | Overlaps with H-1, additive | Same fairness re-baseline question as H-1 |
| **H-4 · Thinking-envelope parser** that strips `<think>…</think>` from the final response | post-processing | Some unknown count of Cat 3 leaks | None — only fires when envelope present |
**The pure-model gap floor:** the 8 same-harness failures are mostly Cat 1 (verbose answers + content errors). H-1 and H-3 together could close *some* of them (where the right answer is buried in the prose and a strip-to-value step would surface it) but not all (where Qwen's actual reasoning was wrong). Rough estimate without doing the work: H-1 + H-3 + H-2 closes 10–25 of the 39 vs-Hermes gap scenarios — moves Qwen-Waggle from 67.9% to roughly **74–83%**, putting it in the "75-85% sovereign-eligible" band the runbook gate criteria framed as defensible.
**The principled comparison:** if we ship H-1/H-3 we MUST also re-run Sonnet through the same shim. Otherwise the comparison is unfair. Cheapest defensible cell: rerun Waggle+Sonnet N=40 with the shim, see if Sonnet stays at 84.6% (probable; Sonnet doesn't need the rail) or also shifts. Then the matched-pair stays clean.
---
## Recommended next step (PM-grade pick list)
| Option | Effort | Yields |
|---|---|---|
| **A. Ship H-2 only** (tool-call JSON validator) | 0.5 day | Closes the crash-class failures; safe ON for any model; no fairness re-baseline needed |
| **B. Ship H-1 + H-3 + H-2 then rerun Qwen N=40 + Sonnet N=40 with shim** | 2 days | Closes the verbose-answer gap; rigorous matched-pair comparison; defensible launch number |
| **C. Accept 67.9% as the sovereign Qwen number** and frame Pillar 1 as "Waggle harness on sovereign 35B model lands at 67.9%, model-bound not harness-bound" | 0 days | Honest framing; preserves Sonnet 86.5% as published-frontier headline |
| **D. Skip Qwen-on-API entirely**, pivot to local-Ollama Qwen variant (the original runbook scaffold `b4e4354`) | days, queued on Ollama serving | Different sovereignty story (no cloud); same underlying model gap |
**Default recommendation:** **B**, but pre-flight with **A** as the cheap safety net. The harness work is genuinely Waggle-product-improving (H-2 protects ANY future model swap; H-1/H-3 make Waggle better at orchestrating non-Sonnet models, which is the whole sovereign-eligible thesis). Once B's measurement is in, decide between publishing the higher number (B's result) or the conservative one (C). Either way, H-2 is free upside.
---
## Cost & error log
- Run wall: fired 2026-05-26 17:19 → rejudge finished 2026-05-27 06:35 → ~13h elapsed wall.
- This includes both the original N=160 run + the trio-rejudge phase (Opus 4.7, Gemini 2.5 Pro, GPT-5.x via LiteLLM).
- Cost: not yet reconciled. Per runbook estimate $5-8 for the run + ~$3-4 for trio-rejudge.
- 4 scenarios from the 160 are missing from the rejudge file (N=156) — likely judge errors or scenarios that errored in-container; the script logs would clarify but it's <3% and not material to the headline.
## Pointers
- Headline data: `runs/rejudge-waggle-qwen36-thinking-n160.jsonl`
- Per-scenario output: `runs/waggle-qwen36-thinking-n160/search/scenario_universe_*/{result.json,agent_response.txt,events.jsonl,entrypoint.log}`
- Sonnet baselines: `runs/rejudge-search-n160.jsonl` (Hermes), `runs/rejudge-waggle-n40.jsonl` (Waggle)
- Source runbook: `benchmarks/gaia2/PILLAR1-QWEN-LOCAL-RUNBOOK.md` (pivot commit `c58f919`)
- Worker entry: `benchmarks/gaia2/waggle-container/waggle_worker.mjs`
- runAgentLoop: `packages/agent/src/agent-loop.ts` (inside the container build)
## Provenance
- Original Sonnet-baseline memo: `PILLAR1-WAGGLE-VS-HERMES-N40-2026-05-22.md` (Waggle on par with Hermes, 86.5% vs 89.2%)
- This memo's gap analysis is conservative (7-sample qualitative review). A full per-scenario taxonomy across all 39 vs-Hermes gap scenarios would refine the H-1/H-2/H-3 closure estimate but isn't required to choose between options A/B/C/D.

View File

@@ -1,169 +0,0 @@
# Pillar 1 — Full Waggle harness ladder · Qwen 3.6 35B-A3B · 2026-05-27
**Question.** The 2026-05-26 N=160 ran with **bare** Waggle (gates OFF, no persona, no prompt-shape) — fairness-with-Hermes design but disclaiming the "as-shipped" product surface. Result: **67.9% trio-strict** on Qwen vs 87.2% on Hermes+Sonnet → 19.3pp gap. Marko's correction: *test the **real** Waggle harness* — turn on Waggle's actual product-distinctive orchestration. This memo measures four cells at N=20 to attribute the gap and pick the lever.
---
> ## ⚠️ CORRECTION (2026-05-28) — the N=20 ladder below is a FALSE POSITIVE
>
> **The F3 prompt-shape result did NOT survive stratified scale-up. See §F4 at the bottom for the authoritative numbers.**
>
> - **F4 Waggle+Qwen+F3 at full N=157: 65.6% trio-strict** (95% CI 57.9-72.6%) — statistically **FLAT** vs bare 67.9% (bare inside the CI), net **−3.2pp** on matched-pair (16 recover / 21 regress).
> - The N=20 "+10.5pp" was **prefix-sampling bias** (GAIA scenarios are ordered by universe 21→30; `limit=20` drew 17/20 from universe_21) compounded with **run-to-run nondeterminism** (Qwen-thinking at temperature). On the *full* universe_21 set, F3 scores 70.6% vs bare 82.4% — it HURTS the very universe the probe claimed it helped.
> - **F4 Sonnet+F3 N=40 = 97.5% is UNCONFIRMED** — it ran on the same biased universe-21-23 prefix. Needs a stratified N≥120 to trust.
> - **Methodology lesson:** an N=20 gate on a non-stratified prefix is not a valid scale-up signal. The GAIA split must be stratified-sampled or run in full.
>
> The four-cell table immediately below is preserved as the (misleading) evidence that motivated F4, NOT as a result.
## Headline result (N=20 PROBE — SUPERSEDED, see correction above)
| Cell | What's on | trio-strict N=20 | matched ∆ vs bare-same-19 |
|---|---|---:|---:|
| Bare (control) | nothing | 73.7% (full-N=156: 67.9%) | — |
| **F1** | + WAGGLE_VERIFICATION_GATE + WAGGLE_SKILL_DISTILLATION_GATE | 70.0% | **net 0pp** (5↔5 cancel) |
| **F2** | + WAGGLE_PERSONA_ID=executive-assistant (composePersonaPrompt) | 75.0% | **+5.3pp** |
| **F3** | + WAGGLE_GAIA2_QWEN_SHAPE=1 (output-discipline appendix) | **85.0%** | **+10.5pp** |
**Trio judges unanimous on F3:** 17 PASS + 3 FAIL, 0 splits. Real content lift, not phrasing artifact.
## Three product findings worth landing
### Finding 1 — In-container ≠ trio-strict; only trio is fairness-defensible
The GAIA 2 in-container `user_message_checker` is a deterministic format checker, not a content judge. It returns `inconclusive` on verbose multi-paragraph answers EVEN WHEN CORRECT. F1's in-container rate was **10%**; trio-strict was **70%**. F2's in-container **5%** → trio **75%**. F3's in-container **20%** → trio **85%**. The deterministic checker's signal correlates with the prompt-shape's success at producing crisp answers, NOT with whether Qwen got the right answer. Earlier session entries calling F1 a "catastrophic regression" based on in-container were wrong and retracted in real time. **Only trio-strict matched the Hermes baseline methodology; only trio-strict matters for the final number.**
### Finding 2 — Waggle's "as-shipped" gates do NOT regress Qwen content; they trade phrasing crispness for reasoning shuffle
F1 (verification + D1 skill distillation, both ON) produced **0pp net change** on the matched-19 subset. 5 scenarios that bare passed, F1 failed; 5 scenarios bare failed, F1 passed — different reasoning paths produce different scenarios. The gates DO induce longer scenarios (event counts 100-600 vs bare's 10-30 on hard scenarios) at extra cost and latency, without trio-strict benefit on Qwen. The earlier hypothesis "gates regress on non-Sonnet models" is **partially correct**: they regress on the deterministic checker (verbose answers, format-broken) but NOT on the LLM-judge content metric. Product implication: **the F1 gates' true cost is latency and token spend, not correctness**, and they can be safely shipped with model-aware toggles.
### Finding 3 — The output-discipline prompt-shape is the highest-leverage harness lever for Qwen
F3 is a single env var (`WAGGLE_GAIA2_QWEN_SHAPE=1`) that appends ~30 lines of explicit final-answer-format rules to the system prompt. Worker code change: 7 lines + a string constant. No agent-loop change, no LLM logic change. Result: **+10.5pp trio-strict matched lift, +11.3pp standalone**. The shape works partially (4/20 produced one-token answers like `45` / `Xóchitl`; 16/20 still produced multi-paragraph "Based on my analysis…" responses despite the prompt prohibition), but the partial discipline AND the trio-strict lift confirm Cat 1 (verbose final answers) is the dominant residual gap and harness-fixable.
## Per-scenario evidence (4-way matched, N=19)
```
B=bare 1=F1 2=F2 3=F3 recovery vs bare
PPPP × 8 — easy / robust to all variants
PPPF × 1 (5bftlu) — F3-only regression (overcompressed: "1" not the answer)
PFPP × 3 — F1 broke, F2+F3 keep — confirms F1 is structurally hurting some scenarios
PFFF × 1 (bnrehm) — model-hard, all enhancements fail
FPPF × 1 (er2clq) — F1 saved it but F3 broke it; persona/shape interact non-linearly
PPFP × 1 (ew5kn5) — F2 alone broke it; F3 keeps it
FPFP × 2 (otvqov, 1xhz8j) — F2 doesn't help; F3 recovers ⬆️
FPPP × 2 (powgzh, 346yda) — all variants except bare get these right ⬆️
```
**5 bare failures total in N=19:**
- `er2clq` (Stockholm conflation): F3 fails — model-attributable, content-wrong even when terse
- `otvqov` (messages participants city): **F3 recovers** ⬆️
- `powgzh` (employed Stockholm contacts): **F3 recovers** ⬆️
- `1xhz8j` (rides booked, ZIP/violent-crime): **F3 recovers** ⬆️
- `346yda` (some entity lookup): **F3 recovers** ⬆️
4 of 5 bare failures recovered under F3 — and the one that doesn't (`er2clq`) is a genuine Qwen content error, not a Cat 1/3 surface error.
## Wiring (committed `6db922d` + working tree pending commit)
Worker rewire in `benchmarks/gaia2/waggle-container/waggle_worker.mjs`:
- F2 imports: `getPersona`, `composePersonaPrompt` from `@waggle/agent/dist/personas.js`
- F2 logic: `WAGGLE_PERSONA_ID` resolved once at startup; `buildSystemPrompt()` composes AGENTS.md + persona
- F3 logic: `WAGGLE_GAIA2_QWEN_SHAPE=1` appends `QWEN_SHAPE_APPENDIX` after persona compose (so shape wins format authority)
- Unset / empty values preserve the 2026-05-22 bare-Waggle baseline behavior
Container env in `external/.../gaia2-cli/runner/gaia2_runner/container_env.py`:
- `_DEFAULT.extra_flags` is the single switch — set to `{"WAGGLE_GAIA2_QWEN_SHAPE": "1"}` for F3, `{"WAGGLE_PERSONA_ID": "executive-assistant"}` for F2, `{"WAGGLE_VERIFICATION_GATE": "1", "WAGGLE_SKILL_DISTILLATION_GATE": "1"}` for F1, `{}` for bare.
Container `localhost/gaia2-waggle:latest` rebuilt 3× during this session: Phase 0 (fix#4 dist), F2 (worker rewire), F3 (shape constant). Final image at run time = F3 build.
## Cost log
- Phase 0 smoke ($0.05) + N=20 F1 (~$1.50) + N=20 F2 (~$1.20) + N=20 F3 (~$1.10) + 3× trio rejudges (~$0.50 each)
- Total session compute: ~$5-6
- Container rebuilds: ~5 min × 3 = ~15 min wall
## What to ship for the F4 N=160 run
**Option A — F3 alone.** Single-knob change from the 2026-05-22 baseline. Expected: ~78-85% trio-strict on full N=160. Cleanest claim ("Waggle's per-model prompt-shape architecture closes 10pp of the Qwen gap").
**Option B — F2+F3 stacked.** Run an N=20 probe first to confirm no compounding regressions, then F4 N=160 + Sonnet N=40 re-baseline. Expected: ~80-87% trio-strict if recoveries union. Most ambitious headline.
**Option C — pick F4 from {F3, F2+F3, F1+F2+F3} by best of three N=20 probes.** ~2h additional compute (~$3-4). Most rigorous selection.
**Recommended:** **Option B.** F3 is the single highest-leverage lever (+10.5pp), F2 adds an independent +5.3pp with mostly non-overlapping recoveries. Stacked is the right "real Waggle harness" headline. The F2+F3 N=20 probe gives confidence before burning the F4 cost ($5-8 + judge).
## Decision rule for F4 trigger
After F2+F3 stacked N=20:
- If ≥80% trio-strict → trigger F4: stacked N=160 + Waggle+Sonnet N=40 matched re-baseline (~$15-25 total) → publishable headline
- If 75-79% → trigger F4 with F3 only (less ambitious, still clean)
- If <75% → root cause the regression vs F3 alone, do not run F4 until understood
## Provenance
- Rebuilt container: `gaia2-waggle:latest` image id `34801a117af7` (F3 final)
- F1 rejudge: `runs/rejudge-waggle-qwen36-f1-gates-n20.jsonl` (14/20)
- F2 rejudge: `runs/rejudge-waggle-qwen36-f2-persona-n20.jsonl` (15/20)
- F3 rejudge: `runs/rejudge-waggle-qwen36-f3-shape-n20.jsonl` (17/20)
- Bare reference: `runs/rejudge-waggle-qwen36-thinking-n160.jsonl` (106/156)
- Worker source: `benchmarks/gaia2/waggle-container/waggle_worker.mjs` HEAD `6db922d` (F2) + F3 patch pending commit
- Container env: `external/.../container_env.py` (working-tree edits pending upstream commit)
## Earlier in-session retractions
- "F1 catastrophic regression" — based on 10% in-container rate; corrected after trio-rejudge showed 70% net-zero
- "F2 persona introduces email-framing bias" — based on N=1 smoke; corrected after N=20 showed +5.3pp lift
- Both retractions surfaced same-session before propagating into the final memo. Documentation discipline: in-container is plumbing, not signal.
---
# §F4 — Authoritative full-scale result (2026-05-28)
Option B was selected from the N=20 ladder: F3-alone (the apparent winner) scaled to Qwen N=160 + a matched Sonnet N=40 re-baseline, both with `WAGGLE_GAIA2_QWEN_SHAPE=1`, all trio-rejudged.
## The numbers
| Cell | N | trio-strict | matched ∆ | judge unanimity |
|---|---:|---:|---|---|
| Bare Waggle+Qwen (2026-05-26) | 156 | 67.9% | baseline | — |
| **F4 Waggle+Qwen+F3** | 157 | **65.6%** (CI 57.9-72.6) | **−3.2pp** vs bare (16 recover / 21 regress) | 155/157 unanimous |
| Hermes+Sonnet (frontier) | 148 | 87.2% | — | — |
| bare Waggle+Sonnet (2026-05-22) | 39 | 84.6% | — | — |
| F4 Waggle+Sonnet+F3 ⚠️ | 40 | 97.5% | +12.8pp vs bare / +10.5pp vs Hermes (0 regress) | 39/40 unanimous |
## What F4 establishes
1. **F3 prompt-shape is a NULL result on Qwen at scale.** 65.6% vs 67.9% bare is statistically indistinguishable (bare sits inside the F4 95% CI). The shape helps simple factoid scenarios and hurts complex multi-step ones — net wash.
2. **The N=20 probe gate was invalid.** Two compounding errors:
- *Prefix-sampling bias.* `limit=N` reads scenarios in dataset order, which is grouped by universe (21→30). N=20 drew 17/20 from universe_21; the Sonnet N=40 drew universes 21-23 only. The full N=160 spans 21-30 with later universes harder. Per-universe proof: bare-Qwen scores 82.4% on universe_21 but 68.0% on universes 23-30.
- *Run-to-run nondeterminism.* Qwen-thinking at temperature produces different outputs per execution. Scenarios F3 "recovered" in the N=20 run regressed in the independent N=160 run. The matched-pair lift was partly a coin-flip the rerun didn't reproduce.
3. **The F3 failure modes at scale** (from the 21 regressions): over-compression (`3`, `Thailand`, `1` — terse but WRONG, the shape truncated correct reasoning into a wrong final token) on complex scenarios, AND non-adherence (2030-char answers still starting "Now I have all the data") where the shape didn't take hold at all. The shape neither reliably compresses nor reliably preserves correctness.
4. **F4 Sonnet+F3 97.5% is UNCONFIRMED, not a result.** It ran on the same biased universe-21-23 prefix (N=40). The Pareto pattern (5 recover / 0 regress, 39/40 unanimous) is striking and *might* be real — Sonnet's self-discipline could compose better with the shape than Qwen-thinking does — but it cannot be claimed without a stratified N≥120 Sonnet+F3 run. **Do not cite 97.5% as a Pillar 1 number.**
## Authoritative Pillar 1 Qwen number — UNCHANGED
The honest sovereign-Qwen harness number remains **67.9% trio-strict (bare Waggle+Qwen 3.6 35B-A3B, N=156)**, ~19-21pp below the Hermes+Sonnet 87.2% frontier. None of F1/F2/F3 moved it at scale:
- F1 (gates): net 0 at N=20, never scaled
- F2 (persona): +5.3pp at N=20, never scaled (and N=20 now known unreliable)
- F3 (shape): +10.5pp at N=20 → **−3.2pp at N=160 (FALSE POSITIVE)**
The Qwen gap to the Sonnet frontier is **model-bound, not harness-bound** — at least, not closeable by any of the three harness levers tried here. The bare-Waggle-on-par-with-Hermes claim (Sonnet, 86.5% vs 89.2%, N=40, 2026-05-22) stands; the sovereign-Qwen lane sits ~20pp lower and the harness levers don't recover it.
## Required follow-up before ANY F3/persona claim
- **Stratified N≥120 probes**, not prefix `limit=N`. Either shuffle the scenario order or sample evenly across universes 21-30. The runner needs a `--shuffle-seed` or stratified-sampling flag (it currently reads in dataset order).
- **pass@k or 3-run majority** to control Qwen-thinking nondeterminism before trusting any matched-pair delta < ~10pp.
- If pursuing the Sonnet+F3 signal: stratified Sonnet+F3 N≥120 vs the same-scenario bare-Sonnet. Only then is 97.5% (or whatever it regresses to) citable.
## F4 provenance
- F4 Qwen: `runs/waggle-qwen36-f4-shape-n160/` + `runs/rejudge-waggle-qwen36-f4-shape-n160.jsonl` (103/157; 3 scenario errors incl. 1 DashScope 429 rate-limit under 4-container parallel load)
- F4 Sonnet: `runs/waggle-sonnet-f4-shape-n40/` + `runs/rejudge-waggle-sonnet-f4-n40.jsonl` (39/40)
- Both ran `WAGGLE_GAIA2_QWEN_SHAPE=1`, image `34801a117af7`, in parallel (Qwen→DashScope-intl, Sonnet→OpenRouter)
## Third in-session retraction (the big one)
- **"F3 closes +10.5pp of the Qwen gap" — RETRACTED.** Held at N=20, failed at N=160 (−3.2pp). Root cause: prefix-sampling bias + run nondeterminism. The earlier two retractions (F1 "regression", F2 "bias") were corrections that turned out *better* than feared; this one is a correction that turned out *worse*. The discipline that matters: the N=20 → scale-up gate was structurally unsound, and the scale-up is what caught it. Always scale-up-to-confirm before claiming a sub-10pp lever.

View File

@@ -1,49 +0,0 @@
# Pillar 1 — Waggle harness vs Hermes reference (GAIA 2 search, N=40)
**Date:** 2026-05-22 · first defensible Waggle-harness number · controlled comparison
## Headline
**Waggle's own agent harness performs on par with the Hermes reference harness on GAIA 2 search.**
| Harness | self-judged | **trio-strict** | 95% CI (trio) |
|---|---:|---:|---:|
| Hermes (reference agent) | 91.9% | **89.2%** | 75–96% |
| **Waggle** (`runAgentLoop`) | 86.5% | **86.5%** | 72–94% |
Self-judged on the full matched 40: Waggle 33/40 = 82.5%, Hermes 35/40 = 87.5%.
Trio-judged on the answerable subset N=37 (3 had no answer/oracle to LLM-judge).
The **2.7pp trio-strict gap has heavily overlapping 95% CIs → not statistically distinguishable** at this N. Waggle's loop is competitive with a SOTA-class reference agent.
## Controlled-variable protocol (only the harness differs)
- **Same model:** Claude Sonnet 4.6 (Waggle via OpenRouter OpenAI-compat; Hermes via Anthropic direct — same model, gateway differs; see caveats).
- **Same single `terminal` tool** (GAIA 2 apps via `gaia2-exec`), **same rendered AGENTS.md**, **same in-container judge**, **same scenarios** (Waggle's 40 are the sorted-order subset of the Hermes N=160).
- **Waggle meta-features OFF** (`skillDistillationGate`/`verificationGate=false`) — Hermes has no such features, so this keeps the task contract identical (see issue #4: skill distillation was replacing the final answer).
## Judge integrity
- **Trio-strict** = all 3 independent judges (Opus 4.7 + Gemini 2.5 Pro + GPT-5) agree PASS, using GAIA 2's own `user_message_checker` (only the judge model varies).
- **Waggle self-judge inflation = +0.0pp** (self 86.5% == trio-strict 86.5%) — even cleaner than Hermes (−2.7pp). Waggle's answers are unambiguously correct when judged.
## Per-scenario (self-judged, matched 40)
- Both PASS: 31 · Waggle-only PASS: 2 (`23_5xzkat`, `23_ans8nx`) · Hermes-only PASS: 4 (`21_bnrehm`, `22_52pwi3`, `22_pepb8u`, `22_x4rb15`).
## Caveats (carried forward)
1. **N=37–40 → CI ≈ ±12pp.** On par, but not powered to resolve a small true gap. Larger N tightens this.
2. **Single-run pass@1.** Run-to-run variance is real (the N=10 probe's 70% was an unlucky sample; `21_1afh09` flipped between runs). pass@k would quench it.
3. **Gateway confound.** Waggle→OpenRouter, Hermes→Anthropic (same model). A single litellm proxy for both removes it.
4. **One split.** Search only. The full matrix adds execution / adaptability / time / ambiguity.
## What this establishes
Pillar 1 is **proven and measurable**: Waggle's harness is a benchmarkable entity in GAIA 2 and lands at reference-harness level. The remaining work is precision (N, pass@k, gateway parity, more splits, + OpenClaw), not feasibility.
## Scale-up to a publishable Tier-1 number
- Full N=160 matched + **pass@k** (k=3) for both harnesses.
- Single litellm proxy (gateway parity).
- Add OpenClaw + Oracle (ceiling); extend to the 5 GAIA 2 splits.
- Budget: ~$90/harness/split at N=160 pass@1 (×k for pass@k) — PM-ratify before the full matrix.
## Artifacts
- Waggle run: `runs/waggle-harness-n40/` · trio re-judge: `runs/rejudge-waggle-n40.jsonl`
- Hermes baseline: `runs/p4-full-hermes-n160/` (matched subset) · trio: `runs/rejudge-search-n160.jsonl`
- Container/worker: `waggle-container/` · judge harness: `rejudge_user_message.py`

View File

@@ -1,65 +0,0 @@
# benchmarks/gaia2 — Gaia2 ARE setup + GEPA dry verification
**Stream:** CC Sesija C (Track D in pre-launch sprint consolidation 2026-04-30).
**Brief:** `D:/Projects/PM-Waggle-OS/briefs/2026-04-30-cc-sesija-C-gaia2-setup-dry-verification.md` (LOCKED).
**Branch base:** `phase-5-deployment-v2` (PM ratification 1B; per LOCKED branch architecture 2026-04-30 §2 — Phase 5 baseline = `gepa-faza-1` `6bc2089`, deployment branch tip `a8283d6`).
**Scope:** ARE platform install + Gaia2 dataset adapter + GEPA-evolved variant dry verification (4 shapes × 10 tasks subset of Search split N=200) + ERL methodology integration plan.
**Cost cap:** $20 hard (retroactive amendment 2026-04-30, was $15) / $8 halt-trigger γ probe (PM ratification γ — first batch as probe).
## STATUS: Phase 3 CLOSED 2026-04-30 — Sesija C STANDBY pending Phase 4 setup
**Re-framed deliverable scope (per probe-validated reality):**
1. ✅ Pipeline integrity verification — adapter contract works end-to-end on real Gaia2 schema (`dry-run-results-memo.md` §5).
2. ✅ Cost reconciliation methodology — paper estimate 9–31× under-shot probe-validated reality; root cause: anchor-then-multiply input-size regime gap (`feedback_anchor_multiply_input_size_regime` memory).
3. ✅ GEPA shape routing OOD verification — Phase 4.5 mechanism activation visible on Gaia2 even on halted runs; arxiv §5.4 evidence (`project_gepa_ood_arxiv_evidence` memory).
4. ✅ Schema fixes documented + committed — JSON-string `data` parse, `apps`-as-array, USER-event extraction, ASCII-only headers.
**Real evaluation deferred to Phase 4 Docker** — narrow-proxy economics non-viable ($163.77 projected for full sweep, 11× cost cap); ARE-native targeted tool calls in Docker bypass the bulk-retrieval driver entirely (~160× input-volume reduction projected).
**NOT done in Sesija C, queued to Phase 4 kickoff or post-launch Phase 3 sprint:**
- Task C5+C6 — full N=40 dry run (Phase 4 pre-flight; 4-invocation probe was the Phase 3 substitute).
- Task C7+C8 — ERL methodology integration plan (`erl-integration-plan.md` not authored).
- Task C9+C10+C11 — cost validation + dry-run results memo + final commit (closure stamped in `dry-run-results-memo.md` §9 instead).
- Phase 4 Docker / WSL / Linux runner host choice + Phase 4 budget allocation.
**Cumulative Sesija C spend:** $16.38 of amended $20 cap. Headroom $3.62.
## Layout
```
benchmarks/gaia2/
├── README.md (this file — directory pointer + closure status)
├── smoke-evidence.md (Phase 2, Task C1+C2 — install + smoke verification + Windows compat)
├── adapter.ts (Phase 3a+3b — narrow-proxy adapter; types + impl)
├── config.yaml (Phase 3b — shapes + cost guards + judge methodology; $15→$20 retroactive)
├── dry-run-results-memo.md (Phase 3b-B + closure stamp §9 — probe results + cost reconciliation)
├── erl-integration-plan.md (NOT YET AUTHORED — punted to Phase 4 kickoff or post-launch Phase 3 sprint)
├── data/ (Phase 3b — Gaia2 task JSONL dumps; gitignored except .gitkeep)
├── runs/ (Phase 3b-B + Phase 4 — per-run JSONL logs; gitignored except .gitkeep)
└── scripts/ (dump-tasks.py + run-dry-verification.ts)
```
## External dependency
ARE platform installed at `D:/Projects/waggle-os/external/meta-agents-research-environments/` (Phase 2, Task C1).
- Repo: https://github.com/facebookresearch/meta-agents-research-environments (MIT).
- **Pinned upstream SHA** (verified via `git rev-parse HEAD` post-clone): `0330191ffef8581e3c0620b78df9c7408bcb98b0` (2026-04-20).
- Dataset: HuggingFace `meta-agents-research-environments/gaia2` (CC-BY-4.0); 800 scenarios across 6 configs (search/execution/adaptability/time/ambiguity/mini, 200 each).
- Install method: `uv sync --frozen` (verified — 95 deps, ~1.5min, exit 0). uv-managed Python 3.10.18.
- Smoke verification: `benchmarks/gaia2/smoke-evidence.md` (Phase 2 deliverable). Windows compat: SIGALRM blocker in scenario timeout enforcement → Phase 4 host choice (Docker/WSL/Linux) deferred to PM.
## Authority chain
- Brief: `briefs/2026-04-30-cc-sesija-C-gaia2-setup-dry-verification.md`.
- §0 evidence: `briefs/2026-04-30-cc-sesija-C-gaia2-setup-evidence.md`.
- Pre-launch sprint LOCKED: `decisions/2026-04-30-pre-launch-sprint-consolidation-LOCKED.md`.
- Branch architecture LOCKED: `decisions/2026-04-30-branch-architecture-opcija-c.md`.
- Benchmark portfolio brief: `briefs/2026-04-29-benchmark-portfolio-refresh-2026-venues.md` (§2.1 Gaia2 + §2.3 ERL).
- Faza 1 closure: `decisions/2026-04-29-gepa-faza1-results.md` (substrate evidence — but note SHA citation `c9bda3d` was Phase 4.7 not Faza 1 terminus per branch-architecture §4.1; real Faza 1 terminus = `6bc2089`).
## Out of scope (post-launch Phase 3 sprint Week 4-8)
- Full N=200 Gaia2 Search + Execution split run.
- ReAct baseline vs ERL-augmented A/B comparison.
- Trio-strict + self-judge dual reporting.
- arxiv submission ka MemAgents Workshop.

View File

@@ -1,45 +0,0 @@
# Waggle harness — first GAIA 2 probe (N=10, PRELIMINARY)
**Date:** 2026-05-22 · Pillar 1 first apples-to-apples · **NOT a conclusion**
## What this is
The first run of **Waggle's own `runAgentLoop`** inside the GAIA 2 rig, vs the Hermes
reference harness, on the **same 10** search-split scenarios (sorted-order subset of the
Hermes N=160). Same model (Sonnet 4.6), same single `terminal` tool, same AGENTS.md,
same in-container judge. Waggle gates OFF (skillDistillation/verification) for fairness —
Hermes has no such meta-features.
## Result (self-judged, matched N=10)
| Harness | PASS | Rate |
|---|---:|---:|
| Hermes (reference) | 9/10 | 90% |
| Waggle | 7/10 | 70% |
Per-scenario: Waggle FAILed `21_1afh09`, `21_5bftlu` (both Hermes PASS); both FAILed `21_eo7tr6`; rest PASS.
## Why this is NOT yet a verdict on harness quality
1. **N=10 → Wilson CI ≈ ±28pp.** 70% vs 90% overlaps massively — not statistically distinguishable.
2. **Run-to-run variance is real.** `21_1afh09` PASSED in the gates-off n1 smoke but FAILED in this
probe under identical config → single-run-per-scenario is noisy at this N. A real comparison needs
larger N and/or pass@k.
3. **Self-judged.** The Hermes headline was validated by an independent trio (JUDGE-DELTA doc). Waggle's
answers must get the same trio re-judge for parity before any comparison is published.
4. **Gateway confound.** Waggle → OpenRouter (Sonnet 4.6, OpenAI-compat); Hermes → Anthropic direct
(Sonnet 4.6). Same model, different gateway. For strict parity, point both at one litellm proxy.
## The value delivered
The **measurement rig works end-to-end** — Waggle's harness is now a benchmarkable entity in GAIA 2.
That was the hard part (native-dep gate, container, socket worker, terminal→gaia2-exec, fairness config).
The digit (70%) is a placeholder until a real run.
## Next for a defensible Pillar-1 number
- Larger N (≥40, ideally the full 160 matched set) + **pass@k** to quench single-run variance.
- **Trio-strict re-judge of Waggle** (reuse `rejudge_user_message.py`) — same protocol as Hermes.
- Optional: single litellm proxy for both harnesses to remove the gateway confound.
- Report Waggle strict + trio-strict alongside Hermes 83.8% / 80.6% under one protocol.
## Artifacts
- Waggle run: `runs/waggle-harness-probe-n10/`
- Hermes baseline: `runs/p4-full-hermes-n160/` (matched subset)
- Container/worker: `waggle-container/` · build: `waggle-container/BUILD.md`

View File

@@ -1,100 +0,0 @@
# Windows Docker runner patches — recovery reference
**Why this file exists:** the gaia2-runner lives under `external/meta-agents-research-environments/`
which is **gitignored**. The 3 patches below make the runner work on Docker Desktop for Windows.
They are plain files on disk (not tracked), so a `docker system prune`, an `external/` reset, or a
fresh `uv sync` would silently wipe them. This file is the tracked source of truth — reapply from here.
**Target files (in the MAIN repo working dir, not this worktree):**
- `D:/Projects/waggle-os/external/meta-agents-research-environments/gaia2-cli/runner/gaia2_runner/launcher.py`
- `D:/Projects/waggle-os/external/meta-agents-research-environments/gaia2-cli/runner/gaia2_runner/runner.py`
All three are no-ops on Linux/macOS (guarded by `platform.system()` or UTF-8-default behavior),
so this is also the basis for a clean upstream PR (Path A in PHASE-4-P4.2-PROGRESS-2026-05-21.md §5).
---
## Patch 1 — drop `--network=host` on Windows (`launcher.py` ~line 410)
Inside `launch_container(...)`, immediately after `container_name = self._container_name(...)`:
```python
# 2026-05-21 Marko patch: drop --network=host on Docker Desktop Windows.
# `host` networking + `-p` port publishing is a broken combo on Docker
# Desktop (host network refers to the Linux VM, not Windows; published
# ports fail to bind). Default bridge network + -p mapping works.
import platform
use_host_network = platform.system() != "Windows" and network == "host"
cmd = [
*self._rt,
"run",
"-d",
f"--name={container_name}",
]
if use_host_network:
cmd.append(f"--network={network}")
elif network != "host":
# Explicit non-host network passed by caller — honor it
cmd.append(f"--network={network}")
# else: Windows + host-default → use Docker's default bridge
```
## Patch 2 — publish adapter port to host (`launcher.py` ~line 444)
In the dynamic-port-allocation block, the `adapter_port` branch must publish the port:
```python
if adapter_port is not None:
cmd.extend(["-e", f"GAIA2_ADAPTER_PORT={adapter_port}"])
cmd.extend(["-p", f"{adapter_port}:{adapter_port}"]) # <-- the added line
```
Without the `-p` line the host-side `_poll_for_response` against `127.0.0.1:8090` fails with
`WinError 10061` because Docker Desktop runs the container in a Linux VM, so "host" loopback
is the VM, not Windows.
## Patch 3 — force UTF-8 on artifact writes (`runner.py` ~line 722)
In the artifact-write block, every `write_text` / `json.dumps` must use UTF-8:
```python
# 2026-05-21 Marko patch: force UTF-8 encoding on all artifact writes.
# Windows default is cp1252 (charmap) which fails on Unicode chars
# like -> OK that appear in agent_response + judge output.
(artifact_dir / "events.jsonl").write_text(events_raw, encoding="utf-8")
(artifact_dir / "agent_response.txt").write_text(agent_response or "", encoding="utf-8")
(artifact_dir / "result.json").write_text(
json.dumps(result, indent=2, default=str, ensure_ascii=False) + "\n",
encoding="utf-8",
)
```
## Patch 4 — UTF-8 in the trace HTML viewer (`trace_viewer.py`) — DISCOVERED 2026-05-22, NOT YET APPLIED
The N=160 finish run surfaced a 4th cp1252 gap: `trace_viewer.py` fails to write the per-scenario
trace HTML when the agent output contains `▸` (U+25B8) and similar:
```
[WARNING] gaia2_runner.trace_viewer: Failed to generate trace for search\scenario_universe_30_uuowj8:
'charmap' codec can't encode character '▸' in position 22339: character maps to <undefined>
```
**Impact:** cosmetic only — affects the per-scenario trace HTML viewer, NOT `result.json` grading
data (Patch 3 already covers the grading artifacts). The N=160 result is unaffected.
**Fix:** find the `write_text(...)` / `open(..., "w")` call(s) in `trace_viewer.py` and add
`encoding="utf-8"`. Bundle with Patches 1–3 in the upstream PR.
---
## Verification after reapply
```powershell
# from gaia2-cli/ with Docker Desktop running:
docker port <container> # should now show 8090/tcp -> 0.0.0.0:8090
# runner status poll should succeed instead of WinError 10061
```
**Provenance:** captured 2026-05-22 from the on-disk patched files (mtime 2026-05-21 19:03/19:11),
which produced the P4.3 N=10 result committed in `b0248b6` (8/10 strict, 8/8 judged-only).

View File

@@ -1,656 +0,0 @@
#!/usr/bin/env tsx
/**
* Sesija C Phase 3b — Gaia2 ARE narrow-proxy adapter (IMPLEMENTATION).
*
* Wraps a Gaia2 task (HuggingFace dataset record) for execution against
* `runRetrievalAgentLoop` from `@waggle/agent`. Selects a prompt shape per
* config (`claude-gen1-v1`, `qwen-thinking-gen1-v1`, baselines).
* Captures per-run telemetry to JSONL.
*
* Phase 3a was types-only (commit b93db4c). Phase 3b adds real function
* bodies. See Phase 3a SCOPE NOTE below for narrow-proxy intent — this
* adapter is NOT a full Gaia2 evaluation; it is cost-projection
* ground-truth + GEPA-variant smoke + type-fit verification.
*
* ─── Reference anchors (unchanged from Phase 3a) ─────────────────────
*
* - Brief: briefs/2026-04-30-cc-sesija-C-gaia2-setup-dry-verification.md §2.2
* - HF dataset: huggingface.co/datasets/meta-agents-research-environments/gaia2 (CC-BY-4.0)
* - ARE pinned SHA: 0330191f (MIT, 2026-04-20)
* - Phase 2 smoke: benchmarks/gaia2/smoke-evidence.md §3+§4
* - GEPA shapes: packages/agent/src/prompt-shapes/gepa-evolved/
* - Retrieval loop: packages/agent/src/retrieval-agent-loop.ts::runRetrievalAgentLoop (line 516)
* - PM ratifications: 1B (branch base), γ (probe), A (narrow-proxy scope)
* - Faza 1 LlmCallFn pattern: benchmarks/gepa/scripts/faza-1/run-gen-1.ts:230
*/
import * as fs from 'node:fs';
import * as path from 'node:path';
import { pathToFileURL } from 'node:url';
import {
runRetrievalAgentLoop,
type AgentRunResult,
type LlmCallFn,
type RetrievalSearchFn,
type MultiStepAgentRunConfig,
} from '../../packages/agent/src/retrieval-agent-loop.js';
import {
registerShape,
REGISTRY,
} from '../../packages/agent/src/prompt-shapes/index.js';
import type { PromptShape } from '../../packages/agent/src/prompt-shapes/types.js';
// ─── Gaia2 dataset record types (HuggingFace schema) ──────────────────
export interface Gaia2HfTask {
readonly id: string;
readonly scenario_id: string;
readonly split: string;
/**
* HF top-level `category` field (mini/standard/agent2agent/noise/...
* routing tag — observed at first probe dump 2026-04-30, NOT
* documented in HF dataset card sample). Treat as informational; not
* consumed by the narrow-proxy adapter.
*/
readonly category?: string;
/**
* NOTE: HF stores `data` on disk as a SERIALIZED JSON STRING (not a
* nested object — HF dataset card sample shows the post-parse form).
* dump-tasks.py parses the string at dump time so on-disk JSONL we
* load here has data as the structured object.
*
* SCHEMA NOTES (verified against first probe dump 2026-04-30,
* `mini/validation/0741_*`):
* - `apps` is an ARRAY of `{app_state, class_name, name}` objects
* (12 apps for `mini` config; HF dataset card showed it as a
* dict for documentation simplicity).
* - `events` is an ARRAY of `{action, class_name, dependencies,
* event_id, event_relative_time, event_time, event_type,
* metadata}` objects. `event_type` ∈ {'USER', 'AGENT', 'ENV', ...}.
* The user instruction lives in events with `event_type==='USER'`,
* `action.function==='send_message_to_agent'`, with the message
* text in `action.args[].value` where `args[].name==='content'`.
* - `metadata.definition` carries scenario-level metadata
* (`duration`, `hints`, `scenario_id`, `start_time`, `tags`) but
* NOT the user-facing instruction. Description extraction routes
* through events. See `extractTaskDescription` impl.
* - `data` may also carry `augmentation` and `version` siblings,
* not consumed by the narrow-proxy adapter.
*/
readonly data: {
readonly metadata: {
readonly definition: Record<string, unknown>;
};
/** Array per real schema; HF card showed dict for doc simplicity. */
readonly apps: readonly Record<string, unknown>[] | Record<string, unknown>;
readonly events: readonly Record<string, unknown>[];
};
}
export interface Gaia2TaskDefinition {
readonly description: string;
readonly capability_tags?: readonly string[];
}
export interface Gaia2AppSnapshot {
readonly class_name: string;
readonly state?: Record<string, unknown>;
}
// ─── Shape selection types ────────────────────────────────────────────
export type ShapeAlias =
| 'claude'
| 'claude-gen1-v1'
| 'claude-gen1-v2'
| 'qwen-thinking'
| 'qwen-thinking-gen1-v1'
| 'qwen-thinking-gen1-v2'
| 'gpt'
| 'gpt-gen1-v1'
| 'gpt-gen1-v2'
| 'qwen-non-thinking'
| 'qwen-non-thinking-gen1-v1'
| 'qwen-non-thinking-gen1-v2'
| 'generic-simple'
| 'generic-simple-gen1-v1'
| 'generic-simple-gen1-v2';
export const BRIEF_DRY_RUN_SHAPES: readonly ShapeAlias[] = [
'claude',
'claude-gen1-v1',
'qwen-thinking',
'qwen-thinking-gen1-v1',
] as const;
// ─── Failure-mode taxonomy ────────────────────────────────────────────
export type FailureMode =
| 'loop_exhausted'
| 'timeout'
| 'parse_fail'
| 'judge_failure'
| 'cost_halt'
| 'llm_error';
// ─── Adapter run record (one row per JSONL output) ────────────────────
export interface Gaia2RunRecord {
readonly task_id: string;
readonly scenario_id: string;
readonly shape: ShapeAlias;
readonly run_number: number;
readonly started_at: string;
readonly ended_at: string;
readonly raw_response: string;
readonly normalized_response: string;
readonly prompt_shape_name: string;
readonly total_tokens_in: number;
readonly total_tokens_out: number;
readonly total_cost_usd: number;
readonly total_latency_ms: number;
readonly steps_taken: number;
readonly retrieval_calls: number;
readonly loop_exhausted: boolean;
readonly pass: boolean | null;
readonly failure_mode: FailureMode | null;
readonly errors: readonly string[];
readonly narrow_proxy_run: boolean;
}
// ─── Adapter config ───────────────────────────────────────────────────
export interface Gaia2AdapterConfig {
readonly task_count_dry_run: number;
readonly shapes: readonly ShapeAlias[];
readonly baseline_shape: ShapeAlias;
readonly judge_methodology:
| 'self-judge-dry-run'
| 'trio-strict'
| 'write-action-verifier'
| 'narrow-proxy-no-judge';
readonly cost_cap_usd: number;
readonly halt_trigger_usd: number;
readonly per_call_halt_usd: number;
readonly probe_invocation_count: number;
readonly output_dir_root: string;
readonly hf_dataset: string;
readonly hf_config: string;
readonly hf_split: string;
}
// ─── Shape → model mapping ────────────────────────────────────────────
// gen1-v1 variants run on the SAME model as their baseline (the prompt
// shape is the manipulated variable, not the model). Aligns with Faza 1
// methodology where shape-evolution effect is measured on a fixed model.
const SHAPE_TO_MODEL: Record<ShapeAlias, string> = {
claude: 'claude-opus-4-7',
'claude-gen1-v1': 'claude-opus-4-7',
'claude-gen1-v2': 'claude-opus-4-7',
'qwen-thinking': 'qwen3.6-35b-a3b',
'qwen-thinking-gen1-v1': 'qwen3.6-35b-a3b',
'qwen-thinking-gen1-v2': 'qwen3.6-35b-a3b',
'qwen-non-thinking': 'qwen3.6-35b-a3b',
'qwen-non-thinking-gen1-v1': 'qwen3.6-35b-a3b',
'qwen-non-thinking-gen1-v2': 'qwen3.6-35b-a3b',
gpt: 'gpt-5.4',
'gpt-gen1-v1': 'gpt-5.4',
'gpt-gen1-v2': 'gpt-5.4',
'generic-simple': 'claude-haiku-4-5',
'generic-simple-gen1-v1': 'claude-haiku-4-5',
'generic-simple-gen1-v2': 'claude-haiku-4-5',
};
// ─── GEPA-evolved shape registration (lazy, idempotent) ───────────────
// Same pattern as benchmarks/gepa/scripts/faza-1/run-gen-1.ts:405-409.
// Shape `name` field is the REGISTRY key (hyphen format). Keep the set
// minimal — we register only the shapes referenced in BRIEF_DRY_RUN_SHAPES.
const GEPA_EVOLVED_FILES: Readonly<Record<ShapeAlias, string>> = Object.freeze({
'claude-gen1-v1': '../../packages/agent/src/prompt-shapes/gepa-evolved/claude-gen1-v1.js',
'claude-gen1-v2': '../../packages/agent/src/prompt-shapes/gepa-evolved/claude-gen1-v2.js',
'qwen-thinking-gen1-v1': '../../packages/agent/src/prompt-shapes/gepa-evolved/qwen-thinking-gen1-v1.js',
'qwen-thinking-gen1-v2': '../../packages/agent/src/prompt-shapes/gepa-evolved/qwen-thinking-gen1-v2.js',
'gpt-gen1-v1': '../../packages/agent/src/prompt-shapes/gepa-evolved/gpt-gen1-v1.js',
'gpt-gen1-v2': '../../packages/agent/src/prompt-shapes/gepa-evolved/gpt-gen1-v2.js',
'qwen-non-thinking-gen1-v1': '../../packages/agent/src/prompt-shapes/gepa-evolved/qwen-non-thinking-gen1-v1.js',
'qwen-non-thinking-gen1-v2': '../../packages/agent/src/prompt-shapes/gepa-evolved/qwen-non-thinking-gen1-v2.js',
'generic-simple-gen1-v1': '../../packages/agent/src/prompt-shapes/gepa-evolved/generic-simple-gen1-v1.js',
'generic-simple-gen1-v2': '../../packages/agent/src/prompt-shapes/gepa-evolved/generic-simple-gen1-v2.js',
// Baselines are pre-registered via prompt-shapes/index.ts; map to empty path.
claude: '',
'qwen-thinking': '',
'qwen-non-thinking': '',
gpt: '',
'generic-simple': '',
});
const registeredShapes = new Set<ShapeAlias>();
export async function ensureShapeRegistered(shape: ShapeAlias): Promise<void> {
if (registeredShapes.has(shape)) return;
if (REGISTRY[shape] !== undefined) {
registeredShapes.add(shape);
return;
}
const filePath = GEPA_EVOLVED_FILES[shape];
if (!filePath) {
throw new Error(
`ensureShapeRegistered: shape "${shape}" not in REGISTRY and not a GEPA-evolved file (baseline shapes are pre-registered via prompt-shapes/index.ts).`,
);
}
// Resolve relative path to absolute file:// URL for ESM dynamic import on Windows.
const adapterDir = path.dirname(new URL(import.meta.url).pathname.replace(/^\//, ''));
const absPath = path.resolve(adapterDir, filePath.replace(/\.js$/, '.ts'));
const mod: Record<string, unknown> = await import(pathToFileURL(absPath).href);
// Find the exported PromptShape (single export per file convention,
// mirrors run-gen-1.ts:215).
const promptShape = Object.values(mod).find(
(v): v is PromptShape =>
typeof v === 'object' &&
v !== null &&
'name' in v &&
'systemPrompt' in v &&
'soloUserPrompt' in v,
);
if (!promptShape) {
throw new Error(`ensureShapeRegistered: no PromptShape export in ${absPath}`);
}
registerShape(promptShape.name, promptShape);
registeredShapes.add(shape);
}
// ─── Loaders ──────────────────────────────────────────────────────────
export function loadGaia2TasksFromJsonl(
filepath: string,
limit: number,
): readonly Gaia2HfTask[] {
if (!fs.existsSync(filepath)) {
throw new Error(
`loadGaia2TasksFromJsonl: file not found: ${filepath}\n` +
`Run benchmarks/gaia2/scripts/dump-tasks.py first to produce this file.`,
);
}
const raw = fs.readFileSync(filepath, 'utf-8');
const lines = raw.split(/\r?\n/).filter((l) => l.trim().length > 0);
const tasks: Gaia2HfTask[] = [];
for (let i = 0; i < Math.min(limit, lines.length); i++) {
try {
tasks.push(JSON.parse(lines[i]) as Gaia2HfTask);
} catch (err: unknown) {
const msg = err instanceof Error ? err.message : String(err);
throw new Error(`loadGaia2TasksFromJsonl: parse error on line ${i + 1}: ${msg}`);
}
}
return tasks;
}
// ─── Task → corpus + question extraction ──────────────────────────────
export function flattenAppStateToCorpus(
task: Gaia2HfTask,
): readonly { id: string; content: string }[] {
const docs: { id: string; content: string }[] = [];
// `apps` is an ARRAY of {app_state, class_name, name} per real schema
// (HF card sample showed dict — doc simplification). Handle both for
// forward-compat.
const appsList: readonly Record<string, unknown>[] = Array.isArray(task.data.apps)
? (task.data.apps as readonly Record<string, unknown>[])
: (Object.values(task.data.apps) as readonly Record<string, unknown>[]);
for (const app of appsList) {
const className = typeof app.class_name === 'string' ? app.class_name : 'UnknownApp';
const name = typeof app.name === 'string' ? app.name : className;
// Real format uses `app_state` key for the app's initial-state JSON;
// HF card sample showed `state`. Try both.
const state = app.app_state ?? app.state ?? app;
let stateJson: string;
try {
stateJson = JSON.stringify(state, null, 2);
} catch {
stateJson = '(unserializable app state)';
}
docs.push({
id: `app:${name}`,
content: `[${className}] ${name}\n${stateJson}`,
});
}
// Task definition itself goes into the corpus so the agent can
// retrieve task-related metadata (tags, hints, duration).
let defJson: string;
try {
defJson = JSON.stringify(task.data.metadata.definition, null, 2);
} catch {
defJson = '(unserializable task definition)';
}
docs.push({
id: 'task:definition',
content: `[Task Definition]\n${defJson}`,
});
return docs;
}
/**
* Extracts the user instruction from a Gaia2 task.
*
* Real schema: instruction lives in events with `event_type === 'USER'`,
* `action.function === 'send_message_to_agent'`, with the text in
* `action.args[]` where `args[i].name === 'content'`. A scenario can have
* multiple USER events injected over time (multi-turn). For the narrow
* proxy we concatenate them into one prompt block — the agent doesn't
* see the time evolution but does see the cumulative ask.
*
* Falls back through:
* 1. definition.description (HF card schema; not present in mini/validation
* first probe but may be present in other configs)
* 2. definition.instruction / definition.task (defensive)
* 3. concatenated user-message events (real schema, primary path)
* 4. stringified definition + tags (last-resort, never empty question)
*/
export function extractTaskDescription(task: Gaia2HfTask): string {
const def = task.data.metadata.definition as Record<string, unknown>;
// Strategy 1: HF card sample fields (defensive).
if (typeof def.description === 'string' && def.description.trim().length > 0) {
return def.description;
}
if (typeof def.instruction === 'string' && def.instruction.trim().length > 0) {
return def.instruction;
}
if (typeof def.task === 'string' && def.task.trim().length > 0) {
return def.task;
}
// Strategy 2: extract from USER events (primary path for real schema).
const events = task.data.events;
const userMessages: string[] = [];
for (const ev of events) {
if (ev.event_type !== 'USER') continue;
const action = ev.action as Record<string, unknown> | undefined;
if (!action) continue;
const args = action.args;
if (Array.isArray(args)) {
for (const arg of args) {
const argObj = arg as Record<string, unknown>;
const name = argObj.name;
const value = argObj.value;
if (
typeof value === 'string' &&
(name === 'content' || name === 'message' || name === 'text')
) {
userMessages.push(value);
}
}
} else if (typeof args === 'object' && args !== null) {
const argsObj = args as Record<string, unknown>;
const candidate = argsObj.content ?? argsObj.message ?? argsObj.text;
if (typeof candidate === 'string' && candidate.trim().length > 0) {
userMessages.push(candidate);
}
}
}
if (userMessages.length > 0) {
return userMessages.length === 1
? userMessages[0]
: userMessages
.map((m, i) => `--- User message ${i + 1}/${userMessages.length} ---\n${m}`)
.join('\n\n');
}
// Strategy 3: fallback (never empty question).
const tags = Array.isArray(def.tags) ? (def.tags as readonly string[]).join(', ') : '';
return `Gaia2 scenario ${task.id} (tags: ${tags || 'none'}). Definition: ${JSON.stringify(def).slice(0, 500)}`;
}
// ─── Simple in-memory FTS RetrievalSearchFn ──────────────────────────
// Substring-match scoring is intentionally minimal — the narrow-proxy
// adapter is NOT trying to optimize retrieval quality. It's verifying
// the adapter contract end-to-end. Phase 4 Docker run uses ARE-native
// app interactions, no synthetic search.
export function buildSimpleSearch(
corpus: readonly { id: string; content: string }[],
): RetrievalSearchFn {
const lowercased = corpus.map((doc) => ({
id: doc.id,
content: doc.content,
lower: doc.content.toLowerCase(),
}));
return async (input) => {
const tokens = input.query
.toLowerCase()
.split(/\s+/)
.filter((t) => t.length > 2);
const scored = lowercased.map((doc) => {
let score = 0;
for (const tok of tokens) {
if (doc.lower.includes(tok)) score += 1;
}
return { id: doc.id, content: doc.content, score };
});
scored.sort((a, b) => b.score - a.score);
const top = scored.slice(0, input.limit).filter((d) => d.score > 0);
if (top.length === 0) {
return { formattedResults: '(no results matched the query in the available app state)', resultCount: 0 };
}
const formatted = top.map((d) => `## ${d.id}\n${d.content}`).join('\n\n---\n\n');
return { formattedResults: formatted, resultCount: top.length };
};
}
// ─── Failure-mode classifier (post-hoc, from AgentRunResult) ─────────
function classifyFailure(
result: AgentRunResult | null,
errMessage: string | null,
): FailureMode | null {
if (!result && !errMessage) return null;
if (errMessage) {
const lower = errMessage.toLowerCase();
if (lower.includes('cost') || lower.includes('halt')) return 'cost_halt';
if (lower.includes('parse')) return 'parse_fail';
if (lower.includes('judge')) return 'judge_failure';
if (lower.includes('timeout')) return 'timeout';
return 'llm_error';
}
if (result?.loopExhausted) return 'loop_exhausted';
return null;
}
// ─── Main: run one task with one shape ────────────────────────────────
export async function runGaia2TaskWithShape(
task: Gaia2HfTask,
shape: ShapeAlias,
config: Gaia2AdapterConfig,
llmCall: LlmCallFn,
runNumber: number = 1,
): Promise<Gaia2RunRecord> {
await ensureShapeRegistered(shape);
const startedAt = new Date().toISOString();
const startMs = Date.now();
const question = extractTaskDescription(task);
const corpus = flattenAppStateToCorpus(task);
const search = buildSimpleSearch(corpus);
const model = SHAPE_TO_MODEL[shape];
let result: AgentRunResult | null = null;
let errMessage: string | null = null;
const runConfig: MultiStepAgentRunConfig = {
modelAlias: model,
persona:
'You are an AI agent operating within a simulated environment with multiple applications. ' +
'Use the search tool to retrieve relevant app state, then answer the user task. ' +
'Be concise and decisive.',
question,
llmCall,
search,
promptShapeOverride: shape,
perCallHaltUsd: config.per_call_halt_usd,
perCellHaltUsd: config.per_call_halt_usd * 5,
maxSteps: 5,
maxRetrievalsPerStep: 8,
contextTag: `gaia2-narrow-proxy:${task.id}:${shape}:run${runNumber}`,
};
try {
result = await runRetrievalAgentLoop(runConfig);
} catch (err: unknown) {
errMessage = err instanceof Error ? err.message : String(err);
}
const endedAt = new Date().toISOString();
const errors: string[] = [];
if (errMessage) errors.push(errMessage);
if (result?.errors) errors.push(...result.errors);
return {
task_id: task.id,
scenario_id: task.scenario_id,
shape,
run_number: runNumber,
started_at: startedAt,
ended_at: endedAt,
raw_response: result?.rawResponse ?? '',
normalized_response: result?.normalizedResponse ?? '',
prompt_shape_name: result?.promptShapeName ?? shape,
total_tokens_in: result?.totalTokensIn ?? 0,
total_tokens_out: result?.totalTokensOut ?? 0,
total_cost_usd: result?.totalCostUsd ?? 0,
total_latency_ms: result?.totalLatencyMs ?? Date.now() - startMs,
steps_taken: result?.stepsTaken ?? 0,
retrieval_calls: result?.retrievalCalls ?? 0,
loop_exhausted: result?.loopExhausted ?? false,
pass: null, // Phase 4 Docker + ARE write-action verifier wires this.
failure_mode: classifyFailure(result, errMessage),
errors,
narrow_proxy_run: true,
};
}
// ─── Driver: γ probe-first sweep ──────────────────────────────────────
export interface DryRunSweepResult {
readonly probe_records: readonly Gaia2RunRecord[];
readonly probe_cost_usd: number;
readonly projected_total_usd: number;
readonly halt_triggered: boolean;
readonly full_sweep_records: readonly Gaia2RunRecord[];
readonly final_total_cost_usd: number;
}
export async function runDryRunSweep(
config: Gaia2AdapterConfig,
llmCall: LlmCallFn,
options: {
tasksFile: string;
/** If true, halt after probe regardless of projection (for Phase 3b sample). */
haltAfterProbe?: boolean;
},
): Promise<DryRunSweepResult> {
const tasks = loadGaia2TasksFromJsonl(options.tasksFile, config.task_count_dry_run);
if (tasks.length === 0) {
throw new Error(`runDryRunSweep: no tasks loaded from ${options.tasksFile}`);
}
// Probe: round-robin invocations across (task, shape) pairs until
// probe_invocation_count is reached.
const probe_records: Gaia2RunRecord[] = [];
let probe_cost_usd = 0;
let invocationsRun = 0;
outer: for (const task of tasks) {
for (const shape of config.shapes) {
if (invocationsRun >= config.probe_invocation_count) break outer;
const rec = await runGaia2TaskWithShape(task, shape, config, llmCall, 1);
probe_records.push(rec);
probe_cost_usd += rec.total_cost_usd;
invocationsRun++;
}
}
// Project full sweep cost from probe.
const totalInvocations = config.shapes.length * config.task_count_dry_run;
const avgCostPerInvocation =
probe_records.length > 0 ? probe_cost_usd / probe_records.length : 0;
const projected_total_usd = avgCostPerInvocation * totalInvocations;
const halt_triggered =
options.haltAfterProbe === true || projected_total_usd > config.halt_trigger_usd;
if (halt_triggered) {
return {
probe_records,
probe_cost_usd,
projected_total_usd,
halt_triggered: true,
full_sweep_records: probe_records,
final_total_cost_usd: probe_cost_usd,
};
}
// Continue full sweep. Skip the (task, shape) pairs already covered
// by the probe.
const fullRecords: Gaia2RunRecord[] = [...probe_records];
let totalCost = probe_cost_usd;
const probeKeys = new Set(probe_records.map((r) => `${r.task_id}::${r.shape}`));
for (const task of tasks) {
for (const shape of config.shapes) {
const key = `${task.id}::${shape}`;
if (probeKeys.has(key)) continue;
// Cost-cap check before each invocation (defensive — per-call halt
// also enforced via runConfig).
if (totalCost >= config.cost_cap_usd) {
return {
probe_records,
probe_cost_usd,
projected_total_usd,
halt_triggered: true,
full_sweep_records: fullRecords,
final_total_cost_usd: totalCost,
};
}
const rec = await runGaia2TaskWithShape(task, shape, config, llmCall, 1);
fullRecords.push(rec);
totalCost += rec.total_cost_usd;
}
}
return {
probe_records,
probe_cost_usd,
projected_total_usd,
halt_triggered: false,
full_sweep_records: fullRecords,
final_total_cost_usd: totalCost,
};
}
// ─── JSONL writer ─────────────────────────────────────────────────────
export function writeRecordsToJsonl(
records: readonly Gaia2RunRecord[],
outputPath: string,
): void {
fs.mkdirSync(path.dirname(outputPath), { recursive: true });
const lines = records.map((r) => JSON.stringify(r)).join('\n');
fs.writeFileSync(outputPath, lines + (lines.length > 0 ? '\n' : ''), 'utf-8');
}
// ─── Re-exports ───────────────────────────────────────────────────────
export type { AgentRunResult, LlmCallFn, RetrievalSearchFn, MultiStepAgentRunConfig };

View File

@@ -1,70 +0,0 @@
# Sesija C Phase 3b — Gaia2 ARE dry verification config (Task C4).
#
# LOCKED FOR:
# - Brief: briefs/2026-04-30-cc-sesija-C-gaia2-setup-dry-verification.md §2.2 Task C4
# - PM ratification γ (first-batch-as-probe; halt at $8)
# - PM ratification A (narrow-proxy scope — pass=null until Phase 4 Docker + ARE verifier)
# - Cost cap unchanged: $15 hard / $10 expected halt / $8 probe-driven halt-and-PM
#
# Adapter consumer: benchmarks/gaia2/adapter.ts → Gaia2AdapterConfig
# Driver: benchmarks/gaia2/scripts/run-dry-verification.ts
# Tasks source: benchmarks/gaia2/data/tasks-<config>-<limit>.jsonl
# (produced by benchmarks/gaia2/scripts/dump-tasks.py)
# ─── Dry-run scope ───────────────────────────────────────────────────────
task_count_dry_run: 10 # Per shape; 4 shapes × 10 = 40 total invocations
# 4-shape sweep per brief Task C5. baseline + GEPA-evolved for both
# claude + qwen-thinking shape families. Faza 1 §B AUTHORIZED:
# claude-gen1-v1 + qwen-thinking-gen1-v1. Baselines provide control.
shapes:
- claude # baseline 1 (claude::base)
- claude-gen1-v1 # GEPA-evolved variant 1 (Faza 1 AUTHORIZED)
- qwen-thinking # baseline 2 (qwen-thinking::base)
- qwen-thinking-gen1-v1 # GEPA-evolved variant 2 (Faza 1 AUTHORIZED + Phase 4.5 mechanism CONFIRMED)
# Control shape for relative measurement.
baseline_shape: claude
# Phase 3 narrow-proxy default (per PM ratification A): no judge, pass=null.
# Phase 4 Docker + ARE verifier flips to write-action-verifier.
# Post-launch Phase 3 sprint Week 7 flips to trio-strict.
judge_methodology: narrow-proxy-no-judge
# ─── Cost guards ─────────────────────────────────────────────────────────
# All caps in USD. Cumulative across Sesija C from $0 starting point.
#
# RETROACTIVE AMENDMENT 2026-04-30 (PM Phase 3 closure ratification):
# Hard cap raised $15 → $20 to accommodate probe overshoot ($16.38 actual
# vs $15 cap = $1.38 overshoot) plus closure memo writing finalization.
# Probe-first protocol γ ROI: $147 saved vs full-sweep blind execution
# ($163.77 projected). Cost amendment is acceptable per probe-first
# discipline (cheap discovery vs expensive blind execution).
# See `feedback_probe_first_roi_demonstration.md` memory entry.
cost_cap_usd: 20 # Hard cap — retroactive amendment 2026-04-30 ($15 → $20)
halt_trigger_usd: 8 # Halt-and-PM trigger (PM ratification γ — unchanged)
per_call_halt_usd: 0.50 # Per-LLM-call ceiling (single-call abort — unchanged)
# PM ratification γ first-batch-as-probe: run this many invocations
# first, project total, halt-and-PM if projection > halt_trigger_usd.
# Default 4 (1 task × 4 shapes) — minimum viable probe sample.
# Phase 3b-B sample (this commit): 2 (1 task × 2 shapes) for schema-fit
# verification only.
probe_invocation_count: 4
# ─── Output ──────────────────────────────────────────────────────────────
output_dir_root: benchmarks/gaia2/runs
# ─── HuggingFace dataset config ──────────────────────────────────────────
# Dataset card: https://huggingface.co/datasets/meta-agents-research-environments/gaia2
# Pinned via task-dump JSONL (see benchmarks/gaia2/data/), not via HF
# revision pin — adapter consumes the dump file, dump script consumes HF.
# Phase 4 Docker run will use ARE's native HF loader (pinned revision via
# ARE config).
hf_dataset: meta-agents-research-environments/gaia2
hf_config: mini # 200 representative scenarios
hf_split: validation # 160 examples in mini/validation

View File

@@ -1,212 +0,0 @@
# Phase 3b-B Probe Results — Cost Reconciliation Memo
**Stream:** CC Sesija C — Gaia2 ARE narrow-proxy adapter
**Brief:** `briefs/2026-04-30-cc-sesija-C-gaia2-setup-dry-verification.md`
**Phase:** 3b-B-2 (sample probe execution, PM ratification γ first-batch-as-probe)
**Date:** 2026-04-30
**Branch:** `feature/gaia2-are-setup` @ `144b242` (post Phase 3b-B-1 driver patch)
**Status:** **PROBE GATE FAIL — halt-and-PM (cost cap exceeded; adapter design must change before Phase 4)**
---
## §1 — TL;DR
| Metric | Paper estimate (§0.3) | Probe-validated actual | Multiple |
|---|---|---|---|
| Per-invocation avg | $0.13–0.45 | **$4.09** | **9–31×** |
| 4-invocation probe total | $0.52–1.80 | **$16.38** | **9–31×** |
| Halt trigger | $8 (γ ratification) | **fired** | n/a |
| Hard cap | $15 | **breached at $16.38** | n/a |
| Projected full sweep ($N=40$) | $5.20–18.00 | **$163.77** | **9–31×** |
The narrow-proxy approach (extract user instruction + flatten ALL app state + retrieve via simple FTS) is **economically non-viable** on Gaia2 mini scenarios. The §0.3 paper estimate was anchored on Faza 1 LoCoMo per-task token sizes (~$0.13/eval); Gaia2 task corpora are roughly **100× larger** per scenario.
Probe halted correctly per PM ratification γ. The cost reality is the legitimate Phase 3 deliverable; the next PM decision is how to proceed for Phase 4 (Docker + ARE) and post-launch Phase 3 sprint.
---
## §2 — Per-invocation breakdown
| # | Shape | Provider call | Tokens in | Tokens out | Cost | Failure mode |
|---|---|---|---|---|---|---|
| 1 | `claude` | `anthropic/claude-opus-4.7` | 1,629,091 | 1,533 | **$8.1838** | `loop_exhausted` (per-call halt $4.07 > $0.50 fired step 2) |
| 2 | `claude-gen1-v1` | `anthropic/claude-opus-4.7` | 1,630,522 | 1,547 | **$8.1913** | `loop_exhausted` (same as #1) |
| 3 | `qwen-thinking` | `qwen/qwen3-30b-a3b-thinking-2507` | 283 | 313 | $0.0005 | `loop_exhausted` (provider rejected step 2: "262144 tokens max, requested 553,378") |
| 4 | `qwen-thinking-gen1-v1` | `qwen/qwen3-30b-a3b-thinking-2507` | 458 | 768 | $0.0013 | `loop_exhausted` (provider rejected step 2: "262144 tokens max, requested 553,669") |
**Total probe cost:** $16.3768. **Wall-clock:** ~2:08 (4 invocations).
### Diagnostic
Both Claude invocations:
- Step 1 succeeded (formatted prompt + retrieval hit) at ~$4.07 each, around 800K input tokens.
- Step 2 prepared (full corpus injected as retrieved context + agent's accumulated working state) reached ~1.6M input tokens cumulative; per-call cost crossed $0.50 halt threshold at $4.07 → loop aborted.
- Output tokens 1.5K (model produced a partial response before halt).
- Cost basis: 1.6M × $15/M in + 1.5K × $75/M out = $24 + $0.11 = $24.11 over 2 calls = $8.18 ÷ 2 = $4.09 per call.
Both Qwen invocations:
- Step 1 succeeded (small token count — Qwen prompt-shape is more concise).
- Step 2 prep injected the full retrieved context, ballooning to 553K input tokens.
- OpenRouter Qwen route enforces a hard 262,144-token context cap. Provider rejected the request server-side. Tokens-in remained low (only the summed *step 1* numbers stuck), cost essentially $0.
- Faza 1 used DashScope direct routing for Qwen which appears to have a higher context cap; OpenRouter route cannot match that envelope.
### Root cause
The narrow-proxy adapter strategy `flattenAppStateToCorpus` dumps **all 12 simulated apps** + full task definition into the searchable corpus. With 12 apps × ~50KB each, the raw corpus is ~600KB. After RetrievalSearchFn runs `simple-substring` matching, the agent receives `top-K=8` matches with full content — easily 200KB injected per turn × 5 max-steps = potential 1MB+ per scenario. Plus accumulated_context audit log layers.
This is the empirical confirmation of the **semantic gap** flagged in Phase 3a SCOPE NOTE: Gaia2 is multi-app tool-use simulation; `runRetrievalAgentLoop` is search-then-finalize. Force-fitting the latter onto the former produces an adapter that retrieves bulk context instead of making targeted tool calls — and the cost difference is exactly the inefficiency you'd predict.
---
## §3 — Discrepancy with §0.3 paper estimate
### What the §0.3 estimate assumed
Phase 2 §0.3 paper estimate (`benchmarks/gaia2/smoke-evidence.md` §0.3 reconstruction):
- Anchored on Faza 1 cost evidence: 135 evals / $43.49 → $0.32/eval avg.
- Applied 2–4× premium for Gaia2 vs LoCoMo (12 apps + 101 tools system overhead + multi-step async).
- Mid-estimate: 40 invocations × $0.25 = $10. Pessimistic: $18.
### What the probe revealed
The 2–4× premium was an under-estimate by an order of magnitude. The actual per-invocation token volume is dominated by **app state corpus injection**, not by system prompt overhead. Specifically:
| Component | LoCoMo per-task (Faza 1) | Gaia2 per-task (probe-validated) |
|---|---|---|
| User question | ~50 tokens | ~150 tokens (multi-line user instruction) |
| Retrieved context | ~3–5K tokens (one conversation) | ~150–500K tokens (12 apps × full state) |
| Agent system prompt | ~300 tokens | ~300 tokens (shape-dependent) |
| **Per-invocation total input** | ~5–10K | ~800K–1.6M |
So the cost-per-invocation ratio is roughly 100×–200× higher, not 2–4×.
### Why the §0.3 estimate methodology was right but result was wrong
Anchoring on Faza 1 cost-per-eval is sound research practice — it's the closest known empirical anchor. The miss was that the LoCoMo conversation length (~3K tokens of context) is in a fundamentally different regime than the Gaia2 environment snapshot (~600K). The estimate didn't break the methodology; it broke the implicit assumption that "Gaia2 scenarios" and "LoCoMo conversations" have comparable per-task input sizes. They don't.
**This is a useful update for the post-launch Phase 3 sprint cost projection** — Phase 3 sprint Week 6 N=200 dry run on full Gaia2 Search split (200 scenarios) at $4-8/invocation × 200 × 4 shapes = $3,200-6,400 in narrow-proxy mode. Full evaluation in ARE-native runtime (Docker, targeted tool calls, NO bulk corpus injection) should be much lower — that's the rationale for moving to Docker for real Phase 4 work.
---
## §4 — Halt-trigger γ behavior — correct
PM ratification γ specified: probe first, halt if projection > halt_trigger ($8). The probe behaved correctly:
```json
{
"probe_invocation_count": 4,
"probe_cost_usd": 16.38,
"projected_total_usd": 163.77,
"halt_triggered": true
}
```
`halt_triggered: true` because either (a) `projected_total_usd > halt_trigger_usd` ($163 > $8) — **YES** OR (b) `--halt-after-probe` CLI flag was set — also yes for this probe. The implementation is defensive: probes always halt for PM review when explicitly invoked with `--halt-after-probe`, AND auto-halt on projection breach.
The cost-cap soft-fence ($15 hard) was breached BY the probe ($16.38) — i.e., the 4-invocation probe alone exceeded the hard cap. This means a probe-first approach with this adapter design cannot operate within the brief's cost envelope. **This is a useful finding, not a failure mode.**
---
## §5 — Schema-fit verification — PASS (apart from cost)
The 4 probe invocations confirmed the adapter pipeline works end-to-end:
| Pipeline component | Verdict | Evidence |
|---|---|---|
| `loadGaia2TasksFromJsonl` parsing | **PASS** | All 2 tasks deserialized cleanly |
| `Gaia2HfTask` schema (post-fix) | **PASS** | Adapter v2 handles `apps` as array + `data` as object after dump-tasks.py JSON parse |
| `extractTaskDescription` from USER events | **PASS** | Real instruction text extracted ("I need to move out, but my budget is tight at the moment...") visible in Claude's partial response |
| `flattenAppStateToCorpus` for array-shaped apps | **PASS** | Apps + class_name + state json flattened into search docs |
| `buildSimpleSearch` substring FTS | **PASS** | At least 1 retrieval call recorded per invocation |
| `ensureShapeRegistered` lazy GEPA loading | **PASS** | `claude-gen1-v1` + `qwen-thinking-gen1-v1` shapes registered + executed (visible in claude-gen1-v1 producing different response style than baseline `claude`) |
| `runRetrievalAgentLoop` invocation | **PASS** | 4/4 invocations reached step 2 |
| `Gaia2RunRecord` JSONL output | **PASS** | All 4 records well-formed |
| Cost-tracking PRICE_TABLE fallback | **PASS** | Both Claude invocations produced wire-accurate cost via Faza 1 prices |
| Failure-mode classification | **PASS** | All 4 marked `loop_exhausted`, errors captured |
**Type-fit and pipeline integrity are validated.** The adapter is correct. The economics are wrong.
---
## §6 — PM decision options
### Option A — Adapter redesign: selective corpus extraction
Modify `flattenAppStateToCorpus` to filter app state by relevance to the user instruction. E.g., for the apartment task, prioritize `RentAFlat` + `Messages` + `Contacts` apps, drop `SandboxLocalFileSystem` + 9 others. Reduces corpus from 600KB → ~50KB, cost from $4 → $0.30 per invocation.
- **Pro:** Stays within narrow-proxy paradigm; ~10× cost reduction; can finish Phase 3b in this session.
- **Con:** Requires app-relevance heuristic (LLM-based pre-filter? Tag-based? Manual mapping?). Adds adapter complexity. Still doesn't match Gaia2 semantics (multi-step tool calls).
### Option B — Defer real evaluation entirely to Phase 4 Docker
Accept that narrow-proxy is too expensive for any meaningful Gaia2 work. Phase 3 deliverable shrinks to "adapter pipeline integrity verified, cost economics surfaced". All real GEPA-variant verification moves to Phase 4 Docker (where ARE-native runtime makes targeted tool calls instead of bulk retrieval).
- **Pro:** Honest scope. Saves ~$15-50 of additional probe-tweaking spend. Phase 4 Docker is the correct architectural target anyway.
- **Con:** No GEPA-variant signal from Phase 3. Brief expectation of "GEPA-variant smoke" not met.
### Option C — Tiny-task subset + Qwen-only on DashScope direct
Probe with 1 task on a much smaller config (e.g., search split smallest scenario; or filter to scenarios with ≤3 apps). Use Qwen via DashScope direct (Faza 1 had this configured) to avoid OpenRouter's 262K cap. Smaller task corpus → fits in budget.
- **Pro:** Salvages partial probe data; cheaper.
- **Con:** Requires DashScope env-var setup (`DASHSCOPE_API_KEY` if rotated since Faza 1) AND task pre-filtering logic. Risks selection bias from cherry-picking scenarios.
### Option D — Cost amendment for Phase 3 + continue with current adapter
Raise Sesija C cost cap from $15 → $50 for Phase 3 only (Phase 4 + post-launch budgets stay separate). Accept $4-8 per invocation. Re-run with smaller `task_count_dry_run` (e.g., 5 instead of 10) → 4 shapes × 5 = 20 invocations × $4 avg = $80. Still over $50 raise.
- **Pro:** Stays with planned methodology.
- **Con:** Cost discipline degraded; sets bad precedent. Not proportional to information value.
### CC recommendation: **Option B (defer to Phase 4 Docker)**
The probe already gave us the most valuable Phase 3 deliverable: a probe-validated cost reality for narrow-proxy on Gaia2. Optimization investments (Option A) would chase narrow-proxy improvements that ARE-native (Docker) bypasses entirely via targeted tool calls. The strategic move is accepting the finding, freezing the adapter as documented, and routing all real evaluation through Phase 4. Phase 3b-B closes with this memo + committed probe outputs.
---
## §7 — Phase 3 close-out signals (if Option B accepted)
- **Cost reality (vs §0.3 paper estimate):** documented (10–30× higher than estimated).
- **GEPA-variant smoke:** PARTIAL — Claude shapes both ran but neither produced a clean evaluation output (loop halted at step 2). Qwen shapes blocked by provider context cap. Visible difference between `claude` (formal-tone partial response) and `claude-gen1-v1` (more analytical-tone partial response with markdown structure) suggests the GEPA-evolved prompt is reaching the model and influencing output style — even on a halted run, the shape-routing pipeline works.
- **Type-fit verification:** PASS — pipeline integrity confirmed across 4 invocations.
- **Phase 4 Docker setup decision input:** Docker remains the correct host for full evaluation. Linux/Docker eliminates Windows SIGALRM blocker (Phase 2) AND solves the OpenRouter context-cap bottleneck (DashScope direct or local model serves longer contexts) AND uses ARE-native targeted tool calls (eliminates bulk-retrieval cost driver).
- **Post-launch Phase 3 sprint Week 4-8 budget input:** N=200 full Gaia2 Search split in narrow-proxy mode would cost ~$3K-6K. In ARE-native Docker mode the budget collapses to the brief's $25-40 estimate. Strongly supports Phase 4 Docker as the right move for the sprint.
---
## §8 — Audit anchors
- Probe output dir: `benchmarks/gaia2/runs/dry-verification-2026-04-29T21-02-52-243Z/` (gitignored; reproducible from `run-dry-verification.ts --tasks data/tasks-mini-2.jsonl --halt-after-probe` + `OPENROUTER_API_KEY` env)
- Probe summary: `summary.json` (committed via this memo's data tables above)
- Probe records: `probe.jsonl` (4 lines, JSONL of `Gaia2RunRecord`)
- Tasks dump: `benchmarks/gaia2/data/tasks-mini-2.jsonl` (gitignored; SHA: re-derivable from dump-tasks.py + HF dataset revision)
- Driver SHA: `144b242` (Phase 3b-B-1 commit) + post-fix em-dash header + post-fix data-string parsing in dump-tasks.py + post-fix apps-as-array handling in adapter.ts
- Prior anchors: `benchmarks/gaia2/smoke-evidence.md`, `benchmarks/gaia2/README.md`
---
---
## §9 — PM RATIFICATION STAMP — Phase 3 closure (2026-04-30)
**Decision:** **Option B ratified** + retroactive cost amendment $15 → $20.
**Phase 3 closure verdict:** **COMPLETE**.
**Phase 3 re-framed deliverable scope (post probe-validated reality):**
1. **Pipeline integrity verification** — PASS (adapter contract works end-to-end on real Gaia2 schema; USER-event instruction extraction + apps-as-array handling + GEPA shape routing + cost-tracking PRICE_TABLE fallback all confirmed in 4 live invocations).
2. **Cost reconciliation methodology** — PASS (anchor-then-multiply methodology gap exposed; probe-first protocol γ saved $147 vs blind full-sweep execution).
3. **GEPA shape routing out-of-distribution verification** — PASS (visible behavioral difference between `claude` baseline and `claude-gen1-v1` on Gaia2 task confirms Phase 4.5 mechanism activation outside Faza 1's LoCoMo training distribution; arxiv §5.4 evidence).
4. **Schema fixes documented + committed** — `data` JSON-string parse, `apps`-as-array handling, USER-event extraction strategy ladder, ASCII-only HTTP headers (em-dash byte-string fix). All four are reusable Phase 4 setup artifacts.
**Real evaluation (full N=200 Gaia2 Search + Execution split):** **deferred to Phase 4 Docker** (per benchmark portfolio brief §5 Week 4–8). ARE-native targeted tool calls bypass the bulk-retrieval cost driver entirely (160× input volume reduction projected from selective `app.api(...)` invocations vs full `app.initial_state` corpus injection).
**Sesija C status:** **STANDBY**. Phase 4 setup is separate decision (Docker / WSL / CI runner host choice + Phase 4 budget allocation + ERL methodology integration plan authoring per Task C7+C8 — all queued to Phase 4 kickoff).
**Cumulative Sesija C spend:** $16.38 of amended $20 cap. Headroom $3.62 retained for any closure-stage micro-spend.
**Refused options for the audit trail:**
- A (narrow-proxy heuristic) — investment in wrong abstraction; throwaway before Phase 4.
- C (DashScope-direct Qwen tiny subset) — selection bias risk; no cross-family generalization signal.
- D ($15 → $50 cost amendment without scope reframe) — full sweep N=40 still $164, 3× over $50; not a real solution unless raised to $200+ which is significant cumulative budget overhead.
**Memory entries created at closure:**
- `feedback_anchor_multiply_input_size_regime.md` — methodology rule for cost projection
- `feedback_probe_first_roi_demonstration.md` — probe-first ROI evidence + amendment precedent
- `project_gepa_ood_arxiv_evidence.md` — arxiv §5.4 cross-domain methodology validation hook
- `project_are_native_docker_architectural_solution.md` — Phase 4 Docker architectural argument
---
**End of memo. Phase 3 CLOSED. Sesija C STANDBY pending Phase 4 setup ratification.**

View File

@@ -1,180 +0,0 @@
#!/usr/bin/env python
"""Offline independent re-judge of the GAIA 2 search-split N=160 run.
WHY: the search split is judged ~entirely by the LLM `user_message_checker`
(semantic equivalence of the agent's final message vs the oracle answer; there
are no app-action oracle events to hard-match). The production run self-judged
(Sonnet 4.6 judging a Sonnet 4.6 agent), so the pass rate may be inflated.
This script re-runs the EXACT GAIA 2 `user_message_checker` (same prompt, same
few-shot examples, same [[Success]]/[[Failure]] parsing — imported directly from
gaia2_core) against independent judge models, holding everything else constant.
Only the judge MODEL changes. Mirrors the C-1 LOCOMO trio-strict discipline.
Run inside the runner venv:
cd external/.../gaia2-cli/runner
./.venv/Scripts/python.exe <thispath> --probe # validate engines only
./.venv/Scripts/python.exe <thispath> --run # full re-judge
"""
from __future__ import annotations
import argparse, glob, json, os, sys
ENV_FILE = r"D:/Projects/waggle-os/external/meta-agents-research-environments/gaia2-cli/.env"
RUN = r"D:/Projects/waggle-os-gaia2-wt/benchmarks/gaia2/runs/p4-full-hermes-n160/search"
CACHE = r"C:/Users/MarkoMarkovic/.cache/gaia2/hf_datasets/meta-agents-research-environments_gaia2-cli/search"
OUT = r"D:/Projects/waggle-os-gaia2-wt/benchmarks/gaia2/runs/rejudge-search-n160.jsonl"
def _load_env():
"""Load gaia2-cli/.env into os.environ (ANTHROPIC_API_KEY lives only there)."""
if os.path.exists(ENV_FILE):
for line in open(ENV_FILE, encoding="utf-8"):
line = line.strip()
if line and not line.startswith("#") and "=" in line:
k, v = line.split("=", 1)
os.environ.setdefault(k.strip(), v.strip().strip('"').strip("'"))
# Judge roster (M6, independent of the Sonnet 4.6 agent+self-judge).
# gemini/ prefix → AI-Studio API-key path (not Vertex). GPT-5 needs drop_params (no temp=0).
JUDGES = [
{"name": "opus-4.7", "model": "claude-opus-4-7", "provider": "anthropic",
"api_key_env": "ANTHROPIC_API_KEY", "base_url": None},
{"name": "gemini-2.5-pro","model": "gemini/gemini-2.5-pro", "provider": None,
"api_key_env": "GEMINI_API_KEY", "base_url": None},
{"name": "gpt-5.x", "model": "openai/gpt-5", "provider": "openai-compat",
"api_key_env": "OPENROUTER_API_KEY","base_url": "https://openrouter.ai/api/v1"},
]
def _arg(action_args, name):
"""scenario events store args as a list of {name,value} dicts."""
if isinstance(action_args, dict):
return action_args.get(name)
for a in action_args or []:
if a.get("name") == name:
return a.get("value")
return None
def extract(scen_dir):
sid = os.path.basename(scen_dir)
sf = os.path.join(CACHE, sid + ".json")
task = oracle = agent = None
if os.path.exists(sf):
d = json.load(open(sf, encoding="utf-8"))
for ev in d.get("events", []):
act = ev.get("action", {}) or {}
fn = act.get("function")
if fn == "send_message_to_agent" and task is None:
task = _arg(act.get("args"), "content")
if fn == "send_message_to_user" and ev.get("class_name") == "OracleEvent" and oracle is None:
oracle = _arg(act.get("args"), "content")
ar = os.path.join(scen_dir, "agent_response.txt")
if os.path.exists(ar):
agent = open(ar, encoding="utf-8", errors="replace").read().strip()
# original self-judge verdict
rf = os.path.join(scen_dir, "result.json")
self_v = None
if os.path.exists(rf):
self_v = json.load(open(rf, encoding="utf-8")).get("success")
return sid, task, agent, oracle, self_v
def _make_engine(judge):
"""Minimal litellm engine matching the (messages, **kwargs) -> (content, info)
contract of gaia2's create_litellm_engine, but WITHOUT temperature — the M6
roster (Opus 4.7, GPT-5) rejects/deprecates temperature and litellm lacks
metadata to drop it. Uniform across all judges so the comparison stays fair."""
import litellm
litellm.drop_params = True
key = os.environ.get(judge["api_key_env"])
if not key:
raise RuntimeError(f"missing {judge['api_key_env']}")
model = judge["model"]
if judge["provider"] in ("openai", "openai-compat") and not model.startswith("openai/"):
model = "openai/" + model
def engine(messages, **kwargs):
try:
r = litellm.completion(model=model, messages=messages,
api_base=judge["base_url"], api_key=key, max_retries=8)
return r.choices[0].message.content, {"model": model}
except Exception as exc:
return None, {"error": str(exc)}
return engine
def build_checker(judge):
from gaia2_core.judge.checkers import LLMChecker
from gaia2_core.judge import prompts as P
eng = _make_engine(judge)
probe, info = eng([{"role": "user", "content": "Say OK"}])
if probe is None:
raise RuntimeError(f"validation failed: {info}")
return LLMChecker(engine=eng, prompt_templates=P.USER_MESSAGE_CHECKER_PROMPT_TEMPLATES,
num_votes=1)
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--probe", action="store_true", help="validate engines + extraction only")
ap.add_argument("--run", action="store_true", help="full re-judge")
ap.add_argument("--limit", type=int, default=None)
ap.add_argument("--run-dir", default=None, help="override RUN (the <output>/search dir to re-judge)")
ap.add_argument("--out", default=None, help="override OUT jsonl path")
args = ap.parse_args()
_load_env()
global RUN, OUT
if args.run_dir:
RUN = args.run_dir
if args.out:
OUT = args.out
dirs = sorted(d for d in glob.glob(os.path.join(RUN, "*")) if os.path.isdir(d))
rows = [extract(d) for d in dirs]
answerable = [r for r in rows if r[1] and r[2] and r[3] is not None]
print(f"scenarios={len(rows)} answerable(task+agent+oracle)={len(answerable)}")
if args.probe:
for j in JUDGES:
try:
build_checker(j); print(f" engine OK: {j['name']} ({j['model']})")
except Exception as e:
print(f" engine FAIL: {j['name']} -> {str(e)[:160]}")
s = answerable[0]
print(f" sample sid={s[0]} self={s[4]} task={s[1][:60]!r} oracle={s[3][:40]!r}")
return
if not args.run:
print("pass --probe or --run"); return
checkers = {}
for j in JUDGES:
try:
checkers[j["name"]] = build_checker(j); print(f"engine ready: {j['name']}")
except Exception as e:
print(f"engine SKIP {j['name']}: {str(e)[:160]}")
todo = answerable[: args.limit] if args.limit else answerable
out = open(OUT, "w", encoding="utf-8")
n = 0
for sid, task, agent, oracle, self_v in todo:
rec = {"scenario_id": sid, "self_judge": self_v, "verdicts": {}}
upa = {"agent_action_call": agent, "oracle_action_call": oracle, "task": task}
for name, chk in checkers.items():
try:
v = chk(upa)
except Exception as e:
v = None; rec.setdefault("errors", {})[name] = str(e)[:120]
rec["verdicts"][name] = v
out.write(json.dumps(rec) + "\n"); out.flush()
n += 1
if n % 10 == 0:
print(f" judged {n}/{len(todo)}")
out.close()
print(f"wrote {n} rows -> {OUT}")
if __name__ == "__main__":
main()

View File

@@ -1 +0,0 @@
{"start_time": 0, "time_increment_in_seconds": 1, "duration": null, "event_log": {"past_events": [{"class_name": "CompletedEvent", "event_type": "ENV", "event_time": 0, "event_relative_time": null, "event_id": "user_request", "successors": [], "dependencies": [], "metadata": {"return_value": "7ec5bf32-7b2d-4bc9-a05f-4fbc6309c4c7", "exception": null, "exception_stack_trace": null, "completed": true}, "action": {"class_name": "AgentUserInterface", "app_name": "AgentUserInterface", "function_name": "send_message_to_agent", "args": {"content": "I need to find the image file in the current directory", "attachments": null, "base64_utf8_encoded_attachment_contents": null}, "resolved_args": {"content": "I need to find the image file in the current directory", "attachments": null, "base64_utf8_encoded_attachment_contents": null}, "operation_type": "write", "action_id": "AgentUserInterface.send_message_to_agent-f328547f-a0b0-43b6-8134-409374f19af3"}}, {"class_name": "CompletedEvent", "event_type": "AGENT", "event_time": 1, "event_relative_time": null, "event_id": "oracle_response", "successors": [], "dependencies": [], "metadata": {"return_value": null, "exception": null, "exception_stack_trace": null, "completed": true}, "action": {"class_name": "AgentUserInterface", "app_name": "AgentUserInterface", "function_name": "send_message_to_user", "args": {"content": "The image file is `llama.jpg`"}, "resolved_args": {"content": "The image file is `llama.jpg`"}, "operation_type": "write", "action_id": "AgentUserInterface.send_message_to_user-3662fe69-85d0-485d-a6c1-c13871de4892"}}]}, "event_queue": {"future_events": []}, "current_time": 1, "apps": [{"app_name": "SandboxLocalFileSystem", "files": {"name": "", "type": "directory", "children": [{"name": "09a45055-0e1d-4d60-938a-f2245558efab.txt", "type": "file"}, {"name": "216779ee-cd9b-421f-b005-98c746797370.txt", "type": "file"}, {"name": "3fafa79b-3942-4ac0-af16-b7577825c215.txt", "type": "file"}, {"name": "74986a41-0b62-48dc-aee0-4bf9ce594572.txt", "type": "file"}, {"name": "a9101dd4-f925-469b-9092-e284f706563a.txt", "type": "file"}, {"name": "c3155da7-88f5-42cd-ae2d-1924ee9c26d5.txt", "type": "file"}, {"name": "c8b51dca-4048-4692-9fb8-fb939ea75100.txt", "type": "file"}, {"name": "d5e17ffc-84cd-4555-a13f-48541b85a45c.txt", "type": "file"}, {"name": "e0cdf73b-b20b-4c07-94a7-ba5af07812a0.txt", "type": "file"}, {"name": "edd2936c-dc00-45bf-a3ce-c83c6de3b351.txt", "type": "file"}, {"name": "llama.jpg", "type": "file"}]}, "tmpdir": "C:\\Users\\MARKOM~1\\AppData\\Local\\Temp\\are_simulation_sandbox_9ejpy6c0\\are_simulation_fs_sandbox_vn40bdw1"}, {"app_name": "AgentUserInterface", "messages": [{"sender": "User", "content": "I need to find the image file in the current directory", "attachments": [], "timestamp": 0.0030155181884765625, "time_read": null, "id": "7ec5bf32-7b2d-4bc9-a05f-4fbc6309c4c7"}, {"sender": "Agent", "content": "The image file is `llama.jpg`", "attachments": [], "timestamp": 1.0, "time_read": 1.0, "id": "756a8795-b684-4913-ac3e-05b166284682"}]}]}

View File

@@ -1 +0,0 @@
{"start_time": 0, "time_increment_in_seconds": 1, "duration": null, "event_log": {"past_events": []}, "event_queue": {"future_events": []}, "current_time": 0, "apps": [{"app_name": "SandboxLocalFileSystem", "files": {"name": "", "type": "directory", "children": [{"name": "09a45055-0e1d-4d60-938a-f2245558efab.txt", "type": "file"}, {"name": "216779ee-cd9b-421f-b005-98c746797370.txt", "type": "file"}, {"name": "3fafa79b-3942-4ac0-af16-b7577825c215.txt", "type": "file"}, {"name": "74986a41-0b62-48dc-aee0-4bf9ce594572.txt", "type": "file"}, {"name": "a9101dd4-f925-469b-9092-e284f706563a.txt", "type": "file"}, {"name": "c3155da7-88f5-42cd-ae2d-1924ee9c26d5.txt", "type": "file"}, {"name": "c8b51dca-4048-4692-9fb8-fb939ea75100.txt", "type": "file"}, {"name": "d5e17ffc-84cd-4555-a13f-48541b85a45c.txt", "type": "file"}, {"name": "e0cdf73b-b20b-4c07-94a7-ba5af07812a0.txt", "type": "file"}, {"name": "edd2936c-dc00-45bf-a3ce-c83c6de3b351.txt", "type": "file"}, {"name": "llama.jpg", "type": "file"}]}, "tmpdir": "C:\\Users\\MARKOM~1\\AppData\\Local\\Temp\\are_simulation_sandbox_9ejpy6c0\\are_simulation_fs_sandbox_vn40bdw1"}, {"app_name": "AgentUserInterface", "messages": []}]}

View File

@@ -1 +0,0 @@
{"task_id": "scenario_find_image_file", "trace_id": null, "score": 1.0, "metadata": {"scenario_id": "scenario_find_image_file", "status": "success", "has_exception": false}}

View File

@@ -1,68 +0,0 @@
{
"metadata": {
"model": "meta-llama/llama3-70b-instruct",
"model_provider": "mock",
"timestamp": "2026-04-29T21:58:18.864353",
"report_version": "3.0"
},
"statistics": {
"per_capability": {
"mini_standard": {
"capability": "mini (standard)",
"total_runs": 3,
"validated_runs": 3,
"success_runs": 0,
"failed_runs": 3,
"exception_runs": 0,
"no_validation_runs": 0,
"success_rate": 0.0,
"success_rate_std": 0.0,
"success_rate_sem": 0.0,
"pass_at_k": 0,
"pass_at_k_percent": 0.0,
"pass_k": 0,
"pass_k_percent": 0.0,
"total_scenarios": 1,
"avg_run_duration": 0.0,
"avg_run_duration_std": 0.0
}
},
"global": {
"total_runs": 3,
"validated_runs": 3,
"success_runs": 0,
"failed_runs": 3,
"exception_runs": 0,
"no_validation_runs": 0,
"pass_at_k": 0,
"pass_at_k_percent": 0.0,
"pass_k": 0,
"pass_k_percent": 0.0,
"total_scenarios": 1,
"macro_success_rate": 0.0,
"macro_success_rate_std": 0.0,
"macro_success_rate_sem": 0.0,
"micro_success_rate": 0.0,
"micro_success_rate_std": 0.0,
"micro_success_rate_sem": 0.0,
"avg_run_duration": 0.0,
"avg_run_duration_std": 0.0,
"job_duration": 31.784090518951416
}
},
"run_configurations": [
{
"phase_name": "standard",
"config": "mini",
"a2a_app_prop": "0.0",
"has_tool_augmentation": "False",
"has_env_events": "False",
"total_runs": 3,
"validated_runs": 3,
"success_runs": 0,
"failed_runs": 3,
"exception_runs": 0,
"no_validation_runs": 0
}
]
}

View File

@@ -1,3 +0,0 @@
{"task_id": "scenario_universe_21_xvc7uo", "trace_id": null, "score": 0.0, "metadata": {"scenario_id": "scenario_universe_21_xvc7uo", "run_number": 1, "status": "failed", "has_exception": true, "exception_type": "AttributeError", "exception_message": "module 'signal' has no attribute 'SIGALRM'"}}
{"task_id": "scenario_universe_21_xvc7uo", "trace_id": null, "score": 0.0, "metadata": {"scenario_id": "scenario_universe_21_xvc7uo", "run_number": 2, "status": "failed", "has_exception": true, "exception_type": "AttributeError", "exception_message": "module 'signal' has no attribute 'SIGALRM'"}}
{"task_id": "scenario_universe_21_xvc7uo", "trace_id": null, "score": 0.0, "metadata": {"scenario_id": "scenario_universe_21_xvc7uo", "run_number": 3, "status": "failed", "has_exception": true, "exception_type": "AttributeError", "exception_message": "module 'signal' has no attribute 'SIGALRM'"}}

View File

@@ -1,85 +0,0 @@
#!/usr/bin/env python3
"""
Sesija C Phase 3b-A — Gaia2 HF dataset → JSONL dumper.
Runs from inside the ARE-installed venv (uv-managed under
external/meta-agents-research-environments/.venv); `datasets` package is
already a transitive dep of meta-agents-research-environments.
Usage:
python benchmarks/gaia2/scripts/dump-tasks.py \\
--hf-config mini \\
--hf-split validation \\
--limit 10 \\
--output benchmarks/gaia2/data/tasks-mini-10.jsonl
Emits one JSON record per line, schema matching benchmarks/gaia2/adapter.ts
`Gaia2HfTask` interface (HF dataset card schema verified 2026-04-30).
"""
import argparse
import json
import sys
from pathlib import Path
def main() -> int:
parser = argparse.ArgumentParser(description="Dump Gaia2 HF tasks to JSONL")
parser.add_argument("--hf-dataset", default="meta-agents-research-environments/gaia2")
parser.add_argument("--hf-config", required=True, help="config name (mini/search/execution/...)")
parser.add_argument("--hf-split", default="validation")
parser.add_argument("--limit", type=int, required=True)
parser.add_argument("--output", required=True, help="Output JSONL path")
args = parser.parse_args()
# `datasets` is installed via meta-agents-research-environments pyproject
# (transitive dep). Run this script under the ARE venv.
try:
from datasets import load_dataset # type: ignore
except ImportError as e:
print(
f"ERROR: `datasets` not installed. Run from ARE venv:\n"
f" cd external/meta-agents-research-environments && uv run python {sys.argv[0]} ...\n"
f"Underlying error: {e}",
file=sys.stderr,
)
return 2
print(f"Loading {args.hf_dataset} config={args.hf_config} split={args.hf_split} ...", file=sys.stderr)
ds = load_dataset(args.hf_dataset, args.hf_config, split=args.hf_split)
total = len(ds)
print(f"Dataset has {total} examples; limiting to {args.limit}", file=sys.stderr)
output_path = Path(args.output)
output_path.parent.mkdir(parents=True, exist_ok=True)
written = 0
with output_path.open("w", encoding="utf-8") as f:
for i, record in enumerate(ds):
if i >= args.limit:
break
# HF stores `data` as a serialized JSON STRING (not a nested
# object). HF dataset card sample showed the post-parse form;
# actual on-disk format is a string. Parse here so the JSONL
# written matches benchmarks/gaia2/adapter.ts Gaia2HfTask
# interface (data: { metadata, apps, events }).
data_field = record.get("data")
if isinstance(data_field, str):
try:
record["data"] = json.loads(data_field)
except json.JSONDecodeError as e:
print(
f"WARN: record {record.get('id')} has unparseable `data` "
f"string ({e}); writing raw string",
file=sys.stderr,
)
f.write(json.dumps(record, ensure_ascii=False))
f.write("\n")
written += 1
print(f"Wrote {written} task records to {output_path}", file=sys.stderr)
return 0
if __name__ == "__main__":
sys.exit(main())

View File

@@ -1,319 +0,0 @@
#!/usr/bin/env tsx
/**
* Sesija C Phase 3b — Gaia2 dry-verification driver (OpenRouter direct).
*
* Wires:
* - benchmarks/gaia2/config.yaml → Gaia2AdapterConfig
* - benchmarks/gaia2/data/*.jsonl → loaded via adapter.loadGaia2TasksFromJsonl
* - OPENROUTER_API_KEY → LlmCallFn (OpenRouter direct chat-completions)
* - benchmarks/gaia2/adapter.ts → runDryRunSweep
* - benchmarks/gaia2/runs/<ISO>/ → JSONL output
*
* Phase 3b-B-1 patch: replaced LiteLLM proxy transport with OpenRouter
* direct per PM ratification Option B (2026-04-30 Phase 3b-B kick-off
* env-var halt resolved). LITELLM_URL + LITELLM_MASTER_KEY were empty
* in the shell; OPENROUTER_API_KEY was present. Proportional pivot for
* 4-invocation probe scope.
*
* Slug map (`LITELLM_TO_OPENROUTER_SLUG`):
* - claude-opus-4-7 → anthropic/claude-opus-4.7 (EXACT match)
* - qwen3.6-35b-a3b → qwen/qwen3-30b-a3b-thinking-2507
* (closest available; 30b vs 35b size delta;
* same a3b architecture + thinking-mode;
* proportional for cost-projection probe)
*
* Cost methodology: per-call cost = inTokens × priceIn/1e6 +
* outTokens × priceOut/1e6, prices from Faza 1 PRICE_TABLE
* (decisions/2026-04-29-gepa-faza1-results.md §F). Opus prices are
* exact (Faza 1 used same model). Qwen prices are the Faza 1
* `qwen3.6-35b-a3b-via-openrouter` row applied to the 30b OpenRouter
* slug as projection-grade approximation.
*
* Usage:
* pnpm tsx benchmarks/gaia2/scripts/run-dry-verification.ts \\
* --tasks benchmarks/gaia2/data/tasks-mini-2.jsonl \\
* --halt-after-probe # (Phase 3b-B-2 sample mode)
*
* Env required:
* OPENROUTER_API_KEY (required; FATAL if unset)
*/
import * as fs from 'node:fs';
import * as path from 'node:path';
import { pathToFileURL } from 'node:url';
import * as yaml from 'js-yaml';
import {
runDryRunSweep,
writeRecordsToJsonl,
type Gaia2AdapterConfig,
type LlmCallFn,
} from '../adapter.js';
// ─── OpenRouter constants ────────────────────────────────────────────
const OPENROUTER_URL = 'https://openrouter.ai/api/v1/chat/completions';
const SUBJECT_MAX_TOKENS_QWEN = 16_000;
const SUBJECT_MAX_TOKENS_DEFAULT = 4_096;
/**
* Translate adapter.ts SHAPE_TO_MODEL names (LiteLLM aliases) into
* OpenRouter slugs. Validated 2026-04-30 against
* https://openrouter.ai/api/v1/models response.
*
* If a shape's underlying model is missing from this map at runtime,
* the call fails fast with a clear error.
*/
const LITELLM_TO_OPENROUTER_SLUG: Readonly<Record<string, string>> = Object.freeze({
'claude-opus-4-7': 'anthropic/claude-opus-4.7',
'qwen3.6-35b-a3b': 'qwen/qwen3-30b-a3b-thinking-2507',
// Fallbacks for shapes not in the brief 4-shape sweep but reachable
// via SHAPE_TO_MODEL in adapter.ts (defensive — would only fire if
// config.yaml is widened beyond BRIEF_DRY_RUN_SHAPES).
'gpt-5.4': 'openai/gpt-4o-2024-08-06', // closest available; flag if used
'claude-haiku-4-5': 'anthropic/claude-haiku-4.5',
});
/**
* Faza 1 PRICE_TABLE rows (USD per 1M tokens), used for projection-grade
* cost computation when OpenRouter response doesn't include `usage.cost`
* (which requires `usage.include: true` in request body).
*
* Source: benchmarks/gepa/scripts/faza-1/run-checkpoint-c.ts:106-110.
* Opus prices are wire-accurate. Qwen 30b prices are approximation
* inherited from `qwen3.6-35b-a3b-via-openrouter`.
*/
const PRICE_TABLE: Readonly<Record<string, { in: number; out: number }>> = Object.freeze({
'anthropic/claude-opus-4.7': { in: 15.0, out: 75.0 },
'qwen/qwen3-30b-a3b-thinking-2507': { in: 0.6, out: 2.4 },
'openai/gpt-4o-2024-08-06': { in: 2.5, out: 10.0 },
'anthropic/claude-haiku-4.5': { in: 1.0, out: 5.0 },
});
// ─── OpenRouter LlmCallFn ────────────────────────────────────────────
function makeOpenRouterCallFn(opts: { apiKey: string }): LlmCallFn {
return async (input) => {
const slug = LITELLM_TO_OPENROUTER_SLUG[input.model];
if (!slug) {
return {
content: '',
inTokens: 0,
outTokens: 0,
costUsd: 0,
latencyMs: 0,
error: `No OpenRouter slug mapping for model "${input.model}". Add to LITELLM_TO_OPENROUTER_SLUG in run-dry-verification.ts.`,
};
}
const isQwen = slug.includes('qwen');
const maxTokens = input.maxTokens ?? (isQwen ? SUBJECT_MAX_TOKENS_QWEN : SUBJECT_MAX_TOKENS_DEFAULT);
const payload: Record<string, unknown> = {
model: slug,
messages: input.messages,
max_tokens: maxTokens,
// OpenRouter `usage.include: true` opt-in returns wire-accurate cost
// in the response. We capture this when present and fall back to
// PRICE_TABLE-based computation otherwise.
usage: { include: true },
};
// Temperature: Opus needs 1.0 sentinel per Faza 1 convention; GPT-5.x
// family rejects temperature override; everything else gets 0.3.
if (slug.startsWith('anthropic/claude-opus')) {
payload.temperature = 1.0;
} else if (slug.startsWith('openai/gpt-5')) {
// omit temperature (provider rejects override)
} else {
payload.temperature = input.temperature ?? 0.3;
}
// Qwen thinking-mode is encoded in the slug itself (`-thinking-2507`)
// for the OpenRouter route — no `enable_thinking` extra_body needed.
// Verified against OpenRouter qwen3-*-thinking-2507 model cards.
const startMs = Date.now();
let lastErr: string | undefined;
for (let attempt = 0; attempt < 2; attempt++) {
try {
const resp = await fetch(OPENROUTER_URL, {
method: 'POST',
headers: {
'Content-Type': 'application/json',
Authorization: `Bearer ${opts.apiKey}`,
// OpenRouter conventions:
'HTTP-Referer': 'https://waggle-os.ai',
// ASCII-only — fetch() rejects non-ByteString header values
// (U+2014 em dash etc. throw "Cannot convert argument to a
// ByteString" at request time).
'X-Title': 'Waggle Sesija C - Gaia2 narrow-proxy probe',
},
body: JSON.stringify(payload),
});
const body = (await resp.json()) as Record<string, unknown>;
if ('error' in body) {
const errObj = body.error as Record<string, unknown> | undefined;
lastErr = String(errObj?.message ?? JSON.stringify(body.error)).slice(0, 300);
if (attempt < 1) {
await new Promise((r) => setTimeout(r, 1500));
continue;
}
return {
content: '',
inTokens: 0,
outTokens: 0,
costUsd: 0,
latencyMs: Date.now() - startMs,
error: lastErr,
};
}
const choices = body.choices as
| Array<{ message?: { content?: string; reasoning?: string } }>
| undefined;
const content = choices?.[0]?.message?.content ?? '';
const usage = body.usage as
| { prompt_tokens?: number; completion_tokens?: number; cost?: number }
| undefined;
const inTokens = usage?.prompt_tokens ?? 0;
const outTokens = usage?.completion_tokens ?? 0;
// Wire-accurate cost preferred; fall back to PRICE_TABLE.
let costUsd = usage?.cost ?? 0;
if (costUsd === 0 && PRICE_TABLE[slug]) {
const pt = PRICE_TABLE[slug];
costUsd = (inTokens * pt.in) / 1_000_000 + (outTokens * pt.out) / 1_000_000;
}
return {
content,
inTokens,
outTokens,
costUsd,
latencyMs: Date.now() - startMs,
};
} catch (err: unknown) {
lastErr = err instanceof Error ? err.message : String(err);
if (attempt < 1) {
await new Promise((r) => setTimeout(r, 1500));
continue;
}
}
}
return {
content: '',
inTokens: 0,
outTokens: 0,
costUsd: 0,
latencyMs: Date.now() - startMs,
error: lastErr ?? 'unknown OpenRouter call failure',
};
};
}
// ─── CLI ──────────────────────────────────────────────────────────────
function parseArgs(): { tasksFile: string; haltAfterProbe: boolean; configPath: string } {
const args = process.argv.slice(2);
let tasksFile = 'benchmarks/gaia2/data/tasks-mini-10.jsonl';
let haltAfterProbe = false;
let configPath = 'benchmarks/gaia2/config.yaml';
for (let i = 0; i < args.length; i++) {
const a = args[i];
if (a === '--tasks' && i + 1 < args.length) {
tasksFile = args[i + 1];
i++;
} else if (a === '--halt-after-probe') {
haltAfterProbe = true;
} else if (a === '--config' && i + 1 < args.length) {
configPath = args[i + 1];
i++;
}
}
return { tasksFile, haltAfterProbe, configPath };
}
function loadConfig(configPath: string): Gaia2AdapterConfig {
const raw = fs.readFileSync(configPath, 'utf-8');
const parsed = yaml.load(raw) as Record<string, unknown>;
return parsed as unknown as Gaia2AdapterConfig;
}
async function main(): Promise<number> {
const { tasksFile, haltAfterProbe, configPath } = parseArgs();
console.error(
`[gaia2-dry-verification] config=${configPath} tasks=${tasksFile} ` +
`haltAfterProbe=${haltAfterProbe}`,
);
const config = loadConfig(configPath);
const apiKey = process.env.OPENROUTER_API_KEY;
if (!apiKey || apiKey.length === 0) {
console.error(
'FATAL: OPENROUTER_API_KEY must be set in env (Phase 3b-B-1 OpenRouter direct fallback per PM ratification Option B).',
);
return 2;
}
const llmCall = makeOpenRouterCallFn({ apiKey });
const startedAt = new Date().toISOString().replace(/[:.]/g, '-');
const outDir = path.resolve(
process.cwd(),
config.output_dir_root,
`dry-verification-${startedAt}`,
);
fs.mkdirSync(outDir, { recursive: true });
const result = await runDryRunSweep(config, llmCall, {
tasksFile,
haltAfterProbe,
});
writeRecordsToJsonl(result.probe_records, path.join(outDir, 'probe.jsonl'));
writeRecordsToJsonl(result.full_sweep_records, path.join(outDir, 'full-sweep.jsonl'));
const summary = {
timestamp_started: new Date().toISOString(),
config_path: configPath,
tasks_file: tasksFile,
transport: 'openrouter-direct' as const,
probe_invocation_count: result.probe_records.length,
probe_cost_usd: result.probe_cost_usd,
projected_total_usd: result.projected_total_usd,
halt_triggered: result.halt_triggered,
full_sweep_invocation_count: result.full_sweep_records.length,
final_total_cost_usd: result.final_total_cost_usd,
cost_cap_usd: config.cost_cap_usd,
halt_trigger_usd: config.halt_trigger_usd,
slug_map: LITELLM_TO_OPENROUTER_SLUG,
price_table: PRICE_TABLE,
};
fs.writeFileSync(
path.join(outDir, 'summary.json'),
JSON.stringify(summary, null, 2),
'utf-8',
);
console.error(`[gaia2-dry-verification] complete — output dir: ${outDir}`);
console.error(JSON.stringify(summary, null, 2));
return result.halt_triggered ? 1 : 0;
}
if (import.meta.url === pathToFileURL(process.argv[1] ?? '').href) {
main().then(
(code) => process.exit(code),
(err) => {
console.error('FATAL:', err instanceof Error ? err.stack : err);
process.exit(3);
},
);
}

View File

@@ -1,152 +0,0 @@
# Phase 2 Smoke Evidence (Sesija C Task C1+C2)
**Date:** 2026-04-30 (executed 2026-04-29 21:55–21:58 local timestamp inside ARE logs; date discrepancy is local-clock drift, branch + commit time stamps are authoritative)
**Branch:** `feature/gaia2-are-setup` @ Phase 1 commit `a72b724`
**Wall-clock:** ~3 minutes total (sync 1.5min + Smoke A 1s + Smoke B-retry 31.8s)
**LLM cost:** **$0.00** (oracle mode + mock provider — no real model invocations)
---
## §1 — ARE platform clone (Task C1.a)
| Field | Value |
|---|---|
| Repo | `https://github.com/facebookresearch/meta-agents-research-environments` |
| Cloned to | `D:/Projects/waggle-os/external/meta-agents-research-environments/` (gitignored) |
| Clone strategy | `git clone --depth 1` (shallow; full history not required for setup verification) |
| **Pinned SHA** | **`0330191ffef8581e3c0620b78df9c7408bcb98b0`** (2026-04-20 11:53:48 +0200, "Format run-config HF split selection fix (#55)") |
| License | MIT |
| Repo size | 95+ Python deps + are/ source tree (~2,000+ Python files) |
| Last upstream commit at clone | 2026-04-20 (10 days before Sesija C kickoff) |
**Note on SHA discipline (per branch architecture LOCKED §4.1 binding):** The pinned SHA above was captured from `git rev-parse HEAD` after clone, not from memory. Cite this SHA going forward, not "latest main".
---
## §2 — Install verification (Task C1.b)
**Method:** `uv sync --frozen` from `external/meta-agents-research-environments/`.
| Field | Value |
|---|---|
| uv version | 0.8.17 |
| Python version | 3.10.18 (uv-managed; auto-selected per `requires-python = ">=3.10"` in `pyproject.toml`; system Python 3.11.9 not used) |
| Dependency count | 95 packages |
| Editable install | meta-agents-research-environments 1.2.0 |
| Key dependencies | `litellm 1.71.1` (LiteLLM-compatible — aligns with our existing routing layer); `huggingface-hub 0.33.4`; `mcp 1.11.0`; `datasets 4.0.0`; `aiohttp 3.13.2`; `httpx 0.28.1` |
| Install duration | ~1.5 min wall-clock |
| Exit code | 0 (clean install) |
**Binary verification:** `uv run are-run --help` and `uv run are-benchmark gaia2-run --help` both return clean usage output.
---
## §3 — Smoke A: Oracle mode, built-in scenario (Task C2.a)
**Command:**
```bash
cd external/meta-agents-research-environments && \
uv run are-run -o -s scenario_find_image_file \
--output_dir benchmarks/gaia2/runs/smoke-c2-2026-04-30/smoke-A-oracle
```
**Result:** **PASS.**
- `ScenarioValidationResult(success=True, exception=None, ...)`
- `Success=100.0%`, 1 scenario completed in <1 second
- Output files: `output.jsonl`, `initial_state.jsonl`, `final_state.jsonl`
- Output format (output.jsonl): `{"task_id": "scenario_find_image_file", "trace_id": null, "score": 1.0, "metadata": {"scenario_id": "scenario_find_image_file", "status": "success", "has_exception": false}}`
**Cosmetic warning (non-blocking):** Windows cp1252 codec cannot encode `✅` (✅) emoji from ARE's logger. UnicodeEncodeError raised inside `logging_config.py:61`, scenario completes normally. Acceptable for headless CI/CD if stdout encoding is set to utf-8.
---
## §4 — Smoke B: Gaia2 mini config × 1 scenario, mock provider (Task C2.b)
**First attempt — `gaia2-run` with default executor:** ❌ FAILED on Windows. Error: `cannot find context for 'fork'`. ARE's `gaia2-run` orchestrates 7 phase/configs (standard/{ambiguity, adaptability, execution, search, time}, agent2agent/mini, noise/mini) using `multiprocessing` with `fork` context — Windows only supports `spawn`. All 7 phases skipped.
**Second attempt — `are-benchmark run` with thread executor:** ✅ **PASS.**
```bash
cd external/meta-agents-research-environments && \
uv run are-benchmark run \
--hf-dataset meta-agents-research-environments/gaia2 \
--hf-config mini \
--hf-split validation \
-l 1 \
--provider mock \
--agent default \
--executor_type thread \
--max_concurrent_scenarios 1 \
--output_dir benchmarks/gaia2/runs/smoke-c2-2026-04-30/smoke-B-gaia2-mock-thread \
--trace_dump_format lite
```
| Metric | Value |
|---|---|
| Dataset loaded | 160 examples in `mini/validation` split |
| Scenarios run | 1 unique × 3 runs (Pass@3 standard) |
| Wall-clock | 31.8 seconds |
| Provider | `mock` (default model alias `meta-llama/llama3-70b-instruct`, no real inference) |
| Success rate | 0.0% (mock provider returns fake responses; expected) |
| Exit code | 0 (1 config attempted, 1 successful) |
| Output files | `output.jsonl`, `benchmark_stats.json` |
**Per-run failure mode (3/3 runs):** `module 'signal' has no attribute 'SIGALRM'` — Windows lacks the Unix `SIGALRM` signal used by ARE's per-scenario timeout enforcement (`scenario_runner.py` raises `AttributeError`). Caught and recorded as `exception_runs` in stats, but blocks ANY scenario from completing on Windows even with mock provider.
**Output format (sample run, output.jsonl line):**
```json
{
"task_id": "scenario_universe_21_xvc7uo",
"trace_id": null,
"score": 0.0,
"metadata": {
"scenario_id": "scenario_universe_21_xvc7uo",
"run_number": 1,
"status": "failed",
"has_exception": true,
"exception_type": "AttributeError",
"exception_message": "module 'signal' has no attribute 'SIGALRM'"
}
}
```
**Stats schema (benchmark_stats.json):** `metadata{model, model_provider, timestamp, report_version}` + `statistics{per_capability{<config>{success_rate, pass_at_k, pass_k, total_runs, ...}}, global{macro_success_rate, micro_success_rate, pass_at_k, pass_k, job_duration, ...}}`. Pass@k + Pass^k are first-class metrics (k=3 by default for Gaia2 standard).
---
## §5 — Windows compat findings + Phase 3+4 implications
| Finding | Where it surfaces | Workaround for Phase 3 (adapter) | Workaround for Phase 4 (dry run) |
|---|---|---|---|
| `multiprocessing.get_context('fork')` fails on Windows | `gaia2-run` full benchmark orchestrator | Use `are-benchmark run` directly (per-config) instead of `gaia2-run`; our adapter wraps single-config calls anyway | Use `--executor_type thread` for parallelism + run configs sequentially |
| `signal.SIGALRM` missing on Windows | `scenario_runner.py` per-scenario timeout enforcement (every scenario, regardless of provider/agent) | **Hard blocker** for Windows — adapter must either (a) patch `signal.SIGALRM` shim, (b) run inside Docker/WSL, or (c) document that dry run requires Linux | **DECISION POINT for PM**: Windows-host dry run blocked. Options: (i) WSL2 / Linux subsystem; (ii) Docker (Dockerfile present in ARE repo); (iii) accept Windows + monkey-patch; (iv) defer Phase 4 to Linux CI runner. |
| Unicode `✅` emoji in logger crashes on cp1252 | `logging_config.py:61` (cosmetic only) | Set `PYTHONIOENCODING=utf-8` env var before invoking | Same env var setting |
**Severity ranking:** SIGALRM is **HIGH** (blocks all scenario execution on Windows host). Fork is **MEDIUM** (blocks `gaia2-run` orchestrator only, our adapter doesn't need it). Unicode is **LOW** (cosmetic).
**Fastest unblock for Phase 4:** Docker. The ARE repo ships a `Dockerfile` (`external/meta-agents-research-environments/Dockerfile`). A containerized run sidesteps both fork + SIGALRM issues since the container runs Linux. Trade-off: Docker daemon dependency on operator machine, ~5-15min initial image build.
**Decision deferred to PM:** Phase 4 Windows host vs Docker vs WSL ratification before Task C5 (dry run execution).
---
## §6 — What the smoke verified (smoke verdict)
| Brief §0.1 acceptance criterion | Verified by Smoke A+B |
|---|---|
| ARE platform installable | ✅ uv sync exit 0, 95 deps |
| Default agent runs scenarios | ✅ Smoke A: scenario_find_image_file PASS oracle; Smoke B: Gaia2 mini scenario_universe_21_xvc7uo executed (failed only on SIGALRM, not on agent logic) |
| HF dataset accessible | ✅ Smoke B downloaded mini/validation 160 examples |
| Output format reproducible | ✅ output.jsonl + benchmark_stats.json schemas captured §3 + §4 above |
| Provider config supports mock + LiteLLM-compatible providers | ✅ `mock` provider works; ARE shares LiteLLM 1.71.1 with our LLM routing layer (zero-friction Phase 3 wiring) |
**Phase 2 verdict: SMOKE PASS** (with documented Windows-compat halt-and-PM trigger for Phase 4 host choice).
---
## §7 — Audit anchors
- ARE clone SHA: `0330191ffef8581e3c0620b78df9c7408bcb98b0`
- Smoke A output: `runs/smoke-c2-2026-04-30/smoke-A-oracle/` (gitignored; reproducible from §3 command)
- Smoke B output: `runs/smoke-c2-2026-04-30/smoke-B-gaia2-mock-thread/` (gitignored; reproducible from §4 command)
- This evidence: `benchmarks/gaia2/smoke-evidence.md`

View File

@@ -1,13 +0,0 @@
# waggle_worker spike — Path A de-risk (PROVEN 2026-05-22)
Proves Waggle's `runAgentLoop` runs with ZERO native deps via a 2-symbol `@waggle/core`
stub (re-exporting the DB-free `createCoreLogger` + `scanForInjection` from hive-mind-core
deep paths), bypassing the `db.js` barrel that eagerly loads `better-sqlite3`/`sqlite-vec`.
Result (isolated dir outside the monorepo, better-sqlite3 NOT resolvable):
{ import_ok: true, runAgentLoop: "function", better_sqlite3: "not-resolvable (clean)" }
→ The `gaia2-waggle` container = node:20-slim + agent dist/ + this 2-symbol stub +
hive-mind-core/dist/{logger.js,injection-scanner.js}. No native rebuild needed.
To re-run: copy `packages/agent/dist` here as `dist/`, then `node proof.mjs`.

View File

@@ -1,16 +0,0 @@
const t0 = Date.now();
try {
const m = await import("./dist/agent-loop.js");
const hasRun = typeof m.runAgentLoop === "function";
// verify better-sqlite3 was never loaded into the module cache
let sqliteLoaded = false;
try { await import("better-sqlite3"); sqliteLoaded = true; } catch { sqliteLoaded = false; }
console.log(JSON.stringify({
import_ok: true,
runAgentLoop_exported: hasRun,
better_sqlite3_resolvable: sqliteLoaded, // false = not even installed = proof loop didn't need it
ms: Date.now() - t0,
}, null, 2));
} catch (e) {
console.log(JSON.stringify({ import_ok: false, error: String(e).slice(0, 300) }, null, 2));
}

View File

@@ -1 +0,0 @@
{"name":"waggle-worker-spike","version":"0.0.0","type":"module"}

View File

@@ -1,72 +0,0 @@
# gaia2-waggle — build & run (Pillar 1: Waggle harness in the GAIA 2 rig)
Wraps **Waggle's own `runAgentLoop`** in the GAIA 2 ARE, fair vs `gaia2-hermes`:
same single `terminal` tool + same `AGENTS.md` + same model (Sonnet 4.6). Only the
loop logic differs. Path-A: no native deps (2-symbol `@waggle/core` stub — proven in
`../spike-waggle-worker/`).
## Files here (tracked recovery copies; staged into `external/.../gaia2-cli/containers/waggle/`)
- `Dockerfile` — gaia2-cli base + Node + waggle payload (models gaia2-hermes)
- `waggle_worker.mjs` — Node worker: socket protocol + terminal tool + runAgentLoop
- `entrypoint.sh` — launches the Node worker (vs hermes' python worker)
- `stub-core/` — the 2-symbol `@waggle/core` stub package
## Build steps (from `external/meta-agents-research-environments/gaia2-cli/`)
```bash
WC=containers/waggle
mkdir -p $WC/payload/node_modules/@waggle
# 1. entrypoints
cp <thisdir>/Dockerfile $WC/Dockerfile
cp <thisdir>/entrypoint.sh $WC/entrypoint.sh
cp <thisdir>/waggle_worker.mjs $WC/payload/waggle_worker.mjs
echo '{"type":"module"}' > $WC/payload/package.json
# init-entrypoint is hermes' VERBATIM (its hermes-isms are inert; it execs /opt/entrypoint.sh
# which the waggle Dockerfile overrides):
cp containers/hermes/gaia2-init-entrypoint.sh $WC/gaia2-init-entrypoint.sh
# 2. @waggle/agent dist (built). NOTE: do NOT copy the source package.json — its
# exports map ({".":"./src/index.ts"}) blocks the /dist/ subpath import
# (ERR_PACKAGE_PATH_NOT_EXPORTED). Write a minimal one with no exports field.
mkdir -p $WC/payload/node_modules/@waggle/agent
cp -r D:/Projects/waggle-os/packages/agent/dist $WC/payload/node_modules/@waggle/agent/dist
echo '{"name":"@waggle/agent","version":"0.0.0","type":"module"}' > $WC/payload/node_modules/@waggle/agent/package.json
# 3. @waggle/core stub
cp -r <thisdir>/stub-core $WC/payload/node_modules/@waggle/core
# 4. @waggle/hive-mind-core — ONLY the DB-free deep modules the stub re-exports.
# Verify logger.js + injection-scanner.js have no further @waggle deps (they don't as of 2026-05-22).
mkdir -p $WC/payload/node_modules/@waggle/hive-mind-core/dist
cp D:/Projects/waggle-os/packages/hive-mind-core/dist/logger.js \
D:/Projects/waggle-os/packages/hive-mind-core/dist/injection-scanner.js \
$WC/payload/node_modules/@waggle/hive-mind-core/dist/
echo '{"name":"@waggle/hive-mind-core","version":"0.0.0","type":"module"}' \
> $WC/payload/node_modules/@waggle/hive-mind-core/package.json
# 5. build — NOTE: local base image is `localhost/gaia2-cli:local` (tag "local", not latest).
# Use the LEGACY builder (DOCKER_BUILDKIT=0): BuildKit treats `localhost/` as a remote
# registry and times out; the legacy builder reads the local image store directly.
DOCKER_BUILDKIT=0 docker build --build-arg GAIA2_CLI_VERSION=local \
-f $WC/Dockerfile -t localhost/gaia2-waggle:latest .
```
## Run (low-N probe — fair vs hermes)
Create `runner/examples/waggle_harness_probe.toml` cloning `waggle_smoke_hermes_n3.toml`
but `image = "localhost/gaia2-waggle:latest"`, `limit = 10`. Then:
```bash
gaia2-runner run-config --config examples/waggle_harness_probe.toml
```
Compare strict + trio-strict (re-judge via `rejudge_user_message.py`) vs the Hermes cell.
## OPEN ITEMS to verify on first build (expect 1-2 iterations)
1. **Socket path** — `entrypoint.sh` exports `WAGGLE_WORKER_SOCK=${HERMES_WORKER_SOCK:-/tmp/hermes-worker.sock}`.
Confirm `containers/hermes/gaia2_adapter.py` actually binds that path (grep its socket default); align if different.
2. **node binary portability** — `COPY --from=node:20-bookworm-slim /usr/local/bin/node` assumes the gaia2-cli
base is glibc/bookworm-compatible. If it's alpine/musl, switch to `node:20-alpine` source or `apt-get install nodejs`.
3. **hive-mind-core deep-module closure** — confirm `logger.js`/`injection-scanner.js` import nothing further
(rerun the spike's isolated import proof against the staged payload before building).
4. **LLM gateway** — worker defaults `BASE_URL=https://openrouter.ai/api/v1` (OpenAI-compat, Sonnet 4.6 via
OpenRouter). Hermes hits Anthropic directly. Same model; document the gateway as a minor confound, or point
both at one litellm proxy for strict parity.

View File

@@ -1,76 +0,0 @@
# gaia2-waggle — GAIA 2 env wrapping WAGGLE's own agent harness (runAgentLoop).
# Models gaia2-hermes, but the agent loop is Waggle's, run as a Node worker.
# Fairness: same single `terminal` tool + same AGENTS.md + same model (Sonnet 4.6);
# only the loop logic differs. Path-A: no better-sqlite3 (2-symbol @waggle/core stub).
#
# BUILD (from gaia2-cli/, after staging waggle payload under containers/waggle/):
# docker build -f containers/waggle/Dockerfile -t localhost/gaia2-waggle:latest .
ARG GAIA2_CLI_IMAGE=localhost/gaia2-cli
ARG GAIA2_CLI_VERSION=latest
# ── setuid wrapper (same as hermes) ────────────────────────────────────
FROM debian:bookworm-slim AS gaia2-exec-builder
RUN echo 'APT::Sandbox::User "root";' | tee -a /etc/apt/apt.conf.d/10sandbox
RUN apt-get update && apt-get install -y --no-install-recommends gcc libc6-dev && rm -rf /var/lib/apt/lists/*
COPY shared/gaia2-exec.c /tmp/gaia2-exec.c
RUN gcc -O2 -o /tmp/gaia2-exec /tmp/gaia2-exec.c && strip /tmp/gaia2-exec
# ── node binary source ──────────────────────────────────────────────────
FROM node:20-bookworm-slim AS nodesrc
# ── Final image ─────────────────────────────────────────────────────────
FROM ${GAIA2_CLI_IMAGE}:${GAIA2_CLI_VERSION}
# Node runtime (self-contained against glibc/libstdc++ present in bookworm base)
COPY --from=nodesrc /usr/local/bin/node /usr/local/bin/node
# Gaia2 privilege-separated user + state (same as hermes)
RUN useradd -r -s /usr/sbin/nologin -d /var/gaia2 gaia2 && mkdir -p /var/gaia2/state
ENV GAIA2_STATE_DIR=/var/gaia2/state
RUN chown -R gaia2:gaia2 /var/gaia2 && chmod 700 /var/gaia2
RUN useradd -m -s /bin/bash -u 1000 agent
# Setuid gaia2 wrapper
COPY --from=gaia2-exec-builder /tmp/gaia2-exec /usr/local/bin/gaia2-exec
RUN chown gaia2:gaia2 /usr/local/bin/gaia2-exec && chmod u+s /usr/local/bin/gaia2-exec
# Restricted agent PATH: app CLIs -> gaia2-exec + curated unix tools + python3 + node
RUN mkdir -p /home/agent/bin && \
for cmd in calendar contacts emails messages chats rent-a-flat city cabs shopping cloud-drive; do \
ln -s /usr/local/bin/gaia2-exec /home/agent/bin/$cmd; done && \
for u in ls pwd date echo cat head tail grep sed awk wc sort mkdir cp mv rm touch chmod jq seq xargs find tee cut tr paste uniq diff base64 basename dirname file mktemp sleep expr test curl; do \
ln -s /usr/bin/$u /home/agent/bin/$u 2>/dev/null || true; done && \
ln -s /usr/local/bin/python3 /home/agent/bin/python3 && \
ln -s /usr/local/bin/node /home/agent/bin/node && \
chown -R agent:agent /home/agent/bin
# Faketime-aware bash/sh wrappers (same as hermes)
RUN printf '#!/usr/bin/bash\nif [ -f /tmp/faketime.rc ]; then\n export LD_PRELOAD="/usr/lib/x86_64-linux-gnu/faketime/libfaketime.so.1"\n export FAKETIME_TIMESTAMP_FILE="/tmp/faketime.rc"\n export FAKETIME_NO_CACHE=1\n unset FAKETIME\nfi\nexec /usr/bin/bash "$@"\n' \
> /home/agent/bin/bash && chmod +x /home/agent/bin/bash && cp /home/agent/bin/bash /home/agent/bin/sh
# Shared adapter + prompt rendering (reuse hermes adapter — same contract)
COPY shared/gaia2_adapter_base.py /opt/gaia2_adapter_base.py
COPY shared/render_agent_prompt.py /opt/render_agent_prompt.py
COPY shared/AGENTS_TEMPLATE.md /opt/AGENTS_TEMPLATE.md
COPY containers/hermes/gaia2_adapter.py /opt/gaia2_adapter.py
# ── Waggle payload: worker + agent dist + 2-symbol @waggle/core stub ─────
# containers/waggle/payload/ is assembled by build-payload.sh before docker build:
# payload/waggle_worker.mjs
# payload/package.json ({"type":"module"})
# payload/node_modules/@waggle/agent/ (built dist + package.json)
# payload/node_modules/@waggle/core/ (stub: index.js + package.json)
# payload/node_modules/@waggle/hive-mind-core/dist/{logger.js,injection-scanner.js,...}
COPY containers/waggle/payload /opt/waggle
COPY containers/waggle/entrypoint.sh /opt/entrypoint.sh
RUN chmod +x /opt/entrypoint.sh
ENV HOME=/home/agent
ENV PATH="/home/agent/bin"
USER root
COPY containers/waggle/gaia2-init-entrypoint.sh /opt/gaia2-init-entrypoint.sh
RUN chmod +x /opt/gaia2-init-entrypoint.sh
ENTRYPOINT []
CMD ["/usr/bin/bash", "/opt/gaia2-init-entrypoint.sh"]

View File

@@ -1,37 +0,0 @@
#!/bin/bash
# entrypoint.sh (gaia2-waggle) — starts the WAGGLE Node worker.
# Adapted from containers/hermes/entrypoint.sh: same lifecycle, launches
# `node /opt/waggle/waggle_worker.mjs` instead of the python hermes worker.
set -o pipefail
LOG=/tmp/entrypoint.log
log() { echo "[$(date +%H:%M:%S)] $*" | tee -a $LOG; }
log "=== entrypoint start (waggle) ==="
if [ -n "${FAKETIME:-}" ]; then
/usr/bin/mkdir -p /dev/shm 2>/dev/null && /usr/bin/chmod 1777 /dev/shm 2>/dev/null || true
log "Faketime enabled: $FAKETIME"
fi
log "User: $(/usr/bin/id -un), PATH: $PATH"
# Socket path MUST match the (reused hermes) gaia2_adapter.py. The adapter's
# default is exported here so the worker's HERMES_WORKER_SOCK fallback matches.
export WAGGLE_WORKER_SOCK="${HERMES_WORKER_SOCK:-/tmp/hermes-worker.sock}"
log "Starting Waggle worker (sock=$WAGGLE_WORKER_SOCK)..."
/usr/local/bin/node /opt/waggle/waggle_worker.mjs >> $LOG 2>&1 &
WORKER_PID=$!
log "Worker PID: $WORKER_PID"
/usr/bin/sleep 2
if ! kill -0 $WORKER_PID 2>/dev/null; then
log "ERROR: Waggle worker exited unexpectedly"
/usr/bin/tail -40 $LOG 2>/dev/null || true
exit 1
fi
log "Waggle worker running"
cleanup() { log "Shutting down..."; kill "$WORKER_PID" 2>/dev/null || true; wait "$WORKER_PID" 2>/dev/null || true; }
trap cleanup EXIT TERM INT
wait $WORKER_PID 2>/dev/null
EXIT_CODE=$?
log "Worker (PID $WORKER_PID) exited with code $EXIT_CODE"
exit $EXIT_CODE

View File

@@ -1,7 +0,0 @@
// Path-A stub for @waggle/core — the ONLY 2 symbols runAgentLoop's runtime chain
// needs (createCoreLogger via turn-context.ts, scanForInjection via injection-scanner.ts).
// Re-exported from hive-mind-core's DB-free deep paths, bypassing the index barrel
// whose `export { MindDB } from './mind/db.js'` eagerly loads better-sqlite3/sqlite-vec.
// Proven (2026-05-22) to import runAgentLoop with zero native deps. See ../spike-waggle-worker/.
export { createCoreLogger } from "@waggle/hive-mind-core/dist/logger.js";
export { scanForInjection } from "@waggle/hive-mind-core/dist/injection-scanner.js";

View File

@@ -1 +0,0 @@
{ "name": "@waggle/core", "version": "0.0.0-gaia2-stub", "type": "module", "main": "index.js" }

View File

@@ -1,199 +0,0 @@
#!/usr/bin/env node
// waggle_worker — bridges Waggle's runAgentLoop to the GAIA 2 ARE adapter.
//
// Mirrors hermes_worker.py's contract so the harness comparison is fair:
// - same single `terminal` tool (GAIA 2 apps invoked as shell cmds via gaia2-exec)
// - same AGENTS.md as system prompt (rendered by gaia2-init-entrypoint.sh)
// - same model (Sonnet 4.6) — the ONLY variable is Waggle's loop logic.
//
// Protocol (JSON lines over Unix socket):
// Worker -> Adapter: {"type":"ready"}
// {"type":"response","run_id","state":"final"|"error","message"}
// Adapter -> Worker: {"type":"message","text","run_id"} / {"type":"interrupt","text"}
import net from "node:net";
import fs from "node:fs";
import os from "node:os";
import { spawn } from "node:child_process";
import { runAgentLoop } from "@waggle/agent/dist/agent-loop.js";
// F2 (2026-05-27): opt-in persona overlay via composePersonaPrompt. Bare-Waggle behavior
// (matching the 2026-05-22 N=40 on-par-with-Hermes baseline) preserved when
// WAGGLE_PERSONA_ID is unset / empty.
import { getPersona, composePersonaPrompt } from "@waggle/agent/dist/personas.js";
const WORKER_SOCK = process.env.WAGGLE_WORKER_SOCK || process.env.HERMES_WORKER_SOCK || "/tmp/waggle-worker.sock";
const AGENTS_MD = `${os.homedir()}/AGENTS.md`;
const MODEL = process.env.MODEL || "claude-sonnet-4-6";
const LITELLM_URL = process.env.BASE_URL || process.env.LITELLM_URL || "https://openrouter.ai/api/v1";
const API_KEY = process.env.API_KEY || process.env.LITELLM_API_KEY || "";
const MAX_TURNS = parseInt(process.env.MAX_ITERATIONS || "90", 10);
const MAX_TOKENS = process.env.MAX_TOKENS ? parseInt(process.env.MAX_TOKENS, 10) : undefined;
function log(...a) { console.log("[waggle-worker]", ...a); }
// Single `terminal` tool: runs a shell command (the agent calls gaia2-exec via AGENTS.md
// instructions). bash -lc so PATH includes gaia2-exec; inherits the agent-user env.
const terminalTool = {
name: "terminal",
description: "Execute a shell command in the sandbox. Use it to call the available GAIA2 app CLIs as described in the system prompt. Returns combined stdout+stderr.",
parameters: {
type: "object",
properties: { command: { type: "string", description: "The shell command to run." } },
required: ["command"],
},
execute: ({ command }) =>
new Promise((resolve) => {
const cmd = typeof command === "string" ? command : String(command ?? "");
const child = spawn("bash", ["-lc", cmd], { cwd: os.homedir(), env: process.env });
let out = "";
const cap = (d) => { out += d.toString(); if (out.length > 200_000) { try { child.kill(); } catch {} } };
child.stdout.on("data", cap);
child.stderr.on("data", cap);
child.on("close", (code) => resolve(out + (code ? `\n[exit ${code}]` : "")));
child.on("error", (e) => resolve(`[terminal error] ${e.message}`));
}),
};
function readAgentsMd() {
try { return fs.readFileSync(AGENTS_MD, "utf8"); }
catch (e) { log("WARN: could not read", AGENTS_MD, e.message); return "You are a helpful agent. Use the terminal tool to complete the task."; }
}
// Fairness with Hermes: Hermes has NO skill-distillation / verification-gate, so for
// "harness is the only variable" to hold, Waggle runs with the same task contract —
// these meta-features OFF. (They also hijacked the final user answer with a skill
// summary in the n1 smoke — a real Waggle bug to fix separately.) Env-overridable so
// an as-shipped (gates ON) variant can be measured later without a rebuild.
const SKILL_GATE = process.env.WAGGLE_SKILL_DISTILLATION_GATE === "1";
const VERIFY_GATE = process.env.WAGGLE_VERIFICATION_GATE === "1";
// F2 (2026-05-27): when WAGGLE_PERSONA_ID is set (e.g. "executive-assistant"), the
// worker composes AGENTS.md (the GAIA 2 tool / app context — REQUIRED) with the
// persona's `systemPrompt` via composePersonaPrompt(). Resolved once at startup so
// every scenario in a run sees the same prompt shape. If the ID is unrecognized we
// log a WARN and fall back to bare AGENTS.md so a typo never silently changes the
// measurement.
const PERSONA_ID = (process.env.WAGGLE_PERSONA_ID || "").trim();
let resolvedPersona = null;
if (PERSONA_ID) {
try {
resolvedPersona = getPersona(PERSONA_ID) ?? null;
if (resolvedPersona) {
log(`F2: persona overlay ON — id=${PERSONA_ID} (${resolvedPersona.name})`);
} else {
log(`F2: WARN — WAGGLE_PERSONA_ID="${PERSONA_ID}" did not resolve to a known persona; falling back to bare AGENTS.md`);
}
} catch (e) {
log(`F2: WARN — getPersona threw (${e?.message || e}); falling back to bare AGENTS.md`);
resolvedPersona = null;
}
}
// F3 (2026-05-27): opt-in output-discipline appendix targeting Qwen-thinking failure
// modes seen in the bare N=160 (Cat 1 verbose multi-paragraph answers + Cat 3
// thinking-mode bleed). Activated by WAGGLE_GAIA2_QWEN_SHAPE=1. Applied as the
// FINAL section of the system prompt so it overrides any persona-introduced framing.
// Composes cleanly with or without the F2 persona overlay (compose-then-append).
const APPLY_QWEN_SHAPE = process.env.WAGGLE_GAIA2_QWEN_SHAPE === "1";
const QWEN_SHAPE_APPENDIX = `
---
## Final Answer Discipline (CRITICAL — read before every send_message_to_user)
Your final \`send_message_to_user\` MUST be the ANSWER, not an analysis. Hard rules:
1. **One short line.** No multi-paragraph response. No headers. No bullet lists. No bold formatting.
2. **No preamble.** Do NOT begin with "Based on my analysis", "Let me", "Now let me", "Here is", "I found", "After analyzing", or "Looking at the data". Just give the value.
3. **No restatement of the question.** The user knows what they asked.
4. **No appended reasoning.** Do NOT include "because…", "since…", "due to…" clauses unless the question explicitly asked for justification.
5. **No "Answer:" / "**Answer:**" prefix.** Just the value itself.
Shape by question type:
- "Which city…?" → \`Stockholm\` (one word, the city name)
- "What is the average…?" → \`45\` (the number, rounded as the question specified)
- "Who is the contact…?" → \`Astrid Lindqvist\` (the name)
- "How many…?" → \`12\` (the count)
- "What time…?" → \`14:30\` (the time)
- Listy "What are the…?" → \`Stockholm, Oslo, Copenhagen\` (comma-separated, no bullets)
Your reasoning ALREADY happened in your \`<think>\` blocks and tool calls. The send_message_to_user is a result delivery, not a reasoning rendition. If you find yourself writing more than ~15 words in send_message_to_user, you are wrong — rewrite shorter.
`;
function buildSystemPrompt() {
const core = readAgentsMd();
let prompt = resolvedPersona ? composePersonaPrompt(core, resolvedPersona) : core;
if (APPLY_QWEN_SHAPE) prompt += QWEN_SHAPE_APPENDIX;
return prompt;
}
async function runOnce(text) {
const res = await runAgentLoop({
litellmUrl: LITELLM_URL,
litellmApiKey: API_KEY,
model: MODEL,
systemPrompt: buildSystemPrompt(),
tools: [terminalTool],
messages: [{ role: "user", content: text }],
maxTurns: MAX_TURNS,
maxTokenBudget: MAX_TOKENS,
stream: true,
skillDistillationGate: SKILL_GATE,
verificationGate: VERIFY_GATE,
});
// AgentResponse — final assistant text. Fall back across likely field names.
return res?.content ?? res?.message ?? res?.finalResponse ?? res?.text ?? "";
}
function connect() {
return new Promise((resolve, reject) => {
const deadline = Date.now() + 60_000;
const attempt = () => {
const sock = net.createConnection(WORKER_SOCK);
sock.once("connect", () => { log("connected", WORKER_SOCK); resolve(sock); });
sock.once("error", () => {
sock.destroy();
if (Date.now() > deadline) reject(new Error(`timeout connecting ${WORKER_SOCK}`));
else setTimeout(attempt, 200);
});
};
attempt();
});
}
function send(sock, obj) { sock.write(JSON.stringify(obj) + "\n"); }
async function main() {
const sock = await connect();
send(sock, { type: "ready" });
let buf = "";
let busy = false;
sock.on("data", async (chunk) => {
buf += chunk.toString();
let i;
while ((i = buf.indexOf("\n")) >= 0) {
const line = buf.slice(0, i).trim();
buf = buf.slice(i + 1);
if (!line) continue;
let msg;
try { msg = JSON.parse(line); } catch { continue; }
if (msg.type === "interrupt") { log("interrupt (ignored — single-turn scenarios)"); continue; }
if (msg.type !== "message") { log("unknown msg type", msg.type); continue; }
if (busy) { log("WARN: message while busy"); }
busy = true;
const runId = msg.run_id || "unknown";
try {
log(`run ${runId}:`, (msg.text || "").slice(0, 100));
const message = await runOnce(msg.text || "");
log(`done ${runId}:`, String(message).slice(0, 100));
send(sock, { type: "response", run_id: runId, state: "final", message: String(message) });
} catch (e) {
log(`error ${runId}:`, e?.message || e);
send(sock, { type: "response", run_id: runId, state: "error", message: `Error: ${e?.message || e}`, errorMessage: String(e?.message || e) });
} finally { busy = false; }
}
});
sock.on("close", () => { log("socket closed"); process.exit(0); });
}
main().catch((e) => { console.error("[waggle-worker] fatal", e); process.exit(1); });

View File

@@ -1,117 +0,0 @@
# GEPA — Tier 2 Prompt-Shapes Evolution
GEPA (Agrawal et al. — Genetic Evolutionary Prompt Adaptation) harness for
evolving the per-model prompt-shape templates in `packages/agent/src/prompt-shapes/`.
**Faza 1 (current):** proof-of-concept pilot. $100 cap. H3 cell only. 5 shapes ×
3 candidates × 2 generations × N=8 evaluations.
**Faza 2 (gated on Faza 1 PASS):** expansion to H2 + H4 cells, scale N to 20+,
3-5 generations, separate ratification + brief.
## Authority chain
- PM brief: `D:/Projects/PM-Waggle-OS/briefs/2026-04-28-cc4-gepa-tier2-evolution-faza1-brief.md`
- Pre-flight report: `D:/Projects/PM-Waggle-OS/briefs/2026-04-28-cc4-faza1-preflight-report.md`
- Amendment 1: `D:/Projects/PM-Waggle-OS/briefs/2026-04-28-cc4-faza1-amendment-1.md`
- Amendment 2: `D:/Projects/PM-Waggle-OS/briefs/2026-04-28-cc4-faza1-amendment-2.md`
- Launch decision (LOCK): `D:/Projects/PM-Waggle-OS/decisions/2026-04-28-gepa-faza1-launch.md`
- Manifest v7 (Amendment 2 supplemented): `benchmarks/preregistration/manifest-v7-gepa-faza1.yaml`
- SHA: `583712dde139ffc87fb1ab21643f68d52c56469ded9e8090a624980b05969beb`
## Substrate freeze
- Branch: `feature/c3-v3-wrapper`
- Commit: `c9bda3d6dd4c0a4f715e09f3757a96d01ff01cd7` (Phase 4.7)
- Worktree: `D:/Projects/waggle-os-faza1-wt` (detached HEAD c9bda3d, race-condition guarded)
## Module map (`src/faza-1/`)
| Module | Purpose | Key references |
|---|---|---|
| `types.ts` | Shared types + `QWEN_TARGETED_SHAPES` / `NON_QWEN_SHAPES` partition | Amendment 2 §3 |
| `fitness.ts` | Per-shape fitness function with retrieval engagement bonus fork | Amendment 2 §3 bands |
| `acceptance.ts` | §F + §F.5 verdict per candidate (false-positive guard) | Launch decision §F |
| `mutation-validator.ts` | Cell-semantic preservation audit (boundary SHAs) | Launch decision §A.4 + §B |
| `kappa-audit.ts` | Drift band detection vs canonical 0.7878 ± 0.05 | Launch decision §F.3 |
| `cost-tracker.ts` | Super-linear governance + halt triggers | Launch decision §A.7 + §D |
| `selection.ts` | Top-1-per-shape + run-aggregate §F.2 | Launch decision §F.2 |
| `mutation-oracle-fork.ts` | Qwen vs non-Qwen template routing | Amendment 2 §4 |
## Boundary anchors (LOCKED at substrate c9bda3d)
| Anchor | SHA-256 | Bytes |
|---|---|---|
| `packages/agent/src/prompt-shapes/types.ts` (whole file) | `1a9fa329e4b66ed9f0abe8bc22cbbf0124e0c879e1e78ec806d557cab25bc94d` | — |
| `MULTI_STEP_ACTION_CONTRACT` constant body | `70a1701dfa126f8dc1df9c116f0a8469da005821ecadc59d9b8f348568e755ba` | 252 |
## Baseline shape SHAs (LOCKED)
| Shape | SHA-256 |
|---|---|
| `claude.ts` | `cbaf0c37b067b025a1fe97f2feeec11fae4070a8b3fcfaad1da8775dda451cc0` |
| `qwen-thinking.ts` | `848a4e4917baa5c7bbcc3bb35fb8cb4b4ac8f0ab537243f14cbef3a99197aacb` |
| `qwen-non-thinking.ts` | `35be379be9a8caafc2c419e32da5f63f92fc83f6f6d70d9df76029c1e8584572` |
| `gpt.ts` | `5dc6d750d52a68feb9d37ad8384b2bcd59d70962066122ff086b0e5888413576` |
| `generic-simple.ts` | `81189817f560e26a69394248d8bd9089cae72c7d40825323e2b7407e36026172` |
## Per-shape fitness fork (Amendment 2 §3)
```
Qwen-targeted (qwen-thinking, qwen-non-thinking):
fitness = trio_strict_pass_rate + retrieval_engagement_bonus − cost_penalty
retrieval_engagement_bonus =
+0.05 if mean retrieval_calls per task ≥ 2.0 (Opus parity proxy)
0.00 if mean retrieval_calls per task ∈ [1.5, 2.0)
−0.05 if mean retrieval_calls per task < 1.5 (Qwen baseline penalty)
Non-Qwen (claude, gpt, generic-simple):
fitness = trio_strict_pass_rate − cost_penalty
```
## §F.5 false-positive guard (Amendment 2 §5)
If best Qwen-shape candidate achieves `+5pp trio_strict delta WITHOUT closing
retrieval engagement gap (mean retrieval_calls < 1.5)`, candidate is REJECTED
as false-positive evolution.
## Tests
110 tests across 7 files (`tests/faza-1/`). Run via `npm test` from repo root
(vitest auto-discovers `benchmarks/*/tests/**/*.test.ts`).
| Test file | Tests |
|---|---|
| `fitness.test.ts` | 30 (5 mandatory boundary cases + 5 routing + cost penalty + invariants) |
| `acceptance.test.ts` | 17 (§F.5 mandatory FAIL + PASS-path + boundaries) |
| `mutation-validator.test.ts` | 19 (SHA boundary + baseline + metadata + imports + Gen 1 mutation) |
| `kappa-audit.test.ts` | 22 (drift band + Cohen's κ + audit log + canonical anchor) |
| `cost-tracker.test.ts` | 23 (HARD_CAP + INTERNAL_HALT + SUPER_LINEAR + audit cadence) |
| `selection.test.ts` | (top-1 + Qwen retrieval ranking + §F.2 boundaries + missing-baseline errors) |
| `mutation-oracle-fork.test.ts` | (shape classification + template paths + placeholder substitution) |
## Halt-and-PM checkpoints
| # | Checkpoint | Cumulative | Trigger |
|---|---|---|---|
| 1 | Pre-A | ~$5 | Post 50-instance corpus + 5-instance spot-audit |
| 2 | A | ~$25 | Post NULL-baseline 5 shapes × 8 instances |
| 3 | B | ~$50-65 | Mid-Gen 1 (after 30 evaluations) |
| 4 | C | ~$100 | Post held-out validation, Faza 1 verdict |
See launch decision §E for binding details.
## Acceptance criteria summary
Per launch decision §F (binding) — all 4 conditions must hold AND no §F.5 trigger:
1. Best GEPA candidate beats NULL-baseline by ≥+5pp on `trio_strict_pass` rate
(`trio_mean ≥ 4.0` per Ask B); for Qwen-targeted shapes, additionally
`mean retrieval_calls per task ≥ 1.7` (50% gap closure).
2. ≥3/5 shapes show positive trio_strict delta vs NULL-baseline.
3. Trio judge κ remains within `±0.05` of canonical `0.7878`
(drift band `[0.7378, 0.8378]`).
4. Zero cell semantic violations detected by `mutation-validator`.
5. **§F.5 false-positive guard:** Qwen candidate with `+5pp trio_strict delta`
AND `mean retrieval_calls < 1.5` → REJECTED.

View File

@@ -1,54 +0,0 @@
# GEPA Mutation Oracle Prompt — Non-Qwen (Claude / GPT / Generic-Simple)
**Template class:** ###TEMPLATE_CLASS###
**Shape under mutation:** ###SHAPE_NAME###
---
You are an expert prompt engineer. Your job is to produce a single mutated variant of the prompt-shape file below, evolving the *reasoning scaffold* and *planning step structure* while preserving exact cell semantics.
## Context (non-Qwen branch)
This shape (###SHAPE_NAME###) does **not** exhibit the retrieval-engagement gap that Phase 4.5 surfaced for Qwen. Standard mutation guidance per brief §3.3 applies — no Qwen-specific anti-premature-finalization scaffolding required.
Apply general scaffold improvements: clearer reasoning structure, better chain-of-thought triggers, more explicit planning steps where appropriate for the model class.
## Failure modes from Phase 4.3 (top-3 T2 categories for this shape)
###FAILURE_MODE_SUMMARY###
## Mutation guidance (binding)
Modify only:
- `systemPrompt(input)` method body (string-building only)
- `soloUserPrompt(input)` method body
- `multiStepKickoffUserPrompt(input)` method body
- `retrievalInjectionUserPrompt(input)` method body
- `metadata.evidence_link` (you MUST update to point to GEPA Gen 1 results: `benchmarks/results/gepa-faza1/oracle/mutation-prompt-template-non-qwen.md`)
Standard mutation directions:
1. **Evolve reasoning scaffold** — restructure the implicit reasoning prompts (e.g., "think step by step" variants, planning bullets)
2. **Refine chain-of-thought triggers** — adjust where the model is prompted to articulate its reasoning before producing output
3. **Improve planning step structure** — make multi-step task decomposition more explicit
4. **Preserve all cell semantic boundaries** — see prohibitions below
## Prohibited modifications (cell semantic LOCK)
- **DO NOT modify** `MULTI_STEP_ACTION_CONTRACT` — this constant lives in `packages/agent/src/prompt-shapes/types.ts` and its bytes are SHA-pinned
- **DO NOT modify** `metadata.{description,modelClass,defaultThinking,defaultMaxTokens}`
- **DO NOT modify** the imports block
- **DO NOT modify** the file structure (must remain a TypeScript module with a single PromptShape export)
- **DO NOT modify** the JSON action contract
- **DO NOT modify** task framing (persona/question/materials section labels)
Mutations violating any of the above will be REJECTED by the mutation validator. Two consecutive invalid mutations trigger halt-and-PM per brief §5.
## Baseline shape file (the input to your mutation)
```typescript
###BASELINE_SHAPE_CONTENT###
```
## Output format
Output ONLY the mutated TypeScript file content. No prose, no commentary, no code fences. The file should be a drop-in replacement for the baseline shape file.

View File

@@ -1,64 +0,0 @@
# GEPA Mutation Oracle Prompt — Qwen-Targeted Shape
**Template class:** ###TEMPLATE_CLASS###
**Shape under mutation:** ###SHAPE_NAME###
---
You are an expert prompt engineer. Your job is to produce a single mutated variant of the prompt-shape file below, evolving the *reasoning scaffold* and *retrieval-engagement guidance* while preserving exact cell semantics.
## Empirical context (binding)
Phase 4.5 audit empirically established that Qwen 3.6 35B-A3B (under both `thinking-on` and `thinking-off` configurations) exhibits a **retrieval-engagement gap** vs Opus 4.7 on the same `MULTI_STEP_ACTION_CONTRACT` tool surface:
- Qwen retrieves **1.33 times per task** on average
- Opus retrieves **2.33 times per task** on average
- Qwen finalizes prematurely; Opus exhausts maxSteps in 67% of retrieval runs
- Resulting H4 trio_mean delta: Qwen scores **−0.65 below** Opus on every task
The mutation surface (prompt-shape body) is the lever to address this. **Your mutation must encourage more aggressive multi-turn retrieval engagement on Qwen.**
## Failure modes from Phase 4.3 (top-3 T2 categories for this shape)
###FAILURE_MODE_SUMMARY###
## Mutation guidance (binding)
Modify only:
- `systemPrompt(input)` method body (string-building only)
- `soloUserPrompt(input)` method body
- `multiStepKickoffUserPrompt(input)` method body
- `retrievalInjectionUserPrompt(input)` method body
- `metadata.evidence_link` (you MUST update to point to GEPA Gen 1 results: `benchmarks/results/gepa-faza1/oracle/mutation-prompt-template-qwen.md`)
Apply Qwen-specific scaffolding:
1. **Emphasize multi-turn retrieval** over single-shot retrieval — make repeated retrieval feel mandatory, not optional
2. **Discourage premature finalization** with explicit phrasing like:
- "Continue retrieving until you have evidence from at least 2 distinct queries before finalizing"
- "Single retrieval is rarely sufficient for synthesis tasks"
3. **Encourage iterative refinement** — each retrieval query should build on prior turn's results
4. **Add anti-premature-finalization scaffolding** in the multiStepKickoff or systemPrompt:
- "Before finalizing, ask: what gap in evidence remains? Issue another retrieval if any gap exists."
- "A complete answer requires triangulation from multiple retrievals"
5. **Preserve all cell semantic boundaries** — see prohibitions below
## Prohibited modifications (cell semantic LOCK)
- **DO NOT modify** `MULTI_STEP_ACTION_CONTRACT` — this constant lives in `packages/agent/src/prompt-shapes/types.ts` and its bytes are SHA-pinned
- **DO NOT modify** `metadata.{description,modelClass,defaultThinking,defaultMaxTokens}`
- **DO NOT modify** the imports block
- **DO NOT modify** the file structure (must remain a TypeScript module with a single PromptShape export)
- **DO NOT modify** the JSON action contract (the `{"action": "retrieve", ...}` / `{"action": "finalize", ...}` shape)
- **DO NOT modify** task framing (persona/question/materials section labels — the template rendering structure)
Mutations that violate any of the above will be REJECTED by the mutation validator (`mutation-validator.ts`), consume one of two retry tolerances, and trigger halt-and-PM if exceeded twice consecutively per brief §5.
## Baseline shape file (the input to your mutation)
```typescript
###BASELINE_SHAPE_CONTENT###
```
## Output format
Output ONLY the mutated TypeScript file content. No prose, no commentary, no code fences. The file should be a drop-in replacement for the baseline shape file.

View File

@@ -1,142 +0,0 @@
"""Compute Checkpoint A v2 aggregates + classify per pre-registered bands."""
import json, sys, io, math, os
sys.stdout = io.TextIOWrapper(sys.stdout.buffer, encoding='utf-8', errors='replace')
JSONL = 'D:/Projects/waggle-os-faza1-wt/benchmarks/results/gepa-faza1/null-baseline/null-baseline-eval.jsonl'
with open(JSONL, encoding='utf-8') as f:
records = [json.loads(l) for l in f if l.strip()]
print(f'TOTAL records: {len(records)}')
total_cost = sum(r['evalCostUsd'] for r in records)
total_subject = sum(r['candidateCostUsd'] for r in records)
total_judge = sum(r['judges']['judgeCostTotal'] for r in records)
print(f'cumulative cost: ${total_cost:.4f} (subject ${total_subject:.4f} + judge ${total_judge:.4f})')
SHAPES = ['claude','qwen-thinking','qwen-non-thinking','gpt','generic-simple']
shape_aggs = {}
for s in SHAPES:
rs = [r for r in records if r['shape'] == s]
if not rs: continue
n = len(rs)
pii = sum(1 for r in rs if r['judges']['trioStrictPassII'])
pi = sum(1 for r in rs if r['judges']['trioStrictPassI'])
mt = sum(r['judges']['trioMean'] for r in rs)/n
mr = sum(r['retrievalCalls'] for r in rs)/n
mc = sum(r['evalCostUsd'] for r in rs)/n
le = sum(1 for r in rs if r['loopExhausted'])/n
ms = sum(r['stepsTaken'] for r in rs)/n
rate = pii/n
z = 1.96
denom = 1 + z*z/n
center = (rate + z*z/(2*n)) / denom
margin = z * math.sqrt(rate*(1-rate)/n + z*z/(4*n*n)) / denom
ci_low = max(0, center - margin)
ci_hi = min(1, center + margin)
shape_aggs[s] = dict(n=n, pii=pii, pi=pi, rate=rate, mt=mt, mr=mr, mc=mc, le=le, ms=ms, ci_low=ci_low, ci_hi=ci_hi)
print('\n=== PER-SHAPE AGGREGATES (REAL) ===')
for s in SHAPES:
a = shape_aggs[s]
print(f' {s:<22} pass_II={a["pii"]}/{a["n"]} ({a["rate"]:.1%}) pass_I={a["pi"]}/{a["n"]} trio={a["mt"]:.3f} retr={a["mr"]:.2f} cost=${a["mc"]:.4f} CI95=[{a["ci_low"]:.3f},{a["ci_hi"]:.3f}]')
artifactual = {'claude': 0.50, 'qwen-thinking': 1.0, 'qwen-non-thinking': 0.75, 'gpt': 0.875, 'generic-simple': 0.875}
print('\n=== DELTA vs artifactual ===')
deltas = {}
for s in SHAPES:
d = (shape_aggs[s]['rate'] - artifactual[s]) * 100
deltas[s] = d
print(f' {s:<22} artifactual={artifactual[s]:.1%} real={shape_aggs[s]["rate"]:.1%} delta={d:+.1f}pp')
print('\n=== PRE-REGISTERED BAND CLASSIFICATION (LOCKED §C) ===')
expected_bounds = {'claude':(35,65),'qwen-thinking':(85,100),'qwen-non-thinking':(60,90),'gpt':(73,100),'generic-simple':(73,100)}
in_per_shape_band = True
for s in SHAPES:
pct = shape_aggs[s]['rate'] * 100
lo, hi = expected_bounds[s]
ok = lo <= pct <= hi
print(f' {s:<22} real {pct:>5.1f}% expected band [{lo}, {hi}] {"IN" if ok else "OUT"}')
if not ok: in_per_shape_band = False
max_abs = max(abs(d) for d in deltas.values())
sign_flips = sum(1 for s in SHAPES if (artifactual[s] >= 0.5) != (shape_aggs[s]['rate'] >= 0.5))
all_neg = all(d <= 0 for d in deltas.values())
all_pos = all(d >= 0 for d in deltas.values())
uniform_shift_ok = (all_neg or all_pos) and (max(deltas.values()) - min(deltas.values()) <= 25)
print('\n=== RAW AGREEMENT + KAPPA ===')
def per_judge(model, t=4.0):
return [next(j['mean']>=t for j in r['judges']['records'] if j['judge_model']==model) for r in records]
opus = per_judge('claude-opus-4-7')
gpt_j = per_judge('gpt-5.4')
mm = per_judge('minimax-m27-via-openrouter')
def raw_agree(a,b): return sum(1 for x,y in zip(a,b) if x==y)/len(a)
def kappa(a,b):
n=len(a); cc=sum(1 for x,y in zip(a,b) if x and y); ii=sum(1 for x,y in zip(a,b) if not x and not y)
ci=sum(1 for x,y in zip(a,b) if x and not y); ic=sum(1 for x,y in zip(a,b) if not x and y)
po=(cc+ii)/n; pa=(cc+ci)/n; pb=(cc+ic)/n; pe=pa*pb+(1-pa)*(1-pb)
return (po-pe)/(1-pe) if pe<1 else 1.0
ra={'opus_gpt':raw_agree(opus,gpt_j),'opus_minimax':raw_agree(opus,mm),'gpt_minimax':raw_agree(gpt_j,mm)}
k={'opus_gpt':kappa(opus,gpt_j),'opus_minimax':kappa(opus,mm),'gpt_minimax':kappa(gpt_j,mm)}
for pair in ['opus_gpt','opus_minimax','gpt_minimax']:
print(f' {pair:<16} raw={ra[pair]:.1%} kappa={k[pair]:+.3f}')
min_ra = min(ra.values())
min_k = min(k.values())
print(f' MIN raw agreement: {min_ra:.1%} (threshold 65%: {"PASS" if min_ra>=0.65 else "FAIL"})')
print(f' MIN kappa: {min_k:+.3f}')
print('\nPer-judge pass rates at 4.0:')
print(f' Opus: {sum(opus)}/{len(opus)} = {sum(opus)/len(opus):.1%}')
print(f' GPT: {sum(gpt_j)}/{len(gpt_j)} = {sum(gpt_j)/len(gpt_j):.1%}')
print(f' MiniMax: {sum(mm)}/{len(mm)} = {sum(mm)/len(mm):.1%}')
artif_min_ra = 0.70
ra_collapse_pp = (artif_min_ra - min_ra) * 100
anomalous_max = max_abs > 30
anomalous_flips = sign_flips > 2
anomalous_ra = ra_collapse_pp > 20
print(f'\n=== ANOMALY CHECK ===')
print(f' max |delta| > 30pp: {anomalous_max} (max={max_abs:.1f}pp)')
print(f' sign flips > 2: {anomalous_flips} (count={sign_flips})')
print(f' raw agreement collapse > 20pp: {anomalous_ra} (artifactual {artif_min_ra:.1%} -> real {min_ra:.1%}, delta {ra_collapse_pp:+.1f}pp)')
is_anomalous = anomalous_max or anomalous_flips or anomalous_ra
is_expected = (in_per_shape_band or uniform_shift_ok) and not is_anomalous
print(f'\nPER-SHAPE BANDS: {"all IN" if in_per_shape_band else "some OUT"}')
print(f'UNIFORM SHIFT: ok={uniform_shift_ok} (all_neg={all_neg}, all_pos={all_pos}, spread={max(deltas.values())-min(deltas.values()):.1f}pp)')
print(f'EXPECTED met: {is_expected}')
print(f'ANOMALOUS triggered: {is_anomalous}')
print(f'CLASSIFICATION: {"EXPECTED -> Gen 1 GO" if is_expected else "ANOMALOUS -> INVESTIGATE"}')
real_per_eval = total_cost / len(records)
gen1_proj = 5 * 3 * 8 * real_per_eval
artif_per_eval = 0.124
sens_pct = (real_per_eval - artif_per_eval) / artif_per_eval * 100
print(f'\n=== COST SENSITIVITY ===')
print(f' artifactual: ${artif_per_eval:.4f}/eval real: ${real_per_eval:.4f}/eval delta: {sens_pct:+.1f}%')
print(f' Gen 1 proj: ${gen1_proj:.4f} $78 halt: {"PASS" if gen1_proj<=78 else "HALT"}')
print('\n=== F-SATURATED PER-SHAPE ===')
n_qual = 0
for s in SHAPES:
a = shape_aggs[s]
qual = a['ci_low'] >= 0.88
if qual: n_qual += 1
print(f' {s:<22} pii={a["pii"]}/{a["n"]} CI_low={a["ci_low"]:.3f} >=0.88? {"Y" if qual else "N"} policy={"F-sat" if qual else "F.1 (>=+5pp)"}')
if n_qual == 0: decision = 'GLOBAL: revoke F-saturated, apply F.1 to all 5'
elif n_qual == 5: decision = 'GLOBAL: re-instate F-saturated for all 5'
else: decision = f'MIXED: {n_qual}/5 qualify (per-shape policy, pre-authorized)'
print(f'Decision: {decision}')
out = {
'shape_aggregates': shape_aggs, 'deltas_pp': deltas, 'classification': 'EXPECTED' if is_expected else 'ANOMALOUS',
'raw_agreement': ra, 'min_raw_agreement': min_ra, 'kappa': k, 'min_kappa': min_k,
'per_judge_pass_rate': {'opus': sum(opus)/len(opus), 'gpt': sum(gpt_j)/len(gpt_j), 'minimax': sum(mm)/len(mm)},
'cost': {'total': total_cost, 'per_eval': real_per_eval, 'sensitivity_pct': sens_pct, 'gen1_projected': gen1_proj},
'F_saturated_n_qualifying': n_qual, 'F_saturated_decision': decision,
}
with open('D:/Projects/waggle-os-faza1-wt/benchmarks/results/gepa-faza1/null-baseline/checkpoint-a-v2-aggregates.json', 'w') as f:
json.dump(out, f, indent=2)
print('\n=> aggregates JSON written')

View File

@@ -1,194 +0,0 @@
#!/usr/bin/env tsx
/**
* GEPA Faza 1 — final κ_trio recompute on combined 135 evals.
*
* Per launch decision §F condition_3 + manifest v7 §canonical_kappa_anchor.
*
* Combines:
* - Gen 1 eval JSONL (120 records)
* - Checkpoint C eval JSONL (15 records)
*
* For each eval, derives per-judge binary pass (mean ≥ 4.0) and computes
* pairwise Cohen's κ for the 3 judge pairs (Opus↔GPT, Opus↔MiniMax,
* GPT↔MiniMax). Reports κ_trio (= min of pairs) + audit verdict.
*
* Per Amendment 5 §judge_metric_design: also reports raw agreement rate
* (primary metric for synthesis Likert).
*/
import * as fs from 'node:fs';
import * as path from 'node:path';
import { fileURLToPath } from 'node:url';
import {
auditKappa,
computeCohensKappa,
CANONICAL_KAPPA,
KAPPA_DRIFT_BAND_LOW,
KAPPA_DRIFT_BAND_HIGH,
} from '../../src/faza-1/kappa-audit.js';
const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);
const REPO_ROOT = path.resolve(__dirname, '../../../..');
const GEN_1_JSONL = path.join(REPO_ROOT, 'benchmarks/results/gepa-faza1/gen-1/gen-1-eval.jsonl');
const CHECKPOINT_C_JSONL = path.join(REPO_ROOT, 'benchmarks/results/gepa-faza1/checkpoint-c/checkpoint-c-eval.jsonl');
const OUT_JSON = path.join(REPO_ROOT, 'benchmarks/results/gepa-faza1/checkpoint-c/final-kappa-audit.json');
const PASS_THRESHOLD = 4.0;
const JUDGE_ORDER = ['claude-opus-4-7', 'gpt-5.4', 'minimax-m27-via-openrouter'] as const;
interface JudgeRec { judge_model: string; mean: number; raw: any }
interface EvalRec {
candidateId: string;
judgeRecords: JudgeRec[];
evalSet?: 'held-out';
}
function loadJsonl(filepath: string): EvalRec[] {
if (!fs.existsSync(filepath)) {
console.error(`MISSING: ${filepath}`);
return [];
}
return fs.readFileSync(filepath, 'utf-8').trim().split(/\n+/).filter(Boolean).map(l => JSON.parse(l));
}
function judgePassPerEval(rec: EvalRec): { opus: boolean | null; gpt: boolean | null; minimax: boolean | null } {
const findBy = (model: string) => rec.judgeRecords.find(j => j.judge_model === model);
const opus = findBy('claude-opus-4-7');
const gpt = findBy('gpt-5.4');
const minimax = findBy('minimax-m27-via-openrouter');
return {
opus: opus && opus.mean > 0 ? opus.mean >= PASS_THRESHOLD : null,
gpt: gpt && gpt.mean > 0 ? gpt.mean >= PASS_THRESHOLD : null,
minimax: minimax && minimax.mean > 0 ? minimax.mean >= PASS_THRESHOLD : null,
};
}
interface ConfusionMatrix {
bothCorrect: number;
bothIncorrect: number;
firstCorrectSecondIncorrect: number;
firstIncorrectSecondCorrect: number;
}
function buildConfusion(pairs: Array<[boolean, boolean]>): ConfusionMatrix {
const m: ConfusionMatrix = { bothCorrect: 0, bothIncorrect: 0, firstCorrectSecondIncorrect: 0, firstIncorrectSecondCorrect: 0 };
for (const [a, b] of pairs) {
if (a && b) m.bothCorrect++;
else if (!a && !b) m.bothIncorrect++;
else if (a && !b) m.firstCorrectSecondIncorrect++;
else m.firstIncorrectSecondCorrect++;
}
return m;
}
function rawAgreement(pairs: Array<[boolean, boolean]>): number {
if (pairs.length === 0) return NaN;
let agree = 0;
for (const [a, b] of pairs) if (a === b) agree++;
return agree / pairs.length;
}
function main() {
const gen1 = loadJsonl(GEN_1_JSONL);
const cpc = loadJsonl(CHECKPOINT_C_JSONL);
const combined = [...gen1, ...cpc];
console.log(`Loaded: gen-1=${gen1.length}, checkpoint-c=${cpc.length}, combined=${combined.length}`);
// Build per-judge pass arrays (skip evals where any judge failed parse)
const opusGpt: Array<[boolean, boolean]> = [];
const opusMinimax: Array<[boolean, boolean]> = [];
const gptMinimax: Array<[boolean, boolean]> = [];
let droppedDueToFailedJudge = 0;
for (const rec of combined) {
const v = judgePassPerEval(rec);
if (v.opus === null || v.gpt === null || v.minimax === null) {
droppedDueToFailedJudge++;
continue;
}
opusGpt.push([v.opus, v.gpt]);
opusMinimax.push([v.opus, v.minimax]);
gptMinimax.push([v.gpt, v.minimax]);
}
console.log(`Effective N (after dropping failed-judge evals): ${opusGpt.length} (dropped: ${droppedDueToFailedJudge})`);
const kOpusGpt = computeCohensKappa(buildConfusion(opusGpt));
const kOpusMinimax = computeCohensKappa(buildConfusion(opusMinimax));
const kGptMinimax = computeCohensKappa(buildConfusion(gptMinimax));
const audit = auditKappa({ kOpusGpt, kOpusMinimax, kGptMinimax });
const rawOG = rawAgreement(opusGpt);
const rawOM = rawAgreement(opusMinimax);
const rawGM = rawAgreement(gptMinimax);
const rawMin = Math.min(rawOG, rawOM, rawGM);
const passRates = {
opus: opusGpt.filter(p => p[0]).length / opusGpt.length,
gpt: opusGpt.filter(p => p[1]).length / opusGpt.length,
minimax: opusMinimax.filter(p => p[1]).length / opusMinimax.length,
};
const result = {
generated_at: new Date().toISOString(),
inputs: {
gen_1_jsonl_records: gen1.length,
checkpoint_c_jsonl_records: cpc.length,
combined_records: combined.length,
effective_n_after_judge_failures: opusGpt.length,
dropped_due_to_failed_judge: droppedDueToFailedJudge,
threshold: PASS_THRESHOLD,
},
pairwise_kappa: {
opus_gpt: kOpusGpt,
opus_minimax: kOpusMinimax,
gpt_minimax: kGptMinimax,
},
pairwise_raw_agreement: {
opus_gpt: rawOG,
opus_minimax: rawOM,
gpt_minimax: rawGM,
min: rawMin,
},
per_judge_pass_rates: passRates,
canonical_kappa: CANONICAL_KAPPA,
drift_band: { low: KAPPA_DRIFT_BAND_LOW, high: KAPPA_DRIFT_BAND_HIGH },
audit: audit,
};
fs.writeFileSync(OUT_JSON, JSON.stringify(result, null, 2));
console.log('');
console.log('━'.repeat(76));
console.log(' Faza 1 final κ audit on combined 135 evals');
console.log('━'.repeat(76));
console.log(` N effective : ${opusGpt.length}`);
console.log(` Per-judge pass rate@4.0 : Opus=${(passRates.opus*100).toFixed(1)}% GPT=${(passRates.gpt*100).toFixed(1)}% MiniMax=${(passRates.minimax*100).toFixed(1)}%`);
console.log('');
console.log(' PAIRWISE Cohen\'s κ:');
console.log(` Opus↔GPT : ${kOpusGpt.toFixed(4)}`);
console.log(` Opus↔MiniMax : ${kOpusMinimax.toFixed(4)}`);
console.log(` GPT↔MiniMax : ${kGptMinimax.toFixed(4)}`);
console.log('');
console.log(' PAIRWISE raw agreement (Amendment 5 PRIMARY for synthesis Likert):');
console.log(` Opus↔GPT : ${(rawOG*100).toFixed(1)}%`);
console.log(` Opus↔MiniMax : ${(rawOM*100).toFixed(1)}%`);
console.log(` GPT↔MiniMax : ${(rawGM*100).toFixed(1)}%`);
console.log(` MIN raw : ${(rawMin*100).toFixed(1)}%`);
console.log('');
console.log(` κ_conservative_trio : ${audit.kConservativeTrio.toFixed(4)}`);
console.log(` Canonical κ : ${CANONICAL_KAPPA.toFixed(4)}`);
console.log(` Drift band : [${KAPPA_DRIFT_BAND_LOW.toFixed(4)}, ${KAPPA_DRIFT_BAND_HIGH.toFixed(4)}]`);
console.log(` §F.3 verdict : ${audit.verdict}`);
console.log(` v6 policy floor (≥0.70): ${audit.v6PolicyFloorPass ? 'PASS' : 'FAIL'}`);
console.log(` Amendment 5 raw 65% min: ${rawMin >= 0.65 ? 'PASS' : 'FAIL'} (observed ${(rawMin*100).toFixed(1)}%)`);
console.log('');
console.log(` Audit log: ${audit.auditLogLine}`);
console.log('');
console.log(`Wrote: ${OUT_JSON}`);
}
main();

View File

@@ -1,488 +0,0 @@
#!/usr/bin/env tsx
/**
* GEPA Faza 1 — H3 NorthLane CFO synthesis corpus generator.
*
* Per launch decision §G step 4 + manifest v7 §corpus_design + Amendment 1 Ask A Option C.
*
* Generates 50 stratified synthesis-task instances via Opus 4.7 oracle.
*
* Cost projection: ~$5 (50 × $0.10/instance avg).
* Halt threshold: $7 (40% buffer per manifest v7 §corpus_design.expected_generation_cost_usd).
*
* Output: benchmarks/results/gepa-faza1/corpus/h3-northlane-cfo-50-instances.jsonl
*
* Usage:
* npx tsx benchmarks/gepa/scripts/faza-1/generate-h3-corpus.ts --dry-run
* # No LLM call. Validates stratification + prompt build only.
*
* npx tsx benchmarks/gepa/scripts/faza-1/generate-h3-corpus.ts --probe
* # Single instance (cell index 0). ~$0.10. Validates LiteLLM connection + JSON parse.
*
* npx tsx benchmarks/gepa/scripts/faza-1/generate-h3-corpus.ts --all
* # All 50 instances. ~$5. Halt at $7. Spot-audit + Pre-A report afterwards.
*
* Authority: launch decision LOCK at decisions/2026-04-28-gepa-faza1-launch.md (PM-Waggle-OS)
*/
import * as fs from 'node:fs';
import * as fsp from 'node:fs/promises';
import * as path from 'node:path';
import * as crypto from 'node:crypto';
import { fileURLToPath } from 'node:url';
import {
TOTAL_INSTANCES,
STRATIFICATION_SEED,
type CorpusInstance,
type StratificationCell,
listStratificationCells,
buildInstanceId,
validateInstance,
runSpotAudit,
corpusSha256,
} from '../../src/faza-1/corpus.js';
import { buildCorpusInstancePrompt } from '../../src/faza-1/corpus-prompt.js';
const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);
const REPO_ROOT = path.resolve(__dirname, '../../../..');
const OUT_DIR = path.join(REPO_ROOT, 'benchmarks/results/gepa-faza1/corpus');
const OUT_JSONL = path.join(OUT_DIR, 'h3-northlane-cfo-50-instances.jsonl');
const RUN_LOG = path.join(OUT_DIR, 'generation-run.log');
const SPOT_AUDIT_REPORT = path.join(OUT_DIR, 'h3-spot-audit-pre-a-report.md');
// ── LLM config (per launch decision §A.1 inheritance) ─────────────────────
const LITELLM_URL = process.env.LITELLM_URL ?? 'http://localhost:4000';
const ORACLE_MODEL = 'claude-opus-4-7';
const ORACLE_MAX_TOKENS = 8000;
const ORACLE_TEMPERATURE = 0.7; // higher for instance variation per manifest v7
const ORACLE_THINKING = true;
const MANIFEST_ANCHOR = 'manifest-v7-gepa-faza1';
// Pricing per pilot runner SHA 8a6251e2 line 129
const PRICE_INPUT_PER_M = 15.0;
const PRICE_OUTPUT_PER_M = 75.0;
// Cost halt per manifest v7 Amendment 3: $15 = 40% buffer over $13.58 actual expected
// (was $7 pre-Amendment-3; raised after probe revealed inherited $0.10/instance estimate
// was 170% off vs actual Opus 4.7 generation cost of $0.27/instance).
const COST_HALT_USD = 15.0;
// ── Logging ────────────────────────────────────────────────────────────────
function log(msg: string): void {
const line = `[${new Date().toISOString()}] ${msg}\n`;
try { fs.appendFileSync(RUN_LOG, line); } catch { /* dir may not exist yet */ }
process.stderr.write(line);
}
// ── CLI ────────────────────────────────────────────────────────────────────
interface Args {
mode: 'dry-run' | 'probe' | 'all' | 'retry-failed';
startIdx?: number;
endIdx?: number;
}
/**
* The 3 cells that failed in the 2026-04-28 first-pass generation due to
* Opus emitting unescaped quotation marks in long doc bodies. Per
* Amendment 4 retry methodology, these cells are re-run with JSON-mode
* response_format + lowered temperature + reduced max_tokens.
*/
const RETRY_FAILED_CELLS: ReadonlyArray<{ family: string; persona: string; stage: string }> = [
{ family: 'F4', persona: 'p2_cfo', stage: 'stage_a_series_b_growth_burning' },
{ family: 'F4', persona: 'p2_cfo', stage: 'stage_b_post_profitable_consolidation' },
{ family: 'F5', persona: 'p1_founder_ceo', stage: 'stage_a_series_b_growth_burning' },
];
function parseArgs(argv: string[]): Args {
let mode: Args['mode'] = 'dry-run';
let startIdx: number | undefined;
let endIdx: number | undefined;
for (let i = 0; i < argv.length; i++) {
const flag = argv[i];
const next = argv[i + 1];
switch (flag) {
case '--dry-run': mode = 'dry-run'; break;
case '--probe': mode = 'probe'; break;
case '--all': mode = 'all'; break;
case '--retry-failed': mode = 'retry-failed'; break;
case '--start': startIdx = Number(next); i++; break;
case '--end': endIdx = Number(next); i++; break;
}
}
return { mode, startIdx, endIdx };
}
// ── LiteLLM call ───────────────────────────────────────────────────────────
interface LlmResult {
content: string;
inTokens: number;
outTokens: number;
costUsd: number;
latencyMs: number;
error?: string;
}
/**
* Per-call Opus oracle options. The default (temperature 1.0, max_tokens 8000,
* no response_format) matches the original generation. JSON-mode retry uses
* temperature 0.3 + max_tokens 6000 + response_format json_object per
* Amendment 4 retry methodology.
*/
interface OpusOracleOptions {
maxTokens?: number;
temperature?: number;
responseFormatJsonObject?: boolean;
}
async function callOpusOracle(
prompt: string,
options: OpusOracleOptions = {},
): Promise<LlmResult> {
const masterKey = process.env.LITELLM_MASTER_KEY;
if (!masterKey) {
throw new Error('LITELLM_MASTER_KEY env not set; cannot call Opus oracle');
}
const payload: Record<string, unknown> = {
model: ORACLE_MODEL,
messages: [{ role: 'user', content: prompt }],
max_tokens: options.maxTokens ?? ORACLE_MAX_TOKENS,
};
// Anthropic Opus 4.7 + response_format=json_object rejects `temperature`
// as deprecated for that mode. Omit temperature when JSON-mode is requested
// (matches the pilot runner's "reasoning model omit temperature" precedent
// for GPT-5.4 + MiniMax). Standard mode keeps temperature.
if (options.responseFormatJsonObject) {
payload.response_format = { type: 'json_object' };
} else {
payload.temperature = options.temperature ?? 1.0;
}
const started = Date.now();
let lastErr: string | undefined;
for (let attempt = 0; attempt < 2; attempt++) {
try {
const resp = await fetch(`${LITELLM_URL}/chat/completions`, {
method: 'POST',
headers: { 'Content-Type': 'application/json', Authorization: `Bearer ${masterKey}` },
body: JSON.stringify(payload),
});
const d: any = await resp.json();
if ('error' in d) {
lastErr = String(d.error?.message ?? JSON.stringify(d.error)).slice(0, 200);
if (attempt < 1) await new Promise(r => setTimeout(r, 1500));
continue;
}
const content = d.choices?.[0]?.message?.content ?? '';
const usage = d.usage ?? {};
const inTok = usage.prompt_tokens ?? 0;
const outTok = usage.completion_tokens ?? 0;
const costUsd = (inTok * PRICE_INPUT_PER_M + outTok * PRICE_OUTPUT_PER_M) / 1_000_000;
return { content, inTokens: inTok, outTokens: outTok, costUsd, latencyMs: Date.now() - started };
} catch (e) {
lastErr = `${(e as Error).name}: ${(e as Error).message}`.slice(0, 200);
if (attempt < 1) await new Promise(r => setTimeout(r, 1500));
continue;
}
}
return {
content: '', inTokens: 0, outTokens: 0, costUsd: 0,
latencyMs: Date.now() - started,
error: lastErr ?? 'unknown error',
};
}
// ── JSON extraction ────────────────────────────────────────────────────────
interface ParsedInstance {
personaText: string;
scenario: string; // optional — some prompts may put scenario inside personaText
sourceDocuments: Array<{ title: string; body: string }>;
question: string;
}
function parseInstanceJson(content: string): ParsedInstance | { error: string } {
// Strip code fence wrappers if present (Opus sometimes adds them despite instructions)
let s = content.trim();
if (s.startsWith('```')) {
s = s.replace(/^```[a-z]*\n?/, '').replace(/```\s*$/, '');
}
// Find first { and last } for tolerant parsing
const firstBrace = s.indexOf('{');
const lastBrace = s.lastIndexOf('}');
if (firstBrace < 0 || lastBrace < 0) {
return { error: `no JSON object found in oracle response (length=${content.length})` };
}
const jsonStr = s.slice(firstBrace, lastBrace + 1);
try {
const parsed = JSON.parse(jsonStr);
// Tolerant schema: accept either a "scenario" field or scenario embedded in personaText
const personaText: string = parsed.personaText ?? '';
const scenario: string = parsed.scenario ?? '';
const sourceDocuments = Array.isArray(parsed.sourceDocuments) ? parsed.sourceDocuments : [];
const question: string = parsed.question ?? '';
if (!personaText || !sourceDocuments.length || !question) {
return { error: `parsed JSON missing required fields (personaText/sourceDocuments/question)` };
}
return { personaText, scenario, sourceDocuments, question };
} catch (e) {
return { error: `JSON parse failed: ${(e as Error).message}` };
}
}
// ── Build full CorpusInstance from parsed oracle output ────────────────────
function assembleInstance(
cell: StratificationCell,
instanceId: string,
parsed: ParsedInstance,
llm: LlmResult,
): CorpusInstance {
const sourceDocuments = parsed.sourceDocuments.map(d => ({
title: d.title,
body: d.body,
charCount: d.body.length,
}));
// If oracle put scenario in personaText, just use personaText as-is.
// Otherwise concatenate "personaText\n\nScenario: scenario".
const fullPersonaText = parsed.scenario
? `${parsed.personaText}\n\nScenario: ${parsed.scenario}`
: parsed.personaText;
const materialsConcat = sourceDocuments
.map(d => `## ${d.title}\n\n${d.body}`)
.join('\n\n---\n\n');
return {
instanceId,
cell,
personaText: fullPersonaText,
scenario: parsed.scenario || extractScenarioFromPersonaText(parsed.personaText),
sourceDocuments,
question: parsed.question,
materialsConcat,
manifestAnchor: MANIFEST_ANCHOR,
generatedBy: ORACLE_MODEL,
generatedAtIso: new Date().toISOString(),
generationCostUsd: llm.costUsd,
};
}
function extractScenarioFromPersonaText(personaText: string): string {
const m = personaText.match(/Scenario:\s*([\s\S]*)/i);
return m ? m[1].trim() : '';
}
// ── Generate one cell ──────────────────────────────────────────────────────
async function generateOneCell(
cell: StratificationCell,
ordinal: number,
options: OpusOracleOptions & { retryNote?: string } = {},
): Promise<CorpusInstance | { error: string }> {
const instanceId = buildInstanceId(cell, ordinal);
const prompt = buildCorpusInstancePrompt({ cell, instanceId });
const noteSuffix = options.retryNote ? ` [${options.retryNote}]` : '';
log(`[${instanceId}] generating via ${ORACLE_MODEL} (prompt ${prompt.length}c)${noteSuffix}`);
const llm = await callOpusOracle(prompt, options);
if (llm.error) {
log(`[${instanceId}] LLM error: ${llm.error}`);
return { error: `LLM error: ${llm.error}` };
}
const parsed = parseInstanceJson(llm.content);
if ('error' in parsed) {
log(`[${instanceId}] parse error: ${parsed.error}; raw content first 200c: ${llm.content.slice(0, 200)}`);
return { error: parsed.error };
}
const instance = assembleInstance(cell, instanceId, parsed, llm);
const validation = validateInstance(instance);
if (!validation.valid) {
log(`[${instanceId}] validation failed: ${validation.violations.join('; ')}`);
return { error: `validation failed: ${validation.violations.join('; ')}` };
}
log(`[${instanceId}] OK; cost=$${llm.costUsd.toFixed(4)}; ${instance.sourceDocuments.length} docs; latency=${llm.latencyMs}ms`);
return instance;
}
// ── Spot-audit report writer (Pre-A halt-and-PM artifact) ──────────────────
function writeSpotAuditReport(instances: CorpusInstance[], totalCostUsd: number): void {
const audit = runSpotAudit(instances);
const sha = corpusSha256(instances);
const md: string[] = [];
md.push('---');
md.push('report_id: 2026-04-28-gepa-faza1-pre-a-corpus-audit');
md.push('date: 2026-04-28');
md.push('checkpoint: Pre-A (corpus quality + NULL kick auth)');
md.push('manifest_anchor: manifest-v7-gepa-faza1');
md.push(`corpus_sha256: ${sha}`);
md.push(`total_instances: ${instances.length}`);
md.push(`total_generation_cost_usd: ${totalCostUsd.toFixed(4)}`);
md.push(`spot_audit_sample_size: ${audit.sampleSize}`);
md.push(`spot_audit_seed: ${STRATIFICATION_SEED}`);
md.push(`halt_on_failure: ${audit.haltOnFailure}`);
md.push('---');
md.push('');
md.push('# Pre-A Halt-and-PM Report — H3 Corpus Quality Audit');
md.push('');
md.push('## TL;DR');
md.push('');
md.push(`Generated **${instances.length}/${TOTAL_INSTANCES}** instances at total cost **$${totalCostUsd.toFixed(2)}** (vs $5 expected, $7 halt). Spot-audit sample of ${audit.sampleSize} random instances (seed=${STRATIFICATION_SEED}) ${audit.haltOnFailure ? 'FAILED — corpus regeneration required.' : 'PASSED — NULL-baseline kick authorized pending PM ratify.'}`);
md.push('');
md.push('## Spot-audit results (per-instance)');
md.push('');
md.push('| Instance ID | Result | Violations |');
md.push('|---|---|---|');
for (const a of audit.perInstance) {
md.push(`| \`${a.instanceId}\` | ${a.result.valid ? '✓ PASS' : '✗ FAIL'} | ${a.result.valid ? '—' : a.result.violations.join('; ')} |`);
}
md.push('');
md.push('## Stratification coverage');
md.push('');
md.push(`All 50 (5 task families × 5 personas × 2 company stages) cells generated in canonical order. Each (family, persona) pair appears exactly twice (once per stage). Stratification verified via library tests (\`corpus.test.ts\` 34 tests passing).`);
md.push('');
md.push('## Audit chain');
md.push('');
md.push(`- Corpus JSONL: \`benchmarks/results/gepa-faza1/corpus/h3-northlane-cfo-50-instances.jsonl\``);
md.push(`- Corpus SHA256: \`${sha}\``);
md.push(`- Generation log: \`benchmarks/results/gepa-faza1/corpus/generation-run.log\``);
md.push(`- Manifest v7 SHA: \`583712dde139ffc87fb1ab21643f68d52c56469ded9e8090a624980b05969beb\``);
md.push(`- Substrate: c9bda3d (Phase 4.7) via worktree D:/Projects/waggle-os-faza1-wt`);
md.push('');
md.push('## PM ratification ask');
md.push('');
md.push(audit.haltOnFailure
? 'CORPUS FAILED spot-audit. **Do NOT authorize NULL-baseline kick.** Recommended action: review failed instances above + re-run generation for failed cells (cost ~$0.20 per re-gen).'
: 'CORPUS PASSED spot-audit. **Authorize NULL-baseline kick** (5 shapes × 8 instances per shape, expected cost ~$20).');
md.push('');
md.push('---');
md.push('');
md.push('**End of Pre-A halt-and-PM report. Standing AWAITING PM ratification.**');
fs.writeFileSync(SPOT_AUDIT_REPORT, md.join('\n'), 'utf-8');
log(`[pre-a] spot-audit report written to ${SPOT_AUDIT_REPORT}`);
}
// ── Main ───────────────────────────────────────────────────────────────────
async function main(): Promise<void> {
const args = parseArgs(process.argv.slice(2));
fs.mkdirSync(OUT_DIR, { recursive: true });
const cells = listStratificationCells();
log(`[generator] mode=${args.mode}; total cells=${cells.length}`);
if (args.mode === 'dry-run') {
log(`[dry-run] validating ${cells.length} stratification cells + prompt builds`);
for (let i = 0; i < cells.length; i++) {
const cell = cells[i];
const id = buildInstanceId(cell, 1);
const prompt = buildCorpusInstancePrompt({ cell, instanceId: id });
if (i < 3 || i === cells.length - 1) {
log(`[dry-run] cell[${i}] = ${id}; prompt = ${prompt.length}c`);
}
}
log(`[dry-run] OK — all ${cells.length} cells produce valid prompts; no LLM call made`);
log(`[dry-run] cost: $0.00`);
return;
}
let targetCells: StratificationCell[];
if (args.mode === 'probe') {
targetCells = cells.slice(args.startIdx ?? 0, (args.startIdx ?? 0) + 1);
} else if (args.mode === 'retry-failed') {
// Filter stratification to only the cells listed in RETRY_FAILED_CELLS.
const lookup = new Set(RETRY_FAILED_CELLS.map(c => `${c.family}|${c.persona}|${c.stage}`));
targetCells = cells.filter(c => lookup.has(`${c.family}|${c.persona}|${c.stage}`));
if (targetCells.length !== RETRY_FAILED_CELLS.length) {
log(`[retry-failed] FATAL: expected ${RETRY_FAILED_CELLS.length} cells, found ${targetCells.length}`);
process.exit(2);
}
} else {
targetCells = cells.slice(args.startIdx ?? 0, args.endIdx ?? cells.length);
}
log(`[${args.mode}] generating ${targetCells.length} instance(s)`);
const instances: CorpusInstance[] = [];
let cumulativeCost = 0;
// Resume support: always load existing JSONL on startup so we never truncate
// an existing corpus. Originally guarded on mode==='all' which broke retry-failed
// mode (corpus was truncated to 3 retry instances; recovery via git checkout
// restored 47 originals; this fix prevents recurrence).
if (fs.existsSync(OUT_JSONL) && args.mode !== 'dry-run') {
const lines = fs.readFileSync(OUT_JSONL, 'utf-8').trim().split(/\n+/).filter(Boolean);
for (const line of lines) {
try {
const inst = JSON.parse(line) as CorpusInstance;
instances.push(inst);
cumulativeCost += inst.generationCostUsd;
} catch { /* skip malformed */ }
}
log(`[resume] loaded ${instances.length} existing instances; cumulative cost = $${cumulativeCost.toFixed(4)}`);
}
// Open JSONL for append
const out = fs.createWriteStream(OUT_JSONL, { flags: instances.length > 0 ? 'a' : 'w' });
const existingIds = new Set(instances.map(i => i.instanceId));
// Per Amendment 4: retry-failed mode uses JSON-mode response_format +
// lower temperature + reduced max_tokens to mitigate the "Opus emits
// unescaped quotes in long doc bodies" failure class observed on first run.
const isRetryMode = args.mode === 'retry-failed';
const cellOptions: OpusOracleOptions & { retryNote?: string } = isRetryMode
? {
responseFormatJsonObject: true,
temperature: 0.3,
maxTokens: 6000,
retryNote: 'JSON-mode retry per Amendment 4',
}
: {};
for (let i = 0; i < targetCells.length; i++) {
const cell = targetCells[i];
const id = buildInstanceId(cell, 1);
if (existingIds.has(id) && !isRetryMode) {
log(`[skip] ${id} already in JSONL`);
continue;
}
if (existingIds.has(id) && isRetryMode) {
// In retry mode, this should not happen (retry targets only failed cells)
log(`[retry-failed] WARNING: ${id} already in JSONL — skipping`);
continue;
}
if (cumulativeCost >= COST_HALT_USD) {
log(`[HALT] cumulative $${cumulativeCost.toFixed(4)} >= $${COST_HALT_USD} cost halt — stopping generation`);
break;
}
const result = await generateOneCell(cell, 1, cellOptions);
if ('error' in result) {
log(`[error] cell ${id} skipped due to: ${result.error}`);
continue;
}
instances.push(result);
cumulativeCost += result.generationCostUsd;
out.write(JSON.stringify(result) + '\n');
log(`[cumulative] $${cumulativeCost.toFixed(4)} / $${COST_HALT_USD} halt; ${instances.length}/${TOTAL_INSTANCES} instances`);
}
out.end();
log(`[done] generated ${instances.length} instances; total cost $${cumulativeCost.toFixed(4)}`);
// Spot-audit + Pre-A report (only meaningful if we have a full or near-full corpus).
// Skipped in retry-failed mode — the corrected Pre-A addendum is authored manually
// by the orchestrating session per Amendment 4 §texture_audit_methodology.
if (args.mode === 'all' && instances.length > 0) {
writeSpotAuditReport(instances, cumulativeCost);
}
}
main().catch((e) => {
console.error('FATAL:', e);
process.exit(2);
});

View File

@@ -1,161 +0,0 @@
#!/usr/bin/env tsx
/**
* GEPA Faza 1 — REGISTRY-injection root-cause diagnostic probe.
*
* Per investigate-report.md §C.3, this probe distinguishes:
* H1 — ESM module-identity mismatch (deep relative path vs package import)
* H2 — registry mutation timing / silent freeze
* H3 — other (tsx loader, vite-node interop, etc.)
*
* Method: import REGISTRY via BOTH paths the failing runner uses, then
* mutate via the script's path and probe via both. Also exercise the
* exact selectShape() call site the agent-loop uses.
*
* Cost: ~$0 (no LLM calls).
*
* Verdicts:
* - "Same object?" === true → mutations propagate; H2/H3 candidates
* - "Same object?" === false → H1 confirmed (module-identity mismatch)
* - selectShape from package returns the mutated shape → fix is unnecessary
* - selectShape from package throws → bug is on the read path
*/
// ── Path A: deep relative import (matches the failing runner's import) ──
import { REGISTRY as RegistryFromScriptDeepPath } from '../../../../packages/agent/src/prompt-shapes/selector.js';
// ── Path B: package import (matches what runRetrievalAgentLoop's call to
// selectShape() goes through internally) ─────────────────────────
import {
REGISTRY as RegistryFromPackage,
selectShape as selectShapeFromPackage,
} from '@waggle/agent';
// ── Path C: package-internal prompt-shapes export (parallel re-export check) ──
import { REGISTRY as RegistryFromPromptShapes } from '@waggle/agent';
const PROBE_SHAPE_NAME = 'claude-gen1-v1-probe';
// Build a minimal valid PromptShape stub matching the type contract.
// We only care that it gets registered + retrieved; method bodies are not
// invoked in this probe.
const probeShape = {
name: PROBE_SHAPE_NAME,
metadata: {
description: 'Probe shape for REGISTRY-injection diagnostic',
modelClass: 'probe',
defaultThinking: false,
defaultMaxTokens: 100,
evidence_link: 'benchmarks/results/gepa-faza1/gen-1/investigate-report.md',
},
systemPrompt: () => 'probe',
soloUserPrompt: () => 'probe',
multiStepKickoffUserPrompt: () => 'probe',
retrievalInjectionUserPrompt: () => 'probe',
};
function log(line: string): void {
process.stdout.write(line + '\n');
}
function header(t: string): void {
log('');
log('━'.repeat(76));
log(` ${t}`);
log('━'.repeat(76));
}
header('GEPA Faza 1 — REGISTRY-injection diagnostic probe');
log(`Probe shape name: ${PROBE_SHAPE_NAME}`);
log(`Node version: ${process.version}`);
log(`tsx : (running via tsx if argv[0] is node + script)`);
// ── Step 1 — pre-mutation snapshot ────────────────────────────────────────
header('STEP 1 — Pre-mutation snapshot (3 import paths)');
const keysScriptPath = Object.keys(RegistryFromScriptDeepPath).sort();
const keysPackagePath = Object.keys(RegistryFromPackage).sort();
const keysPromptShapesPath = Object.keys(RegistryFromPromptShapes).sort();
log(`A) Script deep-relative-path REGISTRY: ${keysScriptPath.length} keys: [${keysScriptPath.join(', ')}]`);
log(`B) Package @waggle/agent REGISTRY: ${keysPackagePath.length} keys: [${keysPackagePath.join(', ')}]`);
log(`C) Package @waggle/agent (2nd import): ${keysPromptShapesPath.length} keys: [${keysPromptShapesPath.join(', ')}]`);
log('');
log(`Object identity A === B: ${RegistryFromScriptDeepPath === RegistryFromPackage}`);
log(`Object identity A === C: ${RegistryFromScriptDeepPath === RegistryFromPromptShapes}`);
log(`Object identity B === C: ${RegistryFromPackage === RegistryFromPromptShapes}`);
// ── Step 2 — mutate via script's path (the failing pattern) ───────────────
header('STEP 2 — Mutate REGISTRY via script deep-relative-path');
(RegistryFromScriptDeepPath as any)[PROBE_SHAPE_NAME] = probeShape;
log(`Mutation via Path A: REGISTRY[${PROBE_SHAPE_NAME}] = probeShape`);
// ── Step 3 — read-back from all 3 paths ───────────────────────────────────
header('STEP 3 — Read-back probe-shape via each path');
const seenInA = RegistryFromScriptDeepPath[PROBE_SHAPE_NAME] !== undefined;
const seenInB = RegistryFromPackage[PROBE_SHAPE_NAME] !== undefined;
const seenInC = RegistryFromPromptShapes[PROBE_SHAPE_NAME] !== undefined;
log(`A) Script-import sees probe-shape: ${seenInA}`);
log(`B) Package-import sees probe-shape: ${seenInB}`);
log(`C) Re-import sees probe-shape: ${seenInC}`);
// ── Step 4 — call selectShape via package (the agent-loop path) ───────────
header('STEP 4 — selectShape({override}) via @waggle/agent (agent-loop path)');
let selectShapeVerdict: 'FOUND' | 'NOT_FOUND' | 'OTHER_ERROR';
let selectShapeError: string | null = null;
try {
const found = selectShapeFromPackage('any-alias-not-relevant', { override: PROBE_SHAPE_NAME });
log(`selectShape returned shape with name="${(found as any).name ?? '<missing>'}"`);
selectShapeVerdict = 'FOUND';
} catch (e) {
selectShapeError = (e as Error).message;
log(`selectShape THREW: ${selectShapeError}`);
selectShapeVerdict = selectShapeError.includes('not in REGISTRY') ? 'NOT_FOUND' : 'OTHER_ERROR';
}
// ── Step 5 — verdict ───────────────────────────────────────────────────────
header('STEP 5 — VERDICT');
const sameObjectAB = RegistryFromScriptDeepPath === RegistryFromPackage;
if (!sameObjectAB) {
log('VERDICT: H1 CONFIRMED — ESM module-identity mismatch');
log(' Script deep-relative-path REGISTRY and @waggle/agent REGISTRY are');
log(' separate object instances. Mutations to one do not propagate.');
log('');
log('Fix: add registerShape(name, shape) API in selector.ts, called from');
log(' a single canonical entry point. Avoid direct REGISTRY mutation.');
} else if (sameObjectAB && seenInA && !seenInB) {
log('VERDICT: H2/H3 — Same object but read mismatch');
log(' This should not happen: identical objects with different key sets.');
log(' Investigate JS engine optimization, hidden Proxy, or freeze-on-read.');
} else if (sameObjectAB && seenInA && seenInB && selectShapeVerdict === 'FOUND') {
log('VERDICT: NOT REPRODUCED — REGISTRY mutation works in this probe.');
log(' The runner-time failure must be due to a different cause (timing,');
log(' loader, dynamic-import side effect on candidate shape). Investigate');
log(' loadCandidates() dynamic import return shape vs static import.');
} else if (sameObjectAB && seenInA && seenInB && selectShapeVerdict !== 'FOUND') {
log('VERDICT: H2/H3 — Read found but selectShape rejected');
log(` Direct read sees probe-shape, but selectShape() error: ${selectShapeError}`);
log(' Investigate selectShape() implementation for hidden state or guard.');
} else {
log('VERDICT: UNKNOWN');
log(` sameObjectAB=${sameObjectAB} seenInA=${seenInA} seenInB=${seenInB} seenInC=${seenInC} selectShape=${selectShapeVerdict}`);
}
log('');
log('Probe complete. Cost: $0 (no LLM calls). Halt-and-PM with verdict above.');
// Exit 0 in all cases — we want PM to read the full output regardless of verdict
process.exit(0);

View File

@@ -1,722 +0,0 @@
#!/usr/bin/env tsx
/**
* GEPA Faza 1 — Checkpoint C held-out validation runner.
*
* Per launch decision §F + §G step 9 + PM brief 2026-04-29 Checkpoint C ratify.
*
* Validates §F.1-passing candidates on 5 held-out instances per candidate
* (NOT in original Gen 1 8-instance sample). Confirms §F.2 PASS isn't
* overfit per §F.5 condition_2 (held-out Pass II within ±15pp of in-sample).
*
* Pre-registered candidates per PM brief 2026-04-29:
* - claude::gen1-v1
* - qwen-thinking::gen1-v1
* - gpt::gen1-v2
*
* Usage:
* --candidates <id,id,id> comma-separated candidate IDs (required)
* --held-out-instances <N> default 5
* --dry-run list planned evals without executing
*
* Held-out sample: deterministicShuffle(corpus, seed=42).slice(8, 8 + N_HELD_OUT)
* — instances 8..12 of the same shuffled order Gen 1 used (Gen 1 used 0..7).
*
* Manifest binding: Amendment 11 (manifest_sha256_post_amendment_11 = fa716ff90a...).
*/
import * as fs from 'node:fs';
import * as path from 'node:path';
import { fileURLToPath, pathToFileURL } from 'node:url';
import {
MindDB,
FrameStore,
SessionStore,
HybridSearch,
createOllamaEmbedder,
type Embedder,
} from '@waggle/core';
import {
runRetrievalAgentLoop,
type LlmCallFn,
type LlmCallInput,
type LlmCallResult as AgentLlmCallResult,
type RetrievalSearchFn,
type AgentRunResult,
// Amendment 8 §canonical_mutation_api: registerShape MUST be imported from
// '@waggle/agent' (same path the agent-loop uses internally) so the mutation
// hits the SAME REGISTRY instance.
REGISTRY,
registerShape,
type PromptShape,
} from '@waggle/agent';
import { type CorpusInstance } from '../../src/faza-1/corpus.js';
import {
NULL_BASELINE_PER_SHAPE,
NULL_BASELINE_AGGREGATE,
type TieredFitnessComponents,
type ShapeName,
} from '../../src/faza-1/types.js';
import {
computeTieredFitness,
} from '../../src/faza-1/fitness.js';
import {
validateCandidate,
} from '../../src/faza-1/mutation-validator.js';
const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);
const REPO_ROOT = path.resolve(__dirname, '../../../..');
const CORPUS_JSONL = path.join(REPO_ROOT, 'benchmarks/results/gepa-faza1/corpus/h3-northlane-cfo-50-instances.jsonl');
const PROMPT_SHAPES_DIR = path.join(REPO_ROOT, 'packages/agent/src/prompt-shapes');
const GEPA_EVOLVED_DIR = path.join(PROMPT_SHAPES_DIR, 'gepa-evolved');
const OUT_DIR = path.join(REPO_ROOT, 'benchmarks/results/gepa-faza1/checkpoint-c');
const OUT_JSONL = path.join(OUT_DIR, 'checkpoint-c-eval.jsonl');
const RUN_LOG = path.join(OUT_DIR, 'checkpoint-c-run.log');
const SUMMARY_JSON = path.join(OUT_DIR, 'checkpoint-c-summary.json');
const SCRATCH_DIR = path.join(REPO_ROOT, 'tmp/gepa-faza1-checkpoint-c');
const GEN_1_JSONL = path.join(REPO_ROOT, 'benchmarks/results/gepa-faza1/gen-1/gen-1-eval.jsonl');
const LITELLM_URL = process.env.LITELLM_URL ?? 'http://localhost:4000';
const OLLAMA_URL = 'http://localhost:11434';
const EMBEDDER_MODEL = 'nomic-embed-text';
const SAMPLING_SEED = 42;
const N_GEN_1_SAMPLE = 8;
const DEFAULT_HELD_OUT = 5;
const SUBJECT_ALIAS = 'qwen3.6-35b-a3b-via-dashscope-direct';
const SUBJECT_MAX_TOKENS = 16000;
const SUBJECT_THINKING = true;
const JUDGES = ['claude-opus-4-7', 'gpt-5.4', 'minimax-m27-via-openrouter'] as const;
const JUDGE_MAX_TOKENS = 3000;
const JUDGE_RETRIES = 3;
const MAX_STEPS = 5;
const MAX_RETRIEVALS_PER_STEP = 8;
const PER_CALL_HALT_USD = 0.40;
const PER_CELL_HALT_USD = 1.00;
const COST_HALT_USD = 8.0; // generous budget for held-out (3 candidates × 5 evals × ~$0.13 = $1.95 expected)
const MODEL_PRICING: Record<string, { in: number; out: number }> = {
'claude-opus-4-7': { in: 15.0, out: 75.0 },
'gpt-5.4': { in: 2.5, out: 10.0 },
'minimax-m27-via-openrouter': { in: 0.7, out: 2.8 },
'qwen3.6-35b-a3b-via-dashscope-direct': { in: 0.20, out: 0.80 },
'qwen3.6-35b-a3b-via-openrouter': { in: 0.6, out: 2.4 },
};
const MANIFEST_ANCHOR = 'manifest-v7-gepa-faza1';
const MANIFEST_SHA_AMENDMENT_11 = 'fa716ff90a4345eb87962789f3a2ab3d54994edc93964f850ad64cf6fbf6d227';
// §F.5 condition_2 overfitting bound: held-out Pass II must be within ±15pp of in-sample Pass II
const F5_OVERFITTING_BOUND_PP = 15;
function log(msg: string): void {
const line = `[${new Date().toISOString()}] ${msg}\n`;
try { fs.appendFileSync(RUN_LOG, line); } catch { /* dir may not exist */ }
process.stderr.write(line);
}
interface Args {
mode: 'dry-run' | 'execute';
candidateIds: string[];
heldOutCount: number;
}
function parseArgs(argv: string[]): Args {
let mode: Args['mode'] = 'execute';
let candidateIds: string[] = [];
let heldOutCount = DEFAULT_HELD_OUT;
for (let i = 0; i < argv.length; i++) {
const f = argv[i];
if (f === '--dry-run') mode = 'dry-run';
else if (f === '--candidates' && i + 1 < argv.length) {
candidateIds = argv[i + 1].split(',').map(s => s.trim()).filter(Boolean);
i++;
} else if (f === '--held-out-instances' && i + 1 < argv.length) {
heldOutCount = parseInt(argv[i + 1], 10);
i++;
}
}
if (candidateIds.length === 0) {
throw new Error('--candidates flag required (comma-separated candidate IDs e.g. claude::gen1-v1,qwen-thinking::gen1-v1,gpt::gen1-v2)');
}
return { mode, candidateIds, heldOutCount };
}
// ── Mulberry32 sampling (same as Gen 1 / NULL-baseline) ──────────────────
function mulberry32(seed: number): () => number {
let t = seed >>> 0;
return () => {
t = (t + 0x6d2b79f5) >>> 0;
let r = t;
r = Math.imul(r ^ (r >>> 15), r | 1);
r ^= r + Math.imul(r ^ (r >>> 7), r | 61);
return ((r ^ (r >>> 14)) >>> 0) / 4294967296;
};
}
function deterministicShuffle<T>(items: ReadonlyArray<T>, seed: number): T[] {
const arr = [...items];
const rand = mulberry32(seed);
for (let i = arr.length - 1; i > 0; i--) {
const j = Math.floor(rand() * (i + 1));
[arr[i], arr[j]] = [arr[j], arr[i]];
}
return arr;
}
function loadCorpus(): CorpusInstance[] {
return fs.readFileSync(CORPUS_JSONL, 'utf-8').trim().split(/\n+/).filter(Boolean).map(l => JSON.parse(l));
}
// ── Candidate loader (subset by ID) ────────────────────────────────────────
interface Candidate {
candidateId: string;
shape: ShapeName;
variant: 'baseline' | 'gen1-v1' | 'gen1-v2';
promptShape: PromptShape;
}
async function loadCandidatesById(candidateIds: string[]): Promise<Candidate[]> {
const out: Candidate[] = [];
for (const id of candidateIds) {
// Format: <shape>::<variant>
const m = id.match(/^([a-z-]+)::([a-z0-9-]+)$/);
if (!m) throw new Error(`Invalid candidate ID format "${id}" — expected "<shape>::<variant>" (e.g., "claude::gen1-v1")`);
const shape = m[1] as ShapeName;
const variant = m[2] as Candidate['variant'];
if (variant === 'baseline') {
const baseline = REGISTRY[shape];
if (!baseline) throw new Error(`baseline shape "${shape}" not in REGISTRY`);
out.push({ candidateId: id, shape, variant: 'baseline', promptShape: baseline });
continue;
}
const filename = `${shape}-${variant}.ts`;
const filepath = path.join(GEPA_EVOLVED_DIR, filename);
if (!fs.existsSync(filepath)) {
throw new Error(`mutation file missing: ${filepath}`);
}
const mod: any = await import(pathToFileURL(filepath).href);
const promptShape = Object.values(mod).find(
(v: any) => v && typeof v === 'object' && 'name' in v && 'systemPrompt' in v && 'soloUserPrompt' in v,
) as PromptShape | undefined;
if (!promptShape) throw new Error(`no PromptShape export found in ${filepath}`);
out.push({ candidateId: id, shape, variant, promptShape });
}
return out;
}
// ── LLM call adapter (same as Gen 1) ──────────────────────────────────────
const llmCall: LlmCallFn = async (input: LlmCallInput): Promise<AgentLlmCallResult> => {
const masterKey = process.env.LITELLM_MASTER_KEY;
if (!masterKey) throw new Error('LITELLM_MASTER_KEY env not set');
const { model, messages } = input;
const isQwen = model.includes('qwen');
const maxTokens = input.maxTokens ?? (isQwen ? SUBJECT_MAX_TOKENS : 4096);
const thinking = input.thinking ?? (isQwen ? SUBJECT_THINKING : true);
const payload: Record<string, unknown> = { model, messages, max_tokens: maxTokens };
if (model.startsWith('claude-opus')) payload.temperature = 1.0;
else if (model === 'gpt-5.4' || model === 'minimax-m27-via-openrouter') {/* omit */}
else payload.temperature = input.temperature ?? 0.3;
if (isQwen) payload.extra_body = { enable_thinking: thinking };
const started = Date.now();
let lastErr: string | undefined;
for (let attempt = 0; attempt < 2; attempt++) {
try {
const resp = await fetch(`${LITELLM_URL}/chat/completions`, {
method: 'POST',
headers: { 'Content-Type': 'application/json', Authorization: `Bearer ${masterKey}` },
body: JSON.stringify(payload),
});
const d: any = await resp.json();
if ('error' in d) {
lastErr = String(d.error?.message ?? JSON.stringify(d.error)).slice(0, 200);
if (attempt < 1) await new Promise(r => setTimeout(r, 1500));
continue;
}
const content = d.choices?.[0]?.message?.content ?? '';
const usage = d.usage ?? {};
const inTok = usage.prompt_tokens ?? 0;
const outTok = usage.completion_tokens ?? 0;
const pricing = MODEL_PRICING[model] ?? { in: 1, out: 4 };
return { content, inTokens: inTok, outTokens: outTok, costUsd: (inTok*pricing.in + outTok*pricing.out)/1_000_000, latencyMs: Date.now()-started };
} catch (e) {
lastErr = `${(e as Error).name}: ${(e as Error).message}`.slice(0, 200);
if (attempt < 1) await new Promise(r => setTimeout(r, 1500));
}
}
return { content: '', inTokens: 0, outTokens: 0, costUsd: 0, latencyMs: Date.now()-started, error: lastErr };
};
// ── Trio judging (same as Gen 1) ──────────────────────────────────────────
const JUDGE_PROMPT_TEMPLATE = `You are evaluating an AI agent's response to a complex knowledge work task. The persona, scenario, materials, and question are provided. The response was generated under one of four configurations (revealed only after scoring): {model_only | model + memory + agent harness} × {Opus 4.7 | Qwen 3.6 35B-A3B}.
You do NOT know which configuration produced this response. Score blind.
Read the persona/scenario/question (provided), skim the materials (provided), then read the response carefully (provided).
Score the response on six dimensions, Likert 1-5:
1. COMPLETENESS — engagement with all material
2. ACCURACY — faithfulness to source materials, no hallucinations
3. SYNTHESIS — connections across inputs, not isolated treatment
4. JUDGMENT — defensible recommendations, tradeoffs acknowledged
5. ACTIONABILITY — would the persona act on this tomorrow
6. STRUCTURE — organization and readability
Output JSON only:
{"completeness":<1-5>,"accuracy":<1-5>,"synthesis":<1-5>,"judgment":<1-5>,"actionability":<1-5>,"structure":<1-5>,"rationale":"<1-2 sentences explaining the lowest scoring dimension>","overall_verdict":"<one of: PASS_STRONG | PASS_ADEQUATE | FAIL_WEAK | FAIL_CRITICAL>"}
PASS_STRONG: mean >= 4.0
PASS_ADEQUATE: mean 3.5-3.99
FAIL_WEAK: mean 2.5-3.49
FAIL_CRITICAL: mean < 2.5
[PERSONA + SCENARIO + QUESTION]
###PSQ###
[MATERIALS]
###MAT###
[RESPONSE TO EVALUATE]
###RES###`;
interface JudgeRecord { judge_model: string; mean: number; cost: number; latency_ms: number; raw: any; retries: number }
interface TrioResult { records: JudgeRecord[]; trioMean: number; trioStrictPassII: boolean; trioStrictPassI: boolean; cost: number }
function parseJudgeJson(text: string): { mean: number; raw: any } | null {
const m = text.match(/\{[\s\S]*\}/);
if (!m) return null;
try {
const obj = JSON.parse(m[0]);
const dims = ['completeness','accuracy','synthesis','judgment','actionability','structure'];
for (const d of dims) if (typeof obj[d] !== 'number' || obj[d] < 1 || obj[d] > 5) return null;
const mean = dims.reduce((s, d) => s + obj[d], 0) / dims.length;
return { mean, raw: obj };
} catch { return null; }
}
async function runJudge(model: string, prompt: string): Promise<JudgeRecord> {
let totalCost = 0, totalLat = 0;
for (let attempt = 0; attempt < JUDGE_RETRIES; attempt++) {
const r = await llmCall({ model, messages: [{ role: 'user', content: prompt }], maxTokens: JUDGE_MAX_TOKENS, thinking: false });
totalCost += r.costUsd; totalLat += r.latencyMs;
if (r.error) continue;
const parsed = parseJudgeJson(r.content);
if (parsed) return { judge_model: model, mean: parsed.mean, cost: totalCost, latency_ms: totalLat, raw: parsed.raw, retries: attempt };
}
return { judge_model: model, mean: 0, cost: totalCost, latency_ms: totalLat, raw: null, retries: JUDGE_RETRIES };
}
async function judgeTrio(instance: CorpusInstance, response: string): Promise<TrioResult> {
const prompt = JUDGE_PROMPT_TEMPLATE
.replace('###PSQ###', `${instance.personaText}\n\nQUESTION: ${instance.question}`)
.replace('###MAT###', instance.materialsConcat)
.replace('###RES###', response);
const records = await Promise.all(JUDGES.map(j => runJudge(j, prompt)));
const valid = records.filter(r => r.mean > 0).map(r => r.mean);
const trioMean = valid.length > 0 ? valid.reduce((a,b)=>a+b,0)/valid.length : 0;
return {
records,
trioMean,
trioStrictPassII: trioMean >= 4.0,
trioStrictPassI: records.filter(r => r.mean >= 3.5).length >= 2,
cost: records.reduce((s,r) => s + r.cost, 0),
};
}
// ── Per-eval orchestration (mirrors Gen 1; uses registerShape for canonical injection) ──
interface EvalRecord {
shape: ShapeName;
candidateId: string;
variant: Candidate['variant'];
instanceId: string;
instanceCell: CorpusInstance['cell'];
candidateResponse: string;
candidateLatencyMs: number;
candidateCostUsd: number;
loopExhausted: boolean;
stepsTaken: number;
retrievalCalls: number;
trioMean: number;
trioStrictPassII: boolean;
trioStrictPassI: boolean;
judgeRecords: JudgeRecord[];
evalCostUsd: number;
manifestAnchor: string;
manifestShaAmendment11: string;
tsIso: string;
evalSet: 'held-out'; // marks this record as Checkpoint C held-out (vs Gen 1 in-sample)
}
async function runOneEval(cand: Candidate, instance: CorpusInstance, embedder: Embedder): Promise<EvalRecord | { error: string }> {
const evalId = `${cand.candidateId}__${instance.instanceId}`;
log(`[${evalId}] start`);
const dbPath = path.join(SCRATCH_DIR, `eval-${cand.candidateId.replace(/[:]/g, '_')}-${instance.instanceId}.sqlite`);
if (fs.existsSync(dbPath)) fs.unlinkSync(dbPath);
const db = new MindDB(dbPath);
const frames = new FrameStore(db);
const sessions = new SessionStore(db);
const hybrid = new HybridSearch(db, embedder);
const gopId = `cp-c-${cand.candidateId.replace(/[:]/g,'_')}-${instance.instanceId}`;
sessions.ensure(gopId, undefined, `Checkpoint C ${cand.candidateId} on ${instance.instanceId}`);
for (const doc of instance.sourceDocuments) frames.createIFrame(gopId, `## ${doc.title}\n\n${doc.body}`, 'important', 'system');
const search: RetrievalSearchFn = async ({ query, limit }) => {
const hits = await hybrid.search(query, { limit, gopId });
return {
formattedResults: hits.length > 0 ? hits.map((s,i)=>`[result ${i+1}, score ${s.finalScore.toFixed(3)}]\n${s.frame.content}`).join('\n\n---\n\n') : '',
resultCount: hits.length,
};
};
let agentResult: AgentRunResult;
try {
// Amendment 8 §canonical_mutation_api: register via @waggle/agent's registerShape
registerShape(cand.promptShape.name, cand.promptShape);
agentResult = await runRetrievalAgentLoop({
modelAlias: SUBJECT_ALIAS,
persona: instance.personaText,
question: instance.question,
llmCall,
search,
maxSteps: MAX_STEPS,
maxRetrievalsPerStep: MAX_RETRIEVALS_PER_STEP,
perCallHaltUsd: PER_CALL_HALT_USD,
perCellHaltUsd: PER_CELL_HALT_USD,
contextTag: evalId,
promptShapeOverride: cand.promptShape.name,
} as any);
} catch (e) {
return { error: `agent loop failed: ${(e as Error).message}` };
}
log(`[${evalId}] subject_done; retrievals=${agentResult.retrievalCalls} steps=${agentResult.stepsTaken} cost=$${agentResult.totalCostUsd.toFixed(4)}`);
const judges = await judgeTrio(instance, agentResult.rawResponse);
const evalCostUsd = agentResult.totalCostUsd + judges.cost;
log(`[${evalId}] judged; trio_mean=${judges.trioMean.toFixed(3)} pass_ii=${judges.trioStrictPassII} retrievals=${agentResult.retrievalCalls} eval_cost=$${evalCostUsd.toFixed(4)}`);
return {
shape: cand.shape, candidateId: cand.candidateId, variant: cand.variant,
instanceId: instance.instanceId, instanceCell: instance.cell,
candidateResponse: agentResult.rawResponse,
candidateLatencyMs: agentResult.totalLatencyMs,
candidateCostUsd: agentResult.totalCostUsd,
loopExhausted: agentResult.loopExhausted,
stepsTaken: agentResult.stepsTaken,
retrievalCalls: agentResult.retrievalCalls,
trioMean: judges.trioMean,
trioStrictPassII: judges.trioStrictPassII,
trioStrictPassI: judges.trioStrictPassI,
judgeRecords: judges.records,
evalCostUsd,
manifestAnchor: MANIFEST_ANCHOR,
manifestShaAmendment11: MANIFEST_SHA_AMENDMENT_11,
tsIso: new Date().toISOString(),
evalSet: 'held-out',
};
}
// ── In-sample lookup from Gen 1 JSONL ─────────────────────────────────────
interface InSampleStats {
candidateId: string;
shape: ShapeName;
evalCount: number;
passIICount: number;
passIIRate: number;
meanRetrieval: number;
}
function loadInSampleStats(candidateIds: string[]): Map<string, InSampleStats> {
const out = new Map<string, InSampleStats>();
if (!fs.existsSync(GEN_1_JSONL)) {
log(`[in-sample] WARN: Gen 1 JSONL not found at ${GEN_1_JSONL}; in-sample stats unavailable`);
return out;
}
const records: EvalRecord[] = [];
for (const line of fs.readFileSync(GEN_1_JSONL, 'utf-8').trim().split(/\n+/).filter(Boolean)) {
try { records.push(JSON.parse(line) as EvalRecord); } catch { /* skip */ }
}
for (const id of candidateIds) {
const candRecs = records.filter(r => r.candidateId === id);
if (candRecs.length === 0) continue;
const passII = candRecs.filter(r => r.trioStrictPassII).length;
const totalRetr = candRecs.reduce((s, r) => s + r.retrievalCalls, 0);
out.set(id, {
candidateId: id,
shape: candRecs[0].shape,
evalCount: candRecs.length,
passIICount: passII,
passIIRate: passII / candRecs.length,
meanRetrieval: totalRetr / candRecs.length,
});
}
return out;
}
// ── Summary writer ─────────────────────────────────────────────────────────
interface PerCandidateSummary {
candidateId: string;
shape: ShapeName;
variant: string;
inSample: { evalCount: number; passIIRate: number; meanRetrieval: number } | null;
heldOut: { evalCount: number; passIIRate: number; meanRetrieval: number; tieredFitness: TieredFitnessComponents };
passIIGapPP: number; // (in-sample - held-out) × 100; positive = held-out worse than in-sample
retrievalGapAbsolute: number; // (in-sample - held-out); positive = held-out lower retrieval
f5_condition_2_verdict: 'PASS' | 'FAIL';
f5_condition_2_detail: string;
phase_5_deployment_authorized: boolean;
}
interface CheckpointCSummary {
manifestAnchor: string;
manifestShaAmendment11: string;
generated_at: string;
candidateIds: string[];
heldOutInstanceIds: string[];
totalEvals: number;
totalCostUsd: number;
perCandidate: PerCandidateSummary[];
f2_verdict_confirmation: 'CONFIRMED' | 'REVERTED' | 'MIXED';
f2_verdict_detail: string;
f5_overfitting_bound_pp: number;
faza_2_deployment_authorization: 'AUTHORIZED' | 'WITHHELD' | 'PARTIAL';
next_steps: string[];
}
function buildCheckpointCSummary(
args: Args,
candidates: Candidate[],
heldOutInstances: CorpusInstance[],
recordsByCandidate: Map<string, EvalRecord[]>,
inSampleStats: Map<string, InSampleStats>,
totalCost: number,
): CheckpointCSummary {
const perCandidate: PerCandidateSummary[] = [];
for (const cand of candidates) {
const recs = recordsByCandidate.get(cand.candidateId) ?? [];
if (recs.length === 0) continue;
const passIICount = recs.filter(r => r.trioStrictPassII).length;
const passIIRate = passIICount / recs.length;
const meanRetrieval = recs.reduce((s, r) => s + r.retrievalCalls, 0) / recs.length;
const inSample = inSampleStats.get(cand.candidateId);
const passIIGapPP = inSample ? (inSample.passIIRate - passIIRate) * 100 : 0;
const retrievalGapAbsolute = inSample ? (inSample.meanRetrieval - meanRetrieval) : 0;
const candidateMetrics = {
candidateId: cand.candidateId,
shape: cand.shape,
evaluations: [],
trioStrictPassRateII: passIIRate,
trioStrictPassRateI: 0,
meanRetrievalCallsPerTask: meanRetrieval,
meanCostUsd: recs.reduce((s, r) => s + r.evalCostUsd, 0) / recs.length,
};
const tieredFitness = computeTieredFitness({
candidate: candidateMetrics,
nullBaselinePassRateII: NULL_BASELINE_PER_SHAPE[cand.shape].trioStrictPassRateII,
nullBaselineMeanRetrievalCallsPerTask: NULL_BASELINE_PER_SHAPE[cand.shape].meanRetrievalCallsPerTask,
mutationValidatorPassed: true, // all candidates validated upstream
saturatedRegime: true,
});
// §F.5 condition_2 verdict: held-out Pass II within ±15pp of in-sample
let f5_condition_2_verdict: 'PASS' | 'FAIL' = 'PASS';
let f5_condition_2_detail = '';
if (!inSample) {
f5_condition_2_verdict = 'FAIL';
f5_condition_2_detail = 'in-sample stats unavailable (Gen 1 JSONL missing or candidate not in Gen 1)';
} else {
const absGapPP = Math.abs(passIIGapPP);
if (absGapPP <= F5_OVERFITTING_BOUND_PP) {
f5_condition_2_verdict = 'PASS';
f5_condition_2_detail = `held-out ${(passIIRate * 100).toFixed(1)}% within ±${F5_OVERFITTING_BOUND_PP}pp of in-sample ${(inSample.passIIRate * 100).toFixed(1)}% (gap=${passIIGapPP.toFixed(1)}pp)`;
} else {
f5_condition_2_verdict = 'FAIL';
f5_condition_2_detail = `held-out ${(passIIRate * 100).toFixed(1)}% diverges from in-sample ${(inSample.passIIRate * 100).toFixed(1)}% by ${absGapPP.toFixed(1)}pp > ${F5_OVERFITTING_BOUND_PP}pp threshold`;
}
}
perCandidate.push({
candidateId: cand.candidateId,
shape: cand.shape,
variant: cand.variant,
inSample: inSample ? { evalCount: inSample.evalCount, passIIRate: inSample.passIIRate, meanRetrieval: inSample.meanRetrieval } : null,
heldOut: { evalCount: recs.length, passIIRate, meanRetrieval, tieredFitness },
passIIGapPP,
retrievalGapAbsolute,
f5_condition_2_verdict,
f5_condition_2_detail,
phase_5_deployment_authorized: f5_condition_2_verdict === 'PASS',
});
}
// §F.2 confirmation: if all candidates PASS §F.5 condition_2 → CONFIRMED;
// if all FAIL → REVERTED; else MIXED
const passCount = perCandidate.filter(c => c.f5_condition_2_verdict === 'PASS').length;
const total = perCandidate.length;
let f2_verdict_confirmation: 'CONFIRMED' | 'REVERTED' | 'MIXED';
let f2_verdict_detail = '';
if (passCount === total) {
f2_verdict_confirmation = 'CONFIRMED';
f2_verdict_detail = `all ${total} held-out candidates PASS §F.5 condition_2 (within ±${F5_OVERFITTING_BOUND_PP}pp)`;
} else if (passCount === 0) {
f2_verdict_confirmation = 'REVERTED';
f2_verdict_detail = `all ${total} held-out candidates FAIL §F.5 condition_2 — Gen 1 §F.2 PASS suspected overfit`;
} else {
f2_verdict_confirmation = 'MIXED';
f2_verdict_detail = `${passCount}/${total} held-out candidates PASS §F.5 condition_2`;
}
let faza_2_deployment_authorization: 'AUTHORIZED' | 'WITHHELD' | 'PARTIAL';
if (f2_verdict_confirmation === 'CONFIRMED') faza_2_deployment_authorization = 'AUTHORIZED';
else if (f2_verdict_confirmation === 'REVERTED') faza_2_deployment_authorization = 'WITHHELD';
else faza_2_deployment_authorization = 'PARTIAL';
const next_steps = [
'Compute κ_trio on combined Gen 1 (120) + Checkpoint C (15) sample for §F.3 verdict',
`Author Faza 1 final summary memo per ${faza_2_deployment_authorization} authorization status`,
'PM ratify final Faza 1 closure decision (decisions/2026-04-XX-gepa-faza1-results.md)',
];
return {
manifestAnchor: MANIFEST_ANCHOR,
manifestShaAmendment11: MANIFEST_SHA_AMENDMENT_11,
generated_at: new Date().toISOString(),
candidateIds: args.candidateIds,
heldOutInstanceIds: heldOutInstances.map(i => i.instanceId),
totalEvals: perCandidate.reduce((s, c) => s + c.heldOut.evalCount, 0),
totalCostUsd: totalCost,
perCandidate,
f2_verdict_confirmation,
f2_verdict_detail,
f5_overfitting_bound_pp: F5_OVERFITTING_BOUND_PP,
faza_2_deployment_authorization,
next_steps,
};
}
// ── Main ──────────────────────────────────────────────────────────────────
async function main(): Promise<void> {
const args = parseArgs(process.argv.slice(2));
fs.mkdirSync(OUT_DIR, { recursive: true });
fs.mkdirSync(SCRATCH_DIR, { recursive: true });
if (!fs.existsSync(RUN_LOG)) fs.writeFileSync(RUN_LOG, '');
const corpus = loadCorpus();
const allShuffled = deterministicShuffle(corpus, SAMPLING_SEED);
const heldOutSample = allShuffled.slice(N_GEN_1_SAMPLE, N_GEN_1_SAMPLE + args.heldOutCount);
log(`[loaded] corpus=${corpus.length}; held-out sample=${args.heldOutCount} via seed=${SAMPLING_SEED} offset=${N_GEN_1_SAMPLE}`);
const candidates = await loadCandidatesById(args.candidateIds);
log(`[loaded] ${candidates.length} candidates: ${candidates.map(c => c.candidateId).join(', ')}`);
// Pre-validate mutation candidates
const TYPES_FILE_PATH = path.join(PROMPT_SHAPES_DIR, 'types.ts');
for (const cand of candidates) {
if (cand.variant === 'baseline') continue;
const filename = `${cand.shape}-${cand.variant}.ts`;
const candPath = path.join(GEPA_EVOLVED_DIR, filename);
try {
const verdict = validateCandidate({
candidateShapeFilePath: candPath,
baselineShapeName: `${cand.shape}.ts`,
typesFilePath: TYPES_FILE_PATH,
expectShapeDiff: true,
});
log(`[validator] ${cand.candidateId} valid=${verdict.valid} violations=${verdict.violations.length}`);
} catch (e) {
log(`[validator] ${cand.candidateId} ERROR ${(e as Error).message}`);
}
}
if (args.mode === 'dry-run') {
log(`[dry-run] would run ${candidates.length} candidates × ${heldOutSample.length} instances = ${candidates.length * heldOutSample.length} evals`);
for (const cand of candidates) {
for (const inst of heldOutSample) {
log(`[dry-run] ${cand.candidateId}__${inst.instanceId}`);
}
}
return;
}
// Resume support
const existing = new Set<string>();
let cumulativeCost = 0;
if (fs.existsSync(OUT_JSONL)) {
for (const line of fs.readFileSync(OUT_JSONL, 'utf-8').trim().split(/\n+/).filter(Boolean)) {
try {
const r = JSON.parse(line) as EvalRecord;
existing.add(`${r.candidateId}__${r.instanceId}`);
cumulativeCost += r.evalCostUsd;
} catch { /* skip */ }
}
log(`[resume] loaded ${existing.size} existing evals; cumulative $${cumulativeCost.toFixed(4)}`);
}
const out = fs.createWriteStream(OUT_JSONL, { flags: existing.size > 0 ? 'a' : 'w' });
const embedder = createOllamaEmbedder({ baseUrl: OLLAMA_URL, model: EMBEDDER_MODEL });
const recordsByCandidate = new Map<string, EvalRecord[]>();
for (const cand of candidates) recordsByCandidate.set(cand.candidateId, []);
// Re-load existing records into recordsByCandidate
if (fs.existsSync(OUT_JSONL)) {
for (const line of fs.readFileSync(OUT_JSONL, 'utf-8').trim().split(/\n+/).filter(Boolean)) {
try {
const r = JSON.parse(line) as EvalRecord;
const list = recordsByCandidate.get(r.candidateId);
if (list) list.push(r);
} catch { /* skip */ }
}
}
let nDone = existing.size;
outer: for (const cand of candidates) {
for (const inst of heldOutSample) {
const key = `${cand.candidateId}__${inst.instanceId}`;
if (existing.has(key)) { log(`[skip] ${key} already in JSONL`); continue; }
if (cumulativeCost >= COST_HALT_USD) { log(`[HALT] cumulative $${cumulativeCost.toFixed(4)} >= $${COST_HALT_USD}`); break outer; }
const r = await runOneEval(cand, inst, embedder);
if ('error' in r) { log(`[skip] ${key}: ${r.error}`); continue; }
out.write(JSON.stringify(r) + '\n');
cumulativeCost += r.evalCostUsd;
nDone++;
const list = recordsByCandidate.get(cand.candidateId);
if (list) list.push(r);
log(`[cumulative] $${cumulativeCost.toFixed(4)} / $${COST_HALT_USD} halt; ${nDone} evals total`);
}
}
out.end();
// Write summary
const inSampleStats = loadInSampleStats(args.candidateIds);
const summary = buildCheckpointCSummary(args, candidates, heldOutSample, recordsByCandidate, inSampleStats, cumulativeCost);
fs.writeFileSync(SUMMARY_JSON, JSON.stringify(summary, null, 2));
log(`[summary] wrote ${SUMMARY_JSON}`);
log(`[F.2] verdict_confirmation=${summary.f2_verdict_confirmation} (${summary.f2_verdict_detail})`);
log(`[F.5] overfitting_bound=±${F5_OVERFITTING_BOUND_PP}pp`);
log(`[Faza 2] deployment_authorization=${summary.faza_2_deployment_authorization}`);
for (const c of summary.perCandidate) {
log(`[F.5] ${c.candidateId}: in-sample=${c.inSample ? (c.inSample.passIIRate * 100).toFixed(1) : 'n/a'}% held-out=${(c.heldOut.passIIRate * 100).toFixed(1)}% gap=${c.passIIGapPP.toFixed(1)}pp verdict=${c.f5_condition_2_verdict}`);
}
log(`[done] ${nDone} evals; total cost $${cumulativeCost.toFixed(4)}`);
}
main().catch(e => { console.error('FATAL:', e); process.exit(2); });

Some files were not shown because too many files have changed in this diff Show More