# @waggle/benchmarks-harness

**Four-cell ablation harness** for Waggle memory + prompt-evolution benchmarks.
Canonical model: **Qwen/Qwen3.6-35B-A3B** (see `docs/plans/BACKLOG-MASTER-2026-04-18.md` §H12).

## Four cells — the ablation grid

Same dataset, same seed, same model across all four. Difference at report
time between a baseline cell and a treatment cell isolates the causal
contribution of the ablated component.

| Cell | Memory | Prompt evolution | Purpose |
|------|--------|------------------|---------|
| `raw` | no | no | Baseline — LLM only, stateless per turn. |
| `filtered` | yes | no | Isolates the memory-layer contribution. (Sprint 12 rename: was `memory-only`.) |
| `compressed` | no | yes | Isolates the GEPA prompt-evolution contribution. (Sprint 12 rename: was `evolve-only`.) |
| `full-context` | yes | yes | Joint contribution (memory × evolution). (Sprint 12 rename: was `full-stack`.) |

## Controls (not cells — diagnostic)

| Control | Purpose |
|---------|---------|
| `verbose-fixed` | Day-1 sanity check. Tells the model to answer verbosely. On a short-factoid benchmark, this should **underperform** `raw`. If it doesn't, audit the harness. |

Out-of-scope for Week 1 (deferred to Week 2 / Week 3 per brief):

- `naive-rag` control
- `oracle-memory` ceiling
- Llama 3.1 8B + Opus 4.6 model integrations
- Gemma 2 9B probe
- Full τ-bench + LongMemEval loaders (synthetic fallback works today)

## CLI

Run from repo root via the `bench` script, or invoke `tsx` directly:

```bash
# Day 1 sanity check — one cell, one instance, dry-run (no LLM required)
npm run bench -- --cell raw --dataset locomo --limit 1 --model qwen3.6-35b-a3b

# Day 2 pre-flight smoke — all 4 cells, 50 instances each
npm run bench -- --all-cells --dataset locomo --limit 50 --model qwen3.6-35b-a3b --budget 115

# Full run
npm run bench -- --all-cells --dataset locomo --full --model qwen3.6-35b-a3b

# Verbose-fixed control (Day 1 sanity)
npm run bench -- --control verbose-fixed --dataset locomo --limit 50 --model qwen3.6-35b-a3b
```

### Flags

| Flag | Default | Notes |
|------|---------|-------|
| `--cell <name>` | — | One of `raw \| filtered \| compressed \| full-context`. |
| `--all-cells` | — | Run all four sequentially with the same dataset + seed. |
| `--control <name>` | — | Currently only `verbose-fixed`. |
| `--dataset <id>` | `synthetic` | `synthetic \| locomo \| longmemeval`. External datasets throw `DatasetMissingError` if the canonical archive is absent; set `BENCH_SYNTHETIC_DATASET=1` to re-enable the dev-only synthetic fallback. |
| `--limit N` | `10` | Cap instances. `--full` = no cap. |
| `--model <id>` | `qwen3.6-35b-a3b` | Id from `config/models.json`. |
| `--seed N` | `42` | Reproducibility — same seed → same instance order + dry-run output. |
| `--budget USD` | `Infinity` | Hard USD cap. Run stops when cumulative cost exceeds. |
| `--output <path>` | auto | JSONL output path. Default: `../results/<cell>-<dataset>-<ts>.jsonl`. |
| `--dry-run` | auto | Stub LLM. Default-on when `LITELLM_URL` env is unset. |
| `--live` | — | Force real LLM calls even if `LITELLM_URL` unset. |
| `--help`, `-h` | — | Usage summary. |

### Environment

| Var | Default |
|-----|---------|
| `LITELLM_URL` | `http://localhost:4000` |
| `LITELLM_API_KEY` | `sk-waggle-dev` |

## Output — per-instance JSONL

One line per instance, flat shape (friendly to `jq`, DuckDB, pandas):

```jsonl
{"turnId":"9fcd1d25-979f-4094-9633-f9bc30471f08","cell":"raw","instance_id":"synthetic_001","model":"qwen3.6-35b-a3b","seed":42,"accuracy":1,"p50_latency_ms":1,"p95_latency_ms":1,"usd_per_query":0.00012,"failure_mode":null}
```

**turnId correlation** — the `turnId` field is a UUID v4 matching the
per-turn trace ID generated by the production agent orchestrator
(`packages/agent/src/turn-context.ts`, H-AUDIT-1). For `raw` and control
cells, where the agent loop isn't exercised, the harness generates the
turnId itself so every row has a correlation key.

## Aggregate summary

Written alongside the JSONL as `<name>.summary.json`:

```json
{
  "run": { "kind": "cell", "name": "raw", "dataset": "synthetic", "model": "qwen3.6-35b-a3b", "seed": 42, ... },
  "counts": { "total": 50, "completed": 50, "failed": 0, "budgetStoppedAt": null },
  "metrics": { "meanAccuracy": 0.82, "p50LatencyMs": 230, "p95LatencyMs": 450, "totalUsd": 0.011, "meanUsdPerQuery": 0.00022 },
  "failureModes": {}
}
```

## Datasets

- `synthetic` — **built-in**, 60 instances, no external download required. Used by smoke tests and as a fallback when external data is missing.
- `locomo` — expects `benchmarks/data/locomo/locomo.jsonl`. Gitignored. Downloaded separately (Week 1 work).
- `longmemeval` — expects `benchmarks/data/longmemeval/longmemeval.jsonl`. Same pattern.

## Reproducibility

`--seed N` fully determines:

1. The order instances are sampled from the dataset
2. The dry-run LLM stub's output (the stub is deterministic from the user prompt alone)

For live LLM runs, `--seed` is still emitted into every JSONL record so
downstream analysis can pin every row to a seed value; the model's own
sampling is controlled via `temperature=0.0` in `src/llm.ts`.

## Smoke test

```bash
npx vitest run benchmarks/harness/tests/smoke.test.ts
```

Covers every acceptance criterion from the Bucket 1 Task 7 brief.