126 lines
5.2 KiB
Markdown
126 lines
5.2 KiB
Markdown
# @waggle/benchmarks-harness
|
||
|
||
**Four-cell ablation harness** for Waggle memory + prompt-evolution benchmarks.
|
||
Canonical model: **Qwen/Qwen3.6-35B-A3B** (see `docs/plans/BACKLOG-MASTER-2026-04-18.md` §H12).
|
||
|
||
## Four cells — the ablation grid
|
||
|
||
Same dataset, same seed, same model across all four. Difference at report
|
||
time between a baseline cell and a treatment cell isolates the causal
|
||
contribution of the ablated component.
|
||
|
||
| Cell | Memory | Prompt evolution | Purpose |
|
||
|------|--------|------------------|---------|
|
||
| `raw` | no | no | Baseline — LLM only, stateless per turn. |
|
||
| `filtered` | yes | no | Isolates the memory-layer contribution. (Sprint 12 rename: was `memory-only`.) |
|
||
| `compressed` | no | yes | Isolates the GEPA prompt-evolution contribution. (Sprint 12 rename: was `evolve-only`.) |
|
||
| `full-context` | yes | yes | Joint contribution (memory × evolution). (Sprint 12 rename: was `full-stack`.) |
|
||
|
||
## Controls (not cells — diagnostic)
|
||
|
||
| Control | Purpose |
|
||
|---------|---------|
|
||
| `verbose-fixed` | Day-1 sanity check. Tells the model to answer verbosely. On a short-factoid benchmark, this should **underperform** `raw`. If it doesn't, audit the harness. |
|
||
|
||
Out-of-scope for Week 1 (deferred to Week 2 / Week 3 per brief):
|
||
|
||
- `naive-rag` control
|
||
- `oracle-memory` ceiling
|
||
- Llama 3.1 8B + Opus 4.6 model integrations
|
||
- Gemma 2 9B probe
|
||
- Full τ-bench + LongMemEval loaders (synthetic fallback works today)
|
||
|
||
## CLI
|
||
|
||
Run from repo root via the `bench` script, or invoke `tsx` directly:
|
||
|
||
```bash
|
||
# Day 1 sanity check — one cell, one instance, dry-run (no LLM required)
|
||
npm run bench -- --cell raw --dataset locomo --limit 1 --model qwen3.6-35b-a3b
|
||
|
||
# Day 2 pre-flight smoke — all 4 cells, 50 instances each
|
||
npm run bench -- --all-cells --dataset locomo --limit 50 --model qwen3.6-35b-a3b --budget 115
|
||
|
||
# Full run
|
||
npm run bench -- --all-cells --dataset locomo --full --model qwen3.6-35b-a3b
|
||
|
||
# Verbose-fixed control (Day 1 sanity)
|
||
npm run bench -- --control verbose-fixed --dataset locomo --limit 50 --model qwen3.6-35b-a3b
|
||
```
|
||
|
||
### Flags
|
||
|
||
| Flag | Default | Notes |
|
||
|------|---------|-------|
|
||
| `--cell <name>` | — | One of `raw \| filtered \| compressed \| full-context`. |
|
||
| `--all-cells` | — | Run all four sequentially with the same dataset + seed. |
|
||
| `--control <name>` | — | Currently only `verbose-fixed`. |
|
||
| `--dataset <id>` | `synthetic` | `synthetic \| locomo \| longmemeval`. External datasets throw `DatasetMissingError` if the canonical archive is absent; set `BENCH_SYNTHETIC_DATASET=1` to re-enable the dev-only synthetic fallback. |
|
||
| `--limit N` | `10` | Cap instances. `--full` = no cap. |
|
||
| `--model <id>` | `qwen3.6-35b-a3b` | Id from `config/models.json`. |
|
||
| `--seed N` | `42` | Reproducibility — same seed → same instance order + dry-run output. |
|
||
| `--budget USD` | `Infinity` | Hard USD cap. Run stops when cumulative cost exceeds. |
|
||
| `--output <path>` | auto | JSONL output path. Default: `../results/<cell>-<dataset>-<ts>.jsonl`. |
|
||
| `--dry-run` | auto | Stub LLM. Default-on when `LITELLM_URL` env is unset. |
|
||
| `--live` | — | Force real LLM calls even if `LITELLM_URL` unset. |
|
||
| `--help`, `-h` | — | Usage summary. |
|
||
|
||
### Environment
|
||
|
||
| Var | Default |
|
||
|-----|---------|
|
||
| `LITELLM_URL` | `http://localhost:4000` |
|
||
| `LITELLM_API_KEY` | `sk-waggle-dev` |
|
||
|
||
## Output — per-instance JSONL
|
||
|
||
One line per instance, flat shape (friendly to `jq`, DuckDB, pandas):
|
||
|
||
```jsonl
|
||
{"turnId":"9fcd1d25-979f-4094-9633-f9bc30471f08","cell":"raw","instance_id":"synthetic_001","model":"qwen3.6-35b-a3b","seed":42,"accuracy":1,"p50_latency_ms":1,"p95_latency_ms":1,"usd_per_query":0.00012,"failure_mode":null}
|
||
```
|
||
|
||
**turnId correlation** — the `turnId` field is a UUID v4 matching the
|
||
per-turn trace ID generated by the production agent orchestrator
|
||
(`packages/agent/src/turn-context.ts`, H-AUDIT-1). For `raw` and control
|
||
cells, where the agent loop isn't exercised, the harness generates the
|
||
turnId itself so every row has a correlation key.
|
||
|
||
## Aggregate summary
|
||
|
||
Written alongside the JSONL as `<name>.summary.json`:
|
||
|
||
```json
|
||
{
|
||
"run": { "kind": "cell", "name": "raw", "dataset": "synthetic", "model": "qwen3.6-35b-a3b", "seed": 42, ... },
|
||
"counts": { "total": 50, "completed": 50, "failed": 0, "budgetStoppedAt": null },
|
||
"metrics": { "meanAccuracy": 0.82, "p50LatencyMs": 230, "p95LatencyMs": 450, "totalUsd": 0.011, "meanUsdPerQuery": 0.00022 },
|
||
"failureModes": {}
|
||
}
|
||
```
|
||
|
||
## Datasets
|
||
|
||
- `synthetic` — **built-in**, 60 instances, no external download required. Used by smoke tests and as a fallback when external data is missing.
|
||
- `locomo` — expects `benchmarks/data/locomo/locomo.jsonl`. Gitignored. Downloaded separately (Week 1 work).
|
||
- `longmemeval` — expects `benchmarks/data/longmemeval/longmemeval.jsonl`. Same pattern.
|
||
|
||
## Reproducibility
|
||
|
||
`--seed N` fully determines:
|
||
|
||
1. The order instances are sampled from the dataset
|
||
2. The dry-run LLM stub's output (the stub is deterministic from the user prompt alone)
|
||
|
||
For live LLM runs, `--seed` is still emitted into every JSONL record so
|
||
downstream analysis can pin every row to a seed value; the model's own
|
||
sampling is controlled via `temperature=0.0` in `src/llm.ts`.
|
||
|
||
## Smoke test
|
||
|
||
```bash
|
||
npx vitest run benchmarks/harness/tests/smoke.test.ts
|
||
```
|
||
|
||
Covers every acceptance criterion from the Bucket 1 Task 7 brief.
|