222 lines
11 KiB
Markdown
222 lines
11 KiB
Markdown
# Pillar 1 — Qwen 3.6 35B-A3B follow-up runbook
|
|
|
|
**Goal:** API-served Qwen 3.6 number for the Waggle harness on GAIA 2 search split.
|
|
Repeat the **2026-05-22 Pillar-1 ON-PAR run** (Waggle harness + Sonnet 4.6 trio-strict **86.5% at N=40**) but with **Qwen 3.6 35B-A3B via LiteLLM → DashScope-intl direct** instead of Sonnet via OpenRouter.
|
|
|
|
## 2026-05-26 PIVOT — API-served, not local
|
|
|
|
This runbook originally scaffolded a **local-Ollama** path. We pivoted to **API-served via LiteLLM** on 2026-05-26 because (1) the LiteLLM proxy already holds all credentials, (2) DashScope-intl direct delivers the TRUE Qwen 3.6 (the OpenRouter route silently regresses to Qwen 3.5 per `models.json:43`), and (3) eliminates the gateway-confound caveat from the Sonnet baseline by routing both agent and judge through one proxy.
|
|
|
|
The sovereign-local variant is preserved at the bottom as an optional follow-up.
|
|
|
|
## Smoke verdict (2026-05-26 17:14)
|
|
|
|
Plumbing **PASS**. Single-scenario judge **inconclusive** on the known-flaky `21_1afh09`:
|
|
|
|
| Signal | Result |
|
|
|---|---|
|
|
| LiteLLM → DashScope-intl auth | ✅ keys resolved via `LITELLM_MASTER_KEY` |
|
|
| Worker → ARE adapter socket | ✅ scenario lifecycle clean |
|
|
| Qwen 3.6 agent engagement | ✅ **28 agent events**, coherent multi-step reasoning ("count 31 Shanghai contacts → sum ages 1402 → avg 45.23 → round 45") |
|
|
| Judge route via LiteLLM | ✅ ran, returned `inconclusive` (not "wrong") |
|
|
| Cost | ✅ ~$0.05 |
|
|
|
|
Scenario `21_1afh09` was flagged in `PILLAR1-WAGGLE-VS-HERMES-N40-2026-05-22.md` as "flipped between runs" — its `user_message_checker` is unstable on phrasing. Trio-rejudge typically resolves these. **Smoke is GREEN.**
|
|
|
|
---
|
|
|
|
## Why this is single-knob
|
|
|
|
Worker (`waggle-container/waggle_worker.mjs`) is fully env-driven. Runner `container_env.py` `_DEFAULT` profile injects from the TOML `[agent]` block into the container:
|
|
|
|
| Env var | Sonnet baseline (2026-05-22) | Qwen 3.6 (this run) |
|
|
|---|---|---|
|
|
| `MODEL` | `claude-sonnet-4-6` | `qwen3.6-35b-a3b` (LiteLLM alias) |
|
|
| `BASE_URL` | `https://openrouter.ai/api/v1` | `http://host.docker.internal:4000/v1` (LiteLLM proxy) |
|
|
| `API_KEY` | OpenRouter key | `sk-waggle-dev` (LiteLLM master key) |
|
|
|
|
LiteLLM alias `qwen3.6-35b-a3b` routes to `openai/qwen3.6-35b-a3b` at `https://dashscope-intl.aliyuncs.com/compatible-mode/v1` (`litellm-config.yaml:211` — Sprint 12 sibling of the deprecated `via-openrouter` route).
|
|
|
|
Same `AGENTS.md` renders, same single `terminal` tool, same `runAgentLoop`. **Only the model differs.** Strict fairness with the 86.5% Sonnet baseline.
|
|
|
|
---
|
|
|
|
## Prerequisites (all verified 2026-05-26)
|
|
|
|
1. **LiteLLM proxy stack up.** From repo root: `docker-compose up -d` brings up `litellm` (port 4000), `postgres`, `redis`, `minio`. Confirm: `curl http://localhost:4000/health` returns auth-required error (= healthy + listening).
|
|
2. **`localhost/gaia2-waggle:latest` built.** Verify: `docker images | grep gaia2-waggle`. Rebuild via `waggle-container/BUILD.md` if the worker or `@waggle/agent/dist` has changed since last build.
|
|
3. **`LITELLM_MASTER_KEY` set in shell.** Default `sk-waggle-dev` per `.env` and `docker-compose.yml`.
|
|
4. **gaia2-runner available.** At `external/meta-agents-research-environments/gaia2-cli/runner/.venv/Scripts/gaia2-runner.exe` (Windows) — installed via `uv sync --frozen`.
|
|
5. **GAIA 2 dataset cached.** First run pulls from HuggingFace. Cached at `~/.cache/gaia2/hf_datasets/`.
|
|
|
|
---
|
|
|
|
## TOMLs
|
|
|
|
Working copies live in `external/meta-agents-research-environments/gaia2-cli/runner/examples/` (`.gitignored` per the existing pattern; embedded verbatim below for reproduction on any fresh checkout).
|
|
|
|
### Smoke (n=1) — `waggle_qwen36_smoke_n1.toml`
|
|
|
|
```toml
|
|
[target]
|
|
dataset = "meta-agents-research-environments/gaia2-cli"
|
|
splits = ["search"]
|
|
limit = 1
|
|
|
|
[agent]
|
|
image = "localhost/gaia2-waggle:latest"
|
|
runtime = "docker"
|
|
provider = "openai-compat"
|
|
model = "qwen3.6-35b-a3b"
|
|
api_key_env = "LITELLM_MASTER_KEY"
|
|
base_url = "http://host.docker.internal:4000/v1"
|
|
thinking = "high"
|
|
|
|
[judge]
|
|
provider = "openai-compat"
|
|
model = "claude-sonnet-4-6"
|
|
api_key_env = "LITELLM_MASTER_KEY"
|
|
base_url = "http://localhost:4000/v1"
|
|
|
|
[run]
|
|
timeout = 1800
|
|
health_timeout = 180
|
|
concurrency = 1
|
|
pass_at = 1
|
|
output_dir = "D:/Projects/waggle-os-gaia2-wt/benchmarks/gaia2/runs/waggle-qwen36-smoke-n1"
|
|
log_level = "INFO"
|
|
```
|
|
|
|
### Full N=160 — `waggle_qwen36_n160.toml`
|
|
|
|
```toml
|
|
[target]
|
|
dataset = "meta-agents-research-environments/gaia2-cli"
|
|
splits = ["search"]
|
|
limit = 160
|
|
|
|
[agent]
|
|
image = "localhost/gaia2-waggle:latest"
|
|
runtime = "docker"
|
|
provider = "openai-compat"
|
|
model = "qwen3.6-35b-a3b"
|
|
api_key_env = "LITELLM_MASTER_KEY"
|
|
base_url = "http://host.docker.internal:4000/v1"
|
|
thinking = "high"
|
|
|
|
[judge]
|
|
provider = "openai-compat"
|
|
model = "claude-sonnet-4-6"
|
|
api_key_env = "LITELLM_MASTER_KEY"
|
|
base_url = "http://localhost:4000/v1"
|
|
|
|
[run]
|
|
timeout = 1800
|
|
health_timeout = 180
|
|
concurrency = 2
|
|
pass_at = 1
|
|
output_dir = "D:/Projects/waggle-os-gaia2-wt/benchmarks/gaia2/runs/waggle-qwen36-n160"
|
|
log_level = "INFO"
|
|
```
|
|
|
|
Difference: `limit` and `concurrency` only. Output dirs distinct.
|
|
|
|
---
|
|
|
|
## Run sequence
|
|
|
|
```powershell
|
|
# 0. Confirm LiteLLM stack
|
|
docker ps --format '{{.Names}} ({{.Status}})' | Select-String 'litellm|postgres|redis|minio'
|
|
|
|
# 1. Set the master key for the runner's host shell
|
|
$env:LITELLM_MASTER_KEY = "sk-waggle-dev"
|
|
|
|
# 2. Smoke first (~3-5 min wall, ~$0.05)
|
|
cd D:/Projects/waggle-os/external/meta-agents-research-environments/gaia2-cli
|
|
& "runner/.venv/Scripts/gaia2-runner.exe" run-config `
|
|
--config "runner/examples/waggle_qwen36_smoke_n1.toml"
|
|
|
|
# 3. Inspect smoke
|
|
Get-Content D:/Projects/waggle-os-gaia2-wt/benchmarks/gaia2/runs/waggle-qwen36-smoke-n1/results.jsonl
|
|
|
|
# 4. If smoke clean (plumbing + non-zero agent events) -> full run (~3-5 hrs wall, ~$5-8)
|
|
& "runner/.venv/Scripts/gaia2-runner.exe" run-config `
|
|
--config "runner/examples/waggle_qwen36_n160.toml"
|
|
|
|
# 5. Trio-strict rejudge
|
|
cd D:/Projects/waggle-os-gaia2-wt/benchmarks/gaia2
|
|
python rejudge_user_message.py `
|
|
--runs-dir runs/waggle-qwen36-n160 `
|
|
--judges opus-4-7,gpt-5-4,gemini-2.5-pro `
|
|
--output runs/rejudge-waggle-qwen36-n160.jsonl
|
|
```
|
|
|
|
---
|
|
|
|
## Gate criteria
|
|
|
|
| Gate | Pass condition |
|
|
|---|---|
|
|
| **Smoke plumbing** | LiteLLM auth resolves; container completes lifecycle; agent emits ≥1 event; judge runs (even if inconclusive). The 28-event scenario `21_1afh09` smoke met this on 2026-05-26. |
|
|
| **Full run health** | error-rate < 10% (per-scenario container exits or judge errors). Concurrency=2 means 2 docker containers + 2 simulated app daemons in parallel — `entrypoint.log` per scenario captures issues. |
|
|
| **Headline** | trio-strict score with 95% CI. **No threshold** — this is "where does API-served Qwen 3.6 land," not a hypothesis test. Sonnet ON-PAR (86.5%, N=40, trio-strict) is the comparator. A trio-strict in the **75-85% band** would be a strong sovereign-eligible result; lower than that frames Pillar 1 as model-bound and worth focused harness work. |
|
|
|
|
---
|
|
|
|
## Known risks & open assumptions
|
|
|
|
1. **`runAgentLoop` against LiteLLM-proxied DashScope** — smoke proved roundtrip works at the OpenAI-compat surface, with tool calls flowing correctly. Full-N parallelism is the next stress test: 2 containers competing for LiteLLM throughput shouldn't hit rate limits at N=160 / 2 = 80 sequential calls per stream.
|
|
|
|
2. **DashScope 16K thinking-tokens cap** (per `models.json:55`, Stage 2 Mini Retry v3 manifest §2.1) — reduced from 64K to avoid tail-latency timeouts. If the agent loop blows through this on multi-step scenarios, expect `loop_exhausted` failures similar to the 2026-04-30 dry-run-results-memo §3 finding.
|
|
|
|
3. **Concurrency=2 GPU/cost contention** — DashScope is API-served, no local GPU constraint. Cost stays linear with N (~$5-8 total at $0.20+$0.80/M).
|
|
|
|
4. **No prompt-shape change** — Waggle's GAIA 2 worker uses `AGENTS.md` + single `terminal` tool. The `qwen-thinking.ts` PromptShape lives in `packages/agent/src/prompt-shapes/` and is a LoCoMo retrieval framework, **not** the GAIA 2 shape. Do not wire it in here — that would break fairness with the 86.5% Sonnet baseline.
|
|
|
|
5. **Scenario `21_1afh09` flakiness** — known unstable per the N=40 memo. Single-instance failure carries no signal; trio-rejudge resolves most "inconclusive" verdicts.
|
|
|
|
---
|
|
|
|
## Comparator targets
|
|
|
|
| Cell | N | trio-strict | source |
|
|
|---|---:|---:|---|
|
|
| Hermes (reference) + Sonnet 4.6 | 40 (matched subset of 160) | **89.2%** | `PILLAR1-WAGGLE-VS-HERMES-N40-2026-05-22.md` |
|
|
| **Waggle + Sonnet 4.6** | 40 | **86.5%** | same memo |
|
|
| Hermes + Sonnet 4.6 | 160 (full search) | 83.8% strict / 86.5% judged-only | `PHASE-4-P4.5-RESULTS-N160-2026-05-22.md` |
|
|
| **Waggle + Qwen 3.6 35B-A3B (API via LiteLLM→DashScope)** | 160 | **TBD** | this runbook (fired 2026-05-26) |
|
|
|
|
The third row is the natural comparator for the new API-served Qwen 3.6 number. A defensible launch claim: *"Waggle harness lands within ε of the Hermes-Sonnet ceiling even with a sovereign-eligible 35B model"* — provided ε is small enough. Defining "small enough" is the next round of analysis once the trio-strict number is in.
|
|
|
|
---
|
|
|
|
## Optional follow-up: sovereign-local variant via Ollama
|
|
|
|
If a fully-local number is later needed (no cloud API surface), the variant just swaps `[agent]`:
|
|
|
|
```toml
|
|
[agent]
|
|
image = "localhost/gaia2-waggle:latest"
|
|
runtime = "docker"
|
|
provider = "openai-compat"
|
|
model = "qwen2.5:32b" # FIXME: confirm exact Ollama tag for Qwen 3.6 35B thinking
|
|
api_key_env = "OPENAI_COMPAT_API_KEY" # set to any non-empty string ("ollama" works)
|
|
base_url = "http://host.docker.internal:11434/v1"
|
|
thinking = "high"
|
|
```
|
|
|
|
Prerequisites: `ollama pull <qwen-3.6-thinking-tag>`, plus `concurrency=1` (single GPU serializes) → wall time ~3-15h instead of ~3-5h. Cost is $0 on agent side, judge stays the same (~$2 in-container Sonnet). Output dir suffix changes to `-ollama-local` to keep the API-served and sovereign-local runs distinct.
|
|
|
|
---
|
|
|
|
## Provenance
|
|
|
|
- Initial scaffold 2026-05-26 (Ollama-only): commit `b4e4354` on `feature/gaia2-are-setup`.
|
|
- 2026-05-26 PM pivot to API-served via LiteLLM → DashScope: this revision.
|
|
- Smoke fired 2026-05-26 17:14 (task ID `benccwrgl`): plumbing PASS, scenario `21_1afh09` judge inconclusive (known flaky).
|
|
- N=160 fired 2026-05-26 17:19 (task ID `blt6winb3`): in progress at the time this runbook update was written; result memo to follow under `PILLAR1-QWEN36-N160-RESULT-2026-05-26.md`.
|
|
- Worker entry: `benchmarks/gaia2/waggle-container/waggle_worker.mjs` (HEAD `6479dfa`).
|
|
- ARE container_env: `external/meta-agents-research-environments/gaia2-cli/runner/gaia2_runner/container_env.py` `_DEFAULT` profile.
|
|
- LiteLLM alias source: `litellm-config.yaml:211` (Sprint 12 Task 2 C3 Stage 2 mini).
|