Files
waggle-os/ops/litellm/README.md
shumeiko.it 16eaf01e14 chore: scrub personal names, maintainer URLs and local paths (P2.2)
Sample data in tests, prompts and UI placeholders uses Alex Morgan
instead of the maintainer's name; decision attributions say "owner".
Personal details in two test fixtures are replaced. Local paths
(D:/Projects/...) and paths into the maintainer's planning repository
are reworded. Clone and install URLs in CONTRIBUTING.md and the getting
started guide use placeholders until the public repository location is
decided (Q9). The Benchmarks page no longer names the maintainer's
hive-mind repository, and the upstream fixes log names the main Waggle
repository instead of an internal git host.
2026-11-01 00:50:13 +01:00

162 lines
7.6 KiB
Markdown

# LiteLLM Routing — Operational Notes
Operational notes + migration log for `litellm-config.yaml` route changes
where the rationale isn't obvious from the config diff alone. Keep this
file narrow: **only entries where a future reader would otherwise not
know why we chose a particular upstream slug or fallback policy.**
---
## Sprint 10 Task 1.4 — DashScope intl primary + OR failover (2026-04-21)
**Key provisioning.** A classic DashScope Model Studio API key (`sk-…`,
not `sk-ws-…`) was provisioned on 2026-04-21 and is held in two places:
- `.env` under `DASHSCOPE_API_KEY=sk-…` (LiteLLM container reads at
startup via `os.environ/DASHSCOPE_API_KEY`).
- Waggle vault under name `alibaba` (credentialType `api_key`).
The key is bound to the **international tenant**, not mainland. Smoke
verification (2026-04-21):
| Endpoint | Result |
|---|---|
| `https://dashscope-intl.aliyuncs.com/compatible-mode/v1/chat/completions` | HTTP 200, target `qwen3.6-35b-a3b` returns valid completion with `reasoning_content` populated (thinking default on) |
| `https://dashscope.aliyuncs.com/compatible-mode/v1/chat/completions` | HTTP 401 `invalid_api_key` — key is intl-bound |
**Routing decision.** `litellm-config.yaml` now carries three Qwen3.6
routes:
| Alias | Upstream | Role |
|---|---|---|
| `qwen3.6-35b-a3b` | `openai/qwen3.6-35b-a3b` @ `dashscope-intl.aliyuncs.com/compatible-mode/v1` | **Canonical primary** — real 3.6 model |
| `qwen3.6-35b-a3b-via-dashscope` | identical to canonical | **Explicit DashScope pin** — for operator scripts that want to bypass a canonical alias flip |
| `qwen3.6-35b-a3b-via-openrouter` | `openrouter/qwen/qwen3.5-35b-a3b` | **Failover** — one-minor regression to 3.5 (OR catalog lacks 3.6-35b-a3b as of 2026-04-21) |
**Failover policy.** Caller-side responsibility, not LiteLLM router:
1. First call: `qwen3.6-35b-a3b` (canonical, DashScope-intl).
2. On HTTP 429 rate-limit or 5xx within a reasonable retry window
(client-judgment), retry on `qwen3.6-35b-a3b-via-openrouter`.
3. OR path returns Qwen3.5-35B-A3B, so log the model substitution in
the caller's trace record so Week-1 / Stage-2 aggregate reports
attribute correctly. The `judge-runner.ts` trace already carries
`judge_model`, so a failover just writes the failover model name
to that field; no schema change needed.
4. Do NOT use `openrouter/auto` as a third-tier failover — it can
silently route to an unrelated model and pollute cost accounting.
LiteLLM's per-route `fallbacks` field is supported (would let us
declare failover in config rather than caller logic) but deliberately
not used here: the Week-1 / Stage-2 trace schema relies on precise
per-call model attribution, and letting LiteLLM transparently swap
models would make traces harder to audit.
**Regression gate.** `scripts/smoke-qwen-dual-route.mjs` calls both
routes with the same prompt, verifies both return HTTP 200, logs
per-route completion, latency, and token counts. Task 1.1 (Qwen
stability matrix) uses the canonical alias for its live run — which
means the matrix now exercises the real 3.6 model, not the 3.5 OR
fallback. Materially improves Stage-2-prep defensibility.
**Follow-up for the owner (when available).** If the key later gets mainland
provisioning too, a second route `qwen3.6-35b-a3b-via-dashscope-mainland`
can be added with the same key pattern — but the intl endpoint is
sufficient for Sprint 10 + Stage 2 benchmarks.
---
## Sprint 10 Task 1.2 — Sonnet 4.6 dated-snapshot repair (2026-04-21)
**Problem.** The `claude-sonnet-4-6` alias previously routed to
`anthropic/claude-sonnet-4-6-20250514`. That dated-snapshot suffix was
never a valid Claude API ID for the Sonnet 4.6 family.
**Where it was caught.** Sprint 9 Task 4 calibration (commit `f9b98aa`)
attempted to exercise the Sonnet route and LiteLLM returned an error
consistent with a `model_not_found` response. The Sprint 9 session
substituted `claude-opus-4-7` for the calibration gate (10/10 PASS) and
queued the Sonnet route repair as Sprint-10 operational work.
**Verification.** Re-fetched the Anthropic models overview on
2026-04-21 (docs redirect target:
<https://platform.claude.com/docs/en/docs/about-claude/models/overview>).
Current table confirms:
- **Claude Sonnet 4.6** — `Claude API ID: claude-sonnet-4-6`, `Claude API
alias: claude-sonnet-4-6`.
- No valid dated-snapshot with the `-20250514` suffix exists for the
Sonnet 4.6 family. The `-20250514` snapshot belongs to the deprecated
Claude **Sonnet 4** (single-digit) family, not Sonnet 4.6, and that
family is scheduled for retirement on 2026-06-15 per the public
deprecation notice.
**Fix.** Repointed the native Anthropic target from
`anthropic/claude-sonnet-4-6-20250514` to the plain alias
`anthropic/claude-sonnet-4-6`, with matching fixes to the two legacy
aliases (`claude-3-5-sonnet-20241022` and `anthropic/claude-sonnet-4-6`)
that routed to the same broken target.
**Scope discipline.** The adjacent `claude-opus-4-6-20250610` route uses
the same dated-snapshot pattern and is therefore a candidate for the
same defect. It is **out of scope for Sprint 10 Task 1.2** — Sprint 10
runs judges on `claude-opus-4-7` (verified good in Sprint 9) and
`claude-sonnet-4-6` (this PR). Opus 4.6 will be audited separately only
if/when a caller needs it. No speculative edits.
**Regression gate.** `scripts/smoke-sonnet-route.mjs` is the one-shot
HTTP probe that verifies the repaired route returns 200 (not 404
`model_not_found`) on the LiteLLM proxy. Task 1.3 (judge calibration on
Sonnet) is the full functional regression — it replays the 10
ground-truth triples through the repaired route and reports match rate
vs. PM human labels.
**PM review gate (open).** Per Sprint 10 brief §1.2, the owner ratifies the
mapping before this PR merges. Open question for the reviewer: do we
also want a separate PR to audit the Opus-4.6 dated snapshot, or leave
that out until a caller trips on it?
---
## Existing route conventions (carried forward)
### Dated snapshots vs plain aliases
Prefer plain family aliases (`claude-opus-4-7`, `claude-sonnet-4-6`)
unless a specific dated snapshot is required for reproducibility. Plain
aliases automatically follow Anthropic's "current" pointer within a
family and survive version bumps without requiring config churn.
Dated snapshots are stable indefinitely once live but carry two risks:
1. Anthropic can decommission a dated snapshot without migrating the
alias forward — the 2026-04-21 incident (Task 1.2 above) is a
canonical example.
2. Dated snapshot IDs are easy to invent with the wrong date format. A
404 on a dated ID always warrants re-verification against the live
docs before trusting the old value.
### OpenRouter bridge routes
Several routes of the form `<provider>-<model>-via-openrouter` exist as
parallel paths through OpenRouter for providers where the native key is
not provisioned or where OR has a more stable upstream (notably Kimi
K2.x, Qwen3.x families). Keep the primary alias pointing at the native
provider when possible; OR routes are failover, not default.
### `drop_params: true`
LiteLLM's global `drop_params` setting strips unsupported params. That
includes provider-specific extras like Anthropic's `thinking` or
DashScope's `extra_body.enable_thinking`. If a caller needs a param that
LiteLLM would otherwise drop, either (a) route around LiteLLM for that
specific call (see `scripts/stage-0-query.mjs` for the pattern), or
(b) pass the param through a route that uses the provider-native SDK.
---
*Maintained as part of Sprint 10 operational queue close-out. Extend
this file with a new `##` section per route change that needs
operational-memory preservation.*