moving

benchmarks/gaia2/waggle-container/BUILD.md

@@ -0,0 +1,72 @@
+# gaia2-waggle — build & run (Pillar 1: Waggle harness in the GAIA 2 rig)
+
+Wraps **Waggle's own `runAgentLoop`** in the GAIA 2 ARE, fair vs `gaia2-hermes`:
+same single `terminal` tool + same `AGENTS.md` + same model (Sonnet 4.6). Only the
+loop logic differs. Path-A: no native deps (2-symbol `@waggle/core` stub — proven in
+`../spike-waggle-worker/`).
+
+## Files here (tracked recovery copies; staged into `external/.../gaia2-cli/containers/waggle/`)
+- `Dockerfile` — gaia2-cli base + Node + waggle payload (models gaia2-hermes)
+- `waggle_worker.mjs` — Node worker: socket protocol + terminal tool + runAgentLoop
+- `entrypoint.sh` — launches the Node worker (vs hermes' python worker)
+- `stub-core/` — the 2-symbol `@waggle/core` stub package
+
+## Build steps (from `external/meta-agents-research-environments/gaia2-cli/`)
+
+```bash
+WC=containers/waggle
+mkdir -p $WC/payload/node_modules/@waggle
+
+# 1. entrypoints
+cp <thisdir>/Dockerfile $WC/Dockerfile
+cp <thisdir>/entrypoint.sh $WC/entrypoint.sh
+cp <thisdir>/waggle_worker.mjs $WC/payload/waggle_worker.mjs
+echo '{"type":"module"}' > $WC/payload/package.json
+# init-entrypoint is hermes' VERBATIM (its hermes-isms are inert; it execs /opt/entrypoint.sh
+# which the waggle Dockerfile overrides):
+cp containers/hermes/gaia2-init-entrypoint.sh $WC/gaia2-init-entrypoint.sh
+
+# 2. @waggle/agent dist (built). NOTE: do NOT copy the source package.json — its
+#    exports map ({".":"./src/index.ts"}) blocks the /dist/ subpath import
+#    (ERR_PACKAGE_PATH_NOT_EXPORTED). Write a minimal one with no exports field.
+mkdir -p $WC/payload/node_modules/@waggle/agent
+cp -r D:/Projects/waggle-os/packages/agent/dist $WC/payload/node_modules/@waggle/agent/dist
+echo '{"name":"@waggle/agent","version":"0.0.0","type":"module"}' > $WC/payload/node_modules/@waggle/agent/package.json
+
+# 3. @waggle/core stub
+cp -r <thisdir>/stub-core $WC/payload/node_modules/@waggle/core
+
+# 4. @waggle/hive-mind-core — ONLY the DB-free deep modules the stub re-exports.
+#    Verify logger.js + injection-scanner.js have no further @waggle deps (they don't as of 2026-05-22).
+mkdir -p $WC/payload/node_modules/@waggle/hive-mind-core/dist
+cp D:/Projects/waggle-os/packages/hive-mind-core/dist/logger.js \
+   D:/Projects/waggle-os/packages/hive-mind-core/dist/injection-scanner.js \
+   $WC/payload/node_modules/@waggle/hive-mind-core/dist/
+echo '{"name":"@waggle/hive-mind-core","version":"0.0.0","type":"module"}' \
+   > $WC/payload/node_modules/@waggle/hive-mind-core/package.json
+
+# 5. build — NOTE: local base image is `localhost/gaia2-cli:local` (tag "local", not latest).
+#    Use the LEGACY builder (DOCKER_BUILDKIT=0): BuildKit treats `localhost/` as a remote
+#    registry and times out; the legacy builder reads the local image store directly.
+DOCKER_BUILDKIT=0 docker build --build-arg GAIA2_CLI_VERSION=local \
+  -f $WC/Dockerfile -t localhost/gaia2-waggle:latest .
+```
+
+## Run (low-N probe — fair vs hermes)
+Create `runner/examples/waggle_harness_probe.toml` cloning `waggle_smoke_hermes_n3.toml`
+but `image = "localhost/gaia2-waggle:latest"`, `limit = 10`. Then:
+```bash
+gaia2-runner run-config --config examples/waggle_harness_probe.toml
+```
+Compare strict + trio-strict (re-judge via `rejudge_user_message.py`) vs the Hermes cell.
+
+## OPEN ITEMS to verify on first build (expect 1-2 iterations)
+1. **Socket path** — `entrypoint.sh` exports `WAGGLE_WORKER_SOCK=${HERMES_WORKER_SOCK:-/tmp/hermes-worker.sock}`.
+   Confirm `containers/hermes/gaia2_adapter.py` actually binds that path (grep its socket default); align if different.
+2. **node binary portability** — `COPY --from=node:20-bookworm-slim /usr/local/bin/node` assumes the gaia2-cli
+   base is glibc/bookworm-compatible. If it's alpine/musl, switch to `node:20-alpine` source or `apt-get install nodejs`.
+3. **hive-mind-core deep-module closure** — confirm `logger.js`/`injection-scanner.js` import nothing further
+   (rerun the spike's isolated import proof against the staged payload before building).
+4. **LLM gateway** — worker defaults `BASE_URL=https://openrouter.ai/api/v1` (OpenAI-compat, Sonnet 4.6 via
+   OpenRouter). Hermes hits Anthropic directly. Same model; document the gateway as a minor confound, or point
+   both at one litellm proxy for strict parity.

benchmarks/gaia2/waggle-container/Dockerfile

@@ -0,0 +1,76 @@
+# gaia2-waggle — GAIA 2 env wrapping WAGGLE's own agent harness (runAgentLoop).
+# Models gaia2-hermes, but the agent loop is Waggle's, run as a Node worker.
+# Fairness: same single `terminal` tool + same AGENTS.md + same model (Sonnet 4.6);
+# only the loop logic differs. Path-A: no better-sqlite3 (2-symbol @waggle/core stub).
+#
+# BUILD (from gaia2-cli/, after staging waggle payload under containers/waggle/):
+#   docker build -f containers/waggle/Dockerfile -t localhost/gaia2-waggle:latest .
+
+ARG GAIA2_CLI_IMAGE=localhost/gaia2-cli
+ARG GAIA2_CLI_VERSION=latest
+
+# ── setuid wrapper (same as hermes) ────────────────────────────────────
+FROM debian:bookworm-slim AS gaia2-exec-builder
+RUN echo 'APT::Sandbox::User "root";' | tee -a /etc/apt/apt.conf.d/10sandbox
+RUN apt-get update && apt-get install -y --no-install-recommends gcc libc6-dev && rm -rf /var/lib/apt/lists/*
+COPY shared/gaia2-exec.c /tmp/gaia2-exec.c
+RUN gcc -O2 -o /tmp/gaia2-exec /tmp/gaia2-exec.c && strip /tmp/gaia2-exec
+
+# ── node binary source ──────────────────────────────────────────────────
+FROM node:20-bookworm-slim AS nodesrc
+
+# ── Final image ─────────────────────────────────────────────────────────
+FROM ${GAIA2_CLI_IMAGE}:${GAIA2_CLI_VERSION}
+
+# Node runtime (self-contained against glibc/libstdc++ present in bookworm base)
+COPY --from=nodesrc /usr/local/bin/node /usr/local/bin/node
+
+# Gaia2 privilege-separated user + state (same as hermes)
+RUN useradd -r -s /usr/sbin/nologin -d /var/gaia2 gaia2 && mkdir -p /var/gaia2/state
+ENV GAIA2_STATE_DIR=/var/gaia2/state
+RUN chown -R gaia2:gaia2 /var/gaia2 && chmod 700 /var/gaia2
+RUN useradd -m -s /bin/bash -u 1000 agent
+
+# Setuid gaia2 wrapper
+COPY --from=gaia2-exec-builder /tmp/gaia2-exec /usr/local/bin/gaia2-exec
+RUN chown gaia2:gaia2 /usr/local/bin/gaia2-exec && chmod u+s /usr/local/bin/gaia2-exec
+
+# Restricted agent PATH: app CLIs -> gaia2-exec + curated unix tools + python3 + node
+RUN mkdir -p /home/agent/bin && \
+    for cmd in calendar contacts emails messages chats rent-a-flat city cabs shopping cloud-drive; do \
+        ln -s /usr/local/bin/gaia2-exec /home/agent/bin/$cmd; done && \
+    for u in ls pwd date echo cat head tail grep sed awk wc sort mkdir cp mv rm touch chmod jq seq xargs find tee cut tr paste uniq diff base64 basename dirname file mktemp sleep expr test curl; do \
+        ln -s /usr/bin/$u /home/agent/bin/$u 2>/dev/null || true; done && \
+    ln -s /usr/local/bin/python3 /home/agent/bin/python3 && \
+    ln -s /usr/local/bin/node /home/agent/bin/node && \
+    chown -R agent:agent /home/agent/bin
+
+# Faketime-aware bash/sh wrappers (same as hermes)
+RUN printf '#!/usr/bin/bash\nif [ -f /tmp/faketime.rc ]; then\n  export LD_PRELOAD="/usr/lib/x86_64-linux-gnu/faketime/libfaketime.so.1"\n  export FAKETIME_TIMESTAMP_FILE="/tmp/faketime.rc"\n  export FAKETIME_NO_CACHE=1\n  unset FAKETIME\nfi\nexec /usr/bin/bash "$@"\n' \
+      > /home/agent/bin/bash && chmod +x /home/agent/bin/bash && cp /home/agent/bin/bash /home/agent/bin/sh
+
+# Shared adapter + prompt rendering (reuse hermes adapter — same contract)
+COPY shared/gaia2_adapter_base.py /opt/gaia2_adapter_base.py
+COPY shared/render_agent_prompt.py /opt/render_agent_prompt.py
+COPY shared/AGENTS_TEMPLATE.md /opt/AGENTS_TEMPLATE.md
+COPY containers/hermes/gaia2_adapter.py /opt/gaia2_adapter.py
+
+# ── Waggle payload: worker + agent dist + 2-symbol @waggle/core stub ─────
+#   containers/waggle/payload/ is assembled by build-payload.sh before docker build:
+#     payload/waggle_worker.mjs
+#     payload/package.json                       ({"type":"module"})
+#     payload/node_modules/@waggle/agent/        (built dist + package.json)
+#     payload/node_modules/@waggle/core/         (stub: index.js + package.json)
+#     payload/node_modules/@waggle/hive-mind-core/dist/{logger.js,injection-scanner.js,...}
+COPY containers/waggle/payload /opt/waggle
+COPY containers/waggle/entrypoint.sh /opt/entrypoint.sh
+RUN chmod +x /opt/entrypoint.sh
+
+ENV HOME=/home/agent
+ENV PATH="/home/agent/bin"
+
+USER root
+COPY containers/waggle/gaia2-init-entrypoint.sh /opt/gaia2-init-entrypoint.sh
+RUN chmod +x /opt/gaia2-init-entrypoint.sh
+ENTRYPOINT []
+CMD ["/usr/bin/bash", "/opt/gaia2-init-entrypoint.sh"]

benchmarks/gaia2/waggle-container/entrypoint.sh

@@ -0,0 +1,37 @@
+#!/bin/bash
+# entrypoint.sh (gaia2-waggle) — starts the WAGGLE Node worker.
+# Adapted from containers/hermes/entrypoint.sh: same lifecycle, launches
+# `node /opt/waggle/waggle_worker.mjs` instead of the python hermes worker.
+set -o pipefail
+LOG=/tmp/entrypoint.log
+log() { echo "[$(date +%H:%M:%S)] $*" | tee -a $LOG; }
+log "=== entrypoint start (waggle) ==="
+
+if [ -n "${FAKETIME:-}" ]; then
+    /usr/bin/mkdir -p /dev/shm 2>/dev/null && /usr/bin/chmod 1777 /dev/shm 2>/dev/null || true
+    log "Faketime enabled: $FAKETIME"
+fi
+log "User: $(/usr/bin/id -un), PATH: $PATH"
+
+# Socket path MUST match the (reused hermes) gaia2_adapter.py. The adapter's
+# default is exported here so the worker's HERMES_WORKER_SOCK fallback matches.
+export WAGGLE_WORKER_SOCK="${HERMES_WORKER_SOCK:-/tmp/hermes-worker.sock}"
+log "Starting Waggle worker (sock=$WAGGLE_WORKER_SOCK)..."
+/usr/local/bin/node /opt/waggle/waggle_worker.mjs >> $LOG 2>&1 &
+WORKER_PID=$!
+log "Worker PID: $WORKER_PID"
+
+/usr/bin/sleep 2
+if ! kill -0 $WORKER_PID 2>/dev/null; then
+    log "ERROR: Waggle worker exited unexpectedly"
+    /usr/bin/tail -40 $LOG 2>/dev/null || true
+    exit 1
+fi
+log "Waggle worker running"
+
+cleanup() { log "Shutting down..."; kill "$WORKER_PID" 2>/dev/null || true; wait "$WORKER_PID" 2>/dev/null || true; }
+trap cleanup EXIT TERM INT
+wait $WORKER_PID 2>/dev/null
+EXIT_CODE=$?
+log "Worker (PID $WORKER_PID) exited with code $EXIT_CODE"
+exit $EXIT_CODE

benchmarks/gaia2/waggle-container/stub-core/index.js

@@ -0,0 +1,7 @@
+// Path-A stub for @waggle/core — the ONLY 2 symbols runAgentLoop's runtime chain
+// needs (createCoreLogger via turn-context.ts, scanForInjection via injection-scanner.ts).
+// Re-exported from hive-mind-core's DB-free deep paths, bypassing the index barrel
+// whose `export { MindDB } from './mind/db.js'` eagerly loads better-sqlite3/sqlite-vec.
+// Proven (2026-05-22) to import runAgentLoop with zero native deps. See ../spike-waggle-worker/.
+export { createCoreLogger } from "@waggle/hive-mind-core/dist/logger.js";
+export { scanForInjection } from "@waggle/hive-mind-core/dist/injection-scanner.js";

benchmarks/gaia2/waggle-container/stub-core/package.json

@@ -0,0 +1 @@
+{ "name": "@waggle/core", "version": "0.0.0-gaia2-stub", "type": "module", "main": "index.js" }

benchmarks/gaia2/waggle-container/waggle_worker.mjs

@@ -0,0 +1,199 @@
+#!/usr/bin/env node
+// waggle_worker — bridges Waggle's runAgentLoop to the GAIA 2 ARE adapter.
+//
+// Mirrors hermes_worker.py's contract so the harness comparison is fair:
+//   - same single `terminal` tool (GAIA 2 apps invoked as shell cmds via gaia2-exec)
+//   - same AGENTS.md as system prompt (rendered by gaia2-init-entrypoint.sh)
+//   - same model (Sonnet 4.6) — the ONLY variable is Waggle's loop logic.
+//
+// Protocol (JSON lines over Unix socket):
+//   Worker -> Adapter: {"type":"ready"}
+//                      {"type":"response","run_id","state":"final"|"error","message"}
+//   Adapter -> Worker: {"type":"message","text","run_id"}  /  {"type":"interrupt","text"}
+import net from "node:net";
+import fs from "node:fs";
+import os from "node:os";
+import { spawn } from "node:child_process";
+import { runAgentLoop } from "@waggle/agent/dist/agent-loop.js";
+// F2 (2026-05-27): opt-in persona overlay via composePersonaPrompt. Bare-Waggle behavior
+// (matching the 2026-05-22 N=40 on-par-with-Hermes baseline) preserved when
+// WAGGLE_PERSONA_ID is unset / empty.
+import { getPersona, composePersonaPrompt } from "@waggle/agent/dist/personas.js";
+
+const WORKER_SOCK = process.env.WAGGLE_WORKER_SOCK || process.env.HERMES_WORKER_SOCK || "/tmp/waggle-worker.sock";
+const AGENTS_MD = `${os.homedir()}/AGENTS.md`;
+const MODEL = process.env.MODEL || "claude-sonnet-4-6";
+const LITELLM_URL = process.env.BASE_URL || process.env.LITELLM_URL || "https://openrouter.ai/api/v1";
+const API_KEY = process.env.API_KEY || process.env.LITELLM_API_KEY || "";
+const MAX_TURNS = parseInt(process.env.MAX_ITERATIONS || "90", 10);
+const MAX_TOKENS = process.env.MAX_TOKENS ? parseInt(process.env.MAX_TOKENS, 10) : undefined;
+
+function log(...a) { console.log("[waggle-worker]", ...a); }
+
+// Single `terminal` tool: runs a shell command (the agent calls gaia2-exec via AGENTS.md
+// instructions). bash -lc so PATH includes gaia2-exec; inherits the agent-user env.
+const terminalTool = {
+  name: "terminal",
+  description: "Execute a shell command in the sandbox. Use it to call the available GAIA2 app CLIs as described in the system prompt. Returns combined stdout+stderr.",
+  parameters: {
+    type: "object",
+    properties: { command: { type: "string", description: "The shell command to run." } },
+    required: ["command"],
+  },
+  execute: ({ command }) =>
+    new Promise((resolve) => {
+      const cmd = typeof command === "string" ? command : String(command ?? "");
+      const child = spawn("bash", ["-lc", cmd], { cwd: os.homedir(), env: process.env });
+      let out = "";
+      const cap = (d) => { out += d.toString(); if (out.length > 200_000) { try { child.kill(); } catch {} } };
+      child.stdout.on("data", cap);
+      child.stderr.on("data", cap);
+      child.on("close", (code) => resolve(out + (code ? `\n[exit ${code}]` : "")));
+      child.on("error", (e) => resolve(`[terminal error] ${e.message}`));
+    }),
+};
+
+function readAgentsMd() {
+  try { return fs.readFileSync(AGENTS_MD, "utf8"); }
+  catch (e) { log("WARN: could not read", AGENTS_MD, e.message); return "You are a helpful agent. Use the terminal tool to complete the task."; }
+}
+
+// Fairness with Hermes: Hermes has NO skill-distillation / verification-gate, so for
+// "harness is the only variable" to hold, Waggle runs with the same task contract —
+// these meta-features OFF. (They also hijacked the final user answer with a skill
+// summary in the n1 smoke — a real Waggle bug to fix separately.) Env-overridable so
+// an as-shipped (gates ON) variant can be measured later without a rebuild.
+const SKILL_GATE = process.env.WAGGLE_SKILL_DISTILLATION_GATE === "1";
+const VERIFY_GATE = process.env.WAGGLE_VERIFICATION_GATE === "1";
+
+// F2 (2026-05-27): when WAGGLE_PERSONA_ID is set (e.g. "executive-assistant"), the
+// worker composes AGENTS.md (the GAIA 2 tool / app context — REQUIRED) with the
+// persona's `systemPrompt` via composePersonaPrompt(). Resolved once at startup so
+// every scenario in a run sees the same prompt shape. If the ID is unrecognized we
+// log a WARN and fall back to bare AGENTS.md so a typo never silently changes the
+// measurement.
+const PERSONA_ID = (process.env.WAGGLE_PERSONA_ID || "").trim();
+let resolvedPersona = null;
+if (PERSONA_ID) {
+  try {
+    resolvedPersona = getPersona(PERSONA_ID) ?? null;
+    if (resolvedPersona) {
+      log(`F2: persona overlay ON — id=${PERSONA_ID} (${resolvedPersona.name})`);
+    } else {
+      log(`F2: WARN — WAGGLE_PERSONA_ID="${PERSONA_ID}" did not resolve to a known persona; falling back to bare AGENTS.md`);
+    }
+  } catch (e) {
+    log(`F2: WARN — getPersona threw (${e?.message || e}); falling back to bare AGENTS.md`);
+    resolvedPersona = null;
+  }
+}
+
+// F3 (2026-05-27): opt-in output-discipline appendix targeting Qwen-thinking failure
+// modes seen in the bare N=160 (Cat 1 verbose multi-paragraph answers + Cat 3
+// thinking-mode bleed). Activated by WAGGLE_GAIA2_QWEN_SHAPE=1. Applied as the
+// FINAL section of the system prompt so it overrides any persona-introduced framing.
+// Composes cleanly with or without the F2 persona overlay (compose-then-append).
+const APPLY_QWEN_SHAPE = process.env.WAGGLE_GAIA2_QWEN_SHAPE === "1";
+const QWEN_SHAPE_APPENDIX = `
+
+---
+
+## Final Answer Discipline (CRITICAL — read before every send_message_to_user)
+
+Your final \`send_message_to_user\` MUST be the ANSWER, not an analysis. Hard rules:
+
+1. **One short line.** No multi-paragraph response. No headers. No bullet lists. No bold formatting.
+2. **No preamble.** Do NOT begin with "Based on my analysis", "Let me", "Now let me", "Here is", "I found", "After analyzing", or "Looking at the data". Just give the value.
+3. **No restatement of the question.** The user knows what they asked.
+4. **No appended reasoning.** Do NOT include "because…", "since…", "due to…" clauses unless the question explicitly asked for justification.
+5. **No "Answer:" / "**Answer:**" prefix.** Just the value itself.
+
+Shape by question type:
+- "Which city…?" → \`Stockholm\` (one word, the city name)
+- "What is the average…?" → \`45\` (the number, rounded as the question specified)
+- "Who is the contact…?" → \`Astrid Lindqvist\` (the name)
+- "How many…?" → \`12\` (the count)
+- "What time…?" → \`14:30\` (the time)
+- Listy "What are the…?" → \`Stockholm, Oslo, Copenhagen\` (comma-separated, no bullets)
+
+Your reasoning ALREADY happened in your \`<think>\` blocks and tool calls. The send_message_to_user is a result delivery, not a reasoning rendition. If you find yourself writing more than ~15 words in send_message_to_user, you are wrong — rewrite shorter.
+
+`;
+
+function buildSystemPrompt() {
+  const core = readAgentsMd();
+  let prompt = resolvedPersona ? composePersonaPrompt(core, resolvedPersona) : core;
+  if (APPLY_QWEN_SHAPE) prompt += QWEN_SHAPE_APPENDIX;
+  return prompt;
+}
+
+async function runOnce(text) {
+  const res = await runAgentLoop({
+    litellmUrl: LITELLM_URL,
+    litellmApiKey: API_KEY,
+    model: MODEL,
+    systemPrompt: buildSystemPrompt(),
+    tools: [terminalTool],
+    messages: [{ role: "user", content: text }],
+    maxTurns: MAX_TURNS,
+    maxTokenBudget: MAX_TOKENS,
+    stream: true,
+    skillDistillationGate: SKILL_GATE,
+    verificationGate: VERIFY_GATE,
+  });
+  // AgentResponse — final assistant text. Fall back across likely field names.
+  return res?.content ?? res?.message ?? res?.finalResponse ?? res?.text ?? "";
+}
+
+function connect() {
+  return new Promise((resolve, reject) => {
+    const deadline = Date.now() + 60_000;
+    const attempt = () => {
+      const sock = net.createConnection(WORKER_SOCK);
+      sock.once("connect", () => { log("connected", WORKER_SOCK); resolve(sock); });
+      sock.once("error", () => {
+        sock.destroy();
+        if (Date.now() > deadline) reject(new Error(`timeout connecting ${WORKER_SOCK}`));
+        else setTimeout(attempt, 200);
+      });
+    };
+    attempt();
+  });
+}
+
+function send(sock, obj) { sock.write(JSON.stringify(obj) + "\n"); }
+
+async function main() {
+  const sock = await connect();
+  send(sock, { type: "ready" });
+  let buf = "";
+  let busy = false;
+  sock.on("data", async (chunk) => {
+    buf += chunk.toString();
+    let i;
+    while ((i = buf.indexOf("\n")) >= 0) {
+      const line = buf.slice(0, i).trim();
+      buf = buf.slice(i + 1);
+      if (!line) continue;
+      let msg;
+      try { msg = JSON.parse(line); } catch { continue; }
+      if (msg.type === "interrupt") { log("interrupt (ignored — single-turn scenarios)"); continue; }
+      if (msg.type !== "message") { log("unknown msg type", msg.type); continue; }
+      if (busy) { log("WARN: message while busy"); }
+      busy = true;
+      const runId = msg.run_id || "unknown";
+      try {
+        log(`run ${runId}:`, (msg.text || "").slice(0, 100));
+        const message = await runOnce(msg.text || "");
+        log(`done ${runId}:`, String(message).slice(0, 100));
+        send(sock, { type: "response", run_id: runId, state: "final", message: String(message) });
+      } catch (e) {
+        log(`error ${runId}:`, e?.message || e);
+        send(sock, { type: "response", run_id: runId, state: "error", message: `Error: ${e?.message || e}`, errorMessage: String(e?.message || e) });
+      } finally { busy = false; }
+    }
+  });
+  sock.on("close", () => { log("socket closed"); process.exit(0); });
+}
+
+main().catch((e) => { console.error("[waggle-worker] fatal", e); process.exit(1); });