waggle-os/docs/briefs/2026-04-26-agentic-knowledge-work-pilot/judge-rubric.md

# Judge Rubric — Trio Ensemble × 6 Dimensions × Likert 1-5

**Purpose:** Calibrated quality assessment of agent responses to knowledge work tasks. Single-axis Yes/No judging (LoCoMo style) is unsuitable for synthesis tasks where "correctness" is multi-dimensional and the question itself is open-ended.

**Judge ensemble (locked):**
- Claude Opus 4.7 (`claude-opus-4-7`)
- GPT-5.4 (`gpt-5.4`)
- MiniMax M2.7 (`minimax-m2.7`)

**Reuses Stage 3 v6 trio infrastructure** — `κ_trio = 0.7878` (substantial agreement) calibrated 2026-04-24. No new judge calibration needed for this pilot. If pilot escalates to full N=400, recalibrate on synthesis-task subset (deferred to expansion brief).

---

## Six dimensions

Each judge scores each cell response on six dimensions, Likert 1-5. **Mean across dimensions = overall score.** Halt threshold: any cell scoring < 2.0 on majority of judges = critical failure flag.

### D1 — Completeness

*Did the response engage with all material provided, or did it ignore key inputs?*

- **5 — Comprehensive**: Engages with every document/thread/memo. Cites or references most. No material is treated as irrelevant without justification.
- **4 — Strong**: Engages with most materials. May skip minor items but justifies omissions.
- **3 — Adequate**: Engages with majority of materials. Some material visibly missed but core covered.
- **2 — Partial**: Significant material omitted without justification. Response treats subset as if it were the whole.
- **1 — Inadequate**: Response engages with minority of materials. Most input is ignored.

### D2 — Accuracy

*Are the facts cited from the materials accurate, or are there hallucinations / misreadings?*

- **5 — Faithful**: All cited facts traceable to materials. No hallucinations. Numbers correct. Names correct.
- **4 — Mostly faithful**: 1-2 minor inaccuracies (wrong number, slight name variant) but no material distortion.
- **3 — Mixed**: Some inaccuracies. Core narrative still defensible from materials.
- **2 — Weak**: Multiple factual errors. Some claims not in materials. Reader would be misled on specific points.
- **1 — Unreliable**: Significant fabrication or misreading. Reader cannot trust the response.

### D3 — Synthesis quality

*Does the response connect inputs across documents/threads/memos, or treat each in isolation?*

- **5 — Deeply synthesized**: Identifies non-obvious connections (e.g., "X in Doc 2 explains Y in Doc 5"). Surfaces interaction effects. Goes beyond the surface of any single input.
- **4 — Strong synthesis**: Connects most inputs. Cross-references where appropriate. May miss 1-2 deeper patterns.
- **3 — Adequate synthesis**: Some connections drawn. Mostly summarizes input-by-input with limited weaving.
- **2 — Weak synthesis**: Treats inputs in isolation. List-like structure mirroring input order.
- **1 — No synthesis**: Disconnected responses to individual inputs. No integration.

### D4 — Judgment quality

*Are the recommendations defensible? Are tradeoffs acknowledged? Is reasoning shown?*

- **5 — Senior-grade**: Recommendations are specific and actionable. Tradeoffs explicitly addressed. Counter-arguments anticipated. Reasoning visible at each step.
- **4 — Strong**: Recommendations are clear and reasoned. Most tradeoffs surfaced. Some implicit reasoning.
- **3 — Adequate**: Recommendations made but reasoning thin. Tradeoffs touched lightly.
- **2 — Weak**: Recommendations feel arbitrary. Tradeoffs ignored or minimized. Reasoning shallow.
- **1 — No judgment**: Recommendations missing, generic, or contradicted by their own analysis.

### D5 — Recommendation actionability

*Could the persona (CFO / Partner / CEO) act on this tomorrow morning, or is it advice-shaped fog?*

- **5 — Immediately actionable**: Specific actions, owners (where applicable), sequencing, success metrics. The persona could open a doc tomorrow and start executing.
- **4 — Mostly actionable**: Most actions are specific. Some require additional definition but the path is clear.
- **3 — Directionally actionable**: Direction is clear; specific next steps require persona to fill in.
- **2 — Vague**: General advice. Persona has to do meaningful translation work to derive actions.
- **1 — Not actionable**: Abstract reasoning without practical pathway. No persona could act on this.

### D6 — Structure / Communication

*Is the response organized for the reader's mental model? Is it the right length? Is it readable under time pressure?*

- **5 — Excellent**: Clear executive structure (e.g., headline → reasoning → asks). Appropriate length. Reader can scan in 60 seconds and read in detail in 5 minutes. Headers, emphasis, sequence used judiciously.
- **4 — Strong**: Well-organized. Reasonable length. Reader navigates easily.
- **3 — Adequate**: Comprehensible. Length OK. Some friction in scanning.
- **2 — Weak**: Disorganized. Too long or too brief. Reader has to work to extract main points.
- **1 — Poor**: Chaotic structure. Significantly mis-sized. Reader gets lost or gives up.

---

## Overall scoring

**Per judge per cell:** mean of D1-D6 = overall score (Likert 1-5)

**Per cell aggregated:**
- Trio mean: (Opus mean + GPT mean + MiniMax mean) / 3
- Trio strict-PASS: at least 2 of 3 judges score ≥ 3.5
- Trio FAIL: at least 2 of 3 judges score < 3.0

**Hypothesis verification (per task):**
- **H2 — Opus multiplier**: Cell B (Opus + memory + harness) trio mean > Cell A (Opus solo) trio mean by ≥ 0.30 Likert points
- **H3 — Qwen multiplier**: Cell D (Qwen + memory + harness) trio mean > Cell C (Qwen solo) trio mean by ≥ 0.30 Likert points
- **H4 — Sovereignty bridge**: Cell D trio mean ≥ Cell A trio mean (Qwen + harness reaches frontier-without-harness)

**Pilot binary verdict:**
- **PILOT PASS** = directional sign correct on H2/H3/H4 in ≥ 2 of 3 tasks (6/9 cells minimum), and no critical failure (no cell scoring < 2.0 on majority of judges)
- **PILOT FAIL** = otherwise

PASS authorizes full N=400 multiplier benchmark. FAIL halts expansion.

---

## Judge prompt template (per cell response)

```
You are evaluating an AI agent's response to a complex knowledge work task. The persona, scenario, materials, and question are provided. The response was generated under one of four configurations (revealed only after scoring): {model_only | model + memory + agent harness} × {Opus 4.7 | Qwen 3.6 35B-A3B}.

You do NOT know which configuration produced this response. Score blind.

Read the persona/scenario/question (provided), skim the materials (provided), then read the response carefully (provided).

Score the response on six dimensions, Likert 1-5:

1. COMPLETENESS — engagement with all material
2. ACCURACY — faithfulness to source materials, no hallucinations
3. SYNTHESIS — connections across inputs, not isolated treatment
4. JUDGMENT — defensible recommendations, tradeoffs acknowledged
5. ACTIONABILITY — would the persona act on this tomorrow
6. STRUCTURE — organization and readability

Output JSON only:
{
  "completeness": <1-5>,
  "accuracy": <1-5>,
  "synthesis": <1-5>,
  "judgment": <1-5>,
  "actionability": <1-5>,
  "structure": <1-5>,
  "rationale": "<1-2 sentences explaining the lowest scoring dimension>",
  "overall_verdict": "<one of: PASS_STRONG | PASS_ADEQUATE | FAIL_WEAK | FAIL_CRITICAL>"
}

PASS_STRONG: mean ≥ 4.0
PASS_ADEQUATE: mean 3.5-3.99
FAIL_WEAK: mean 2.5-3.49
FAIL_CRITICAL: mean < 2.5

[PERSONA + SCENARIO + QUESTION]
[MATERIALS]
[RESPONSE TO EVALUATE]
```

---

## Output JSONL schema (per cell, per task)

Each cell × task produces one record:

```json
{
  "task_id": "task-1" | "task-2" | "task-3",
  "cell_id": "A" | "B" | "C" | "D",
  "model": "claude-opus-4-7" | "qwen3.6-35b-a3b",
  "configuration": "solo" | "memory-harness",
  "candidate_response": "<full response text>",
  "candidate_latency_ms": <int>,
  "candidate_tokens_in": <int>,
  "candidate_tokens_out": <int>,
  "candidate_cost_usd": <float>,
  "judge_opus": {
    "completeness": <int>,
    "accuracy": <int>,
    "synthesis": <int>,
    "judgment": <int>,
    "actionability": <int>,
    "structure": <int>,
    "rationale": "<string>",
    "overall_verdict": "<string>",
    "mean": <float>
  },
  "judge_gpt": { ... same shape ... },
  "judge_minimax": { ... same shape ... },
  "trio_mean": <float>,
  "trio_strict_pass": <bool>,
  "trio_critical_fail": <bool>,
  "manifest_anchor": "pilot-2026-04-26-v1",
  "head_sha": "<git commit SHA at execution>"
}
```

12 records total (3 tasks × 4 cells).

---

## Aggregate summary file

After execution, produce `pilot-summary.json`:

```json
{
  "pilot_id": "agentic-knowledge-work-pilot-2026-04-26",
  "execution_window_utc": "<ISO start> to <ISO end>",
  "total_cost_usd": <float>,
  "total_judge_calls": 36,
  "total_candidate_calls": 12,
  "results_per_task": {
    "task-1": {
      "cell_A_trio_mean": <float>,
      "cell_B_trio_mean": <float>,
      "cell_C_trio_mean": <float>,
      "cell_D_trio_mean": <float>,
      "h2_delta_opus": <B - A>,
      "h3_delta_qwen": <D - C>,
      "h4_delta_sovereignty": <D - A>,
      "h2_directional_pass": <bool>,
      "h3_directional_pass": <bool>,
      "h4_directional_pass": <bool>
    },
    "task-2": { ... },
    "task-3": { ... }
  },
  "aggregate": {
    "h2_pass_count": <int 0-3>,
    "h3_pass_count": <int 0-3>,
    "h4_pass_count": <int 0-3>,
    "critical_failures": <int>,
    "pilot_verdict": "PASS" | "FAIL"
  }
}
```

PM and Marko adjudicate from this summary file.