# Judge Rubric — Trio Ensemble × 6 Dimensions × Likert 1-5 **Purpose:** Calibrated quality assessment of agent responses to knowledge work tasks. Single-axis Yes/No judging (LoCoMo style) is unsuitable for synthesis tasks where "correctness" is multi-dimensional and the question itself is open-ended. **Judge ensemble (locked):** - Claude Opus 4.7 (`claude-opus-4-7`) - GPT-5.4 (`gpt-5.4`) - MiniMax M2.7 (`minimax-m2.7`) **Reuses Stage 3 v6 trio infrastructure** — `κ_trio = 0.7878` (substantial agreement) calibrated 2026-04-24. No new judge calibration needed for this pilot. If pilot escalates to full N=400, recalibrate on synthesis-task subset (deferred to expansion brief). --- ## Six dimensions Each judge scores each cell response on six dimensions, Likert 1-5. **Mean across dimensions = overall score.** Halt threshold: any cell scoring < 2.0 on majority of judges = critical failure flag. ### D1 — Completeness *Did the response engage with all material provided, or did it ignore key inputs?* - **5 — Comprehensive**: Engages with every document/thread/memo. Cites or references most. No material is treated as irrelevant without justification. - **4 — Strong**: Engages with most materials. May skip minor items but justifies omissions. - **3 — Adequate**: Engages with majority of materials. Some material visibly missed but core covered. - **2 — Partial**: Significant material omitted without justification. Response treats subset as if it were the whole. - **1 — Inadequate**: Response engages with minority of materials. Most input is ignored. ### D2 — Accuracy *Are the facts cited from the materials accurate, or are there hallucinations / misreadings?* - **5 — Faithful**: All cited facts traceable to materials. No hallucinations. Numbers correct. Names correct. - **4 — Mostly faithful**: 1-2 minor inaccuracies (wrong number, slight name variant) but no material distortion. - **3 — Mixed**: Some inaccuracies. Core narrative still defensible from materials. - **2 — Weak**: Multiple factual errors. Some claims not in materials. Reader would be misled on specific points. - **1 — Unreliable**: Significant fabrication or misreading. Reader cannot trust the response. ### D3 — Synthesis quality *Does the response connect inputs across documents/threads/memos, or treat each in isolation?* - **5 — Deeply synthesized**: Identifies non-obvious connections (e.g., "X in Doc 2 explains Y in Doc 5"). Surfaces interaction effects. Goes beyond the surface of any single input. - **4 — Strong synthesis**: Connects most inputs. Cross-references where appropriate. May miss 1-2 deeper patterns. - **3 — Adequate synthesis**: Some connections drawn. Mostly summarizes input-by-input with limited weaving. - **2 — Weak synthesis**: Treats inputs in isolation. List-like structure mirroring input order. - **1 — No synthesis**: Disconnected responses to individual inputs. No integration. ### D4 — Judgment quality *Are the recommendations defensible? Are tradeoffs acknowledged? Is reasoning shown?* - **5 — Senior-grade**: Recommendations are specific and actionable. Tradeoffs explicitly addressed. Counter-arguments anticipated. Reasoning visible at each step. - **4 — Strong**: Recommendations are clear and reasoned. Most tradeoffs surfaced. Some implicit reasoning. - **3 — Adequate**: Recommendations made but reasoning thin. Tradeoffs touched lightly. - **2 — Weak**: Recommendations feel arbitrary. Tradeoffs ignored or minimized. Reasoning shallow. - **1 — No judgment**: Recommendations missing, generic, or contradicted by their own analysis. ### D5 — Recommendation actionability *Could the persona (CFO / Partner / CEO) act on this tomorrow morning, or is it advice-shaped fog?* - **5 — Immediately actionable**: Specific actions, owners (where applicable), sequencing, success metrics. The persona could open a doc tomorrow and start executing. - **4 — Mostly actionable**: Most actions are specific. Some require additional definition but the path is clear. - **3 — Directionally actionable**: Direction is clear; specific next steps require persona to fill in. - **2 — Vague**: General advice. Persona has to do meaningful translation work to derive actions. - **1 — Not actionable**: Abstract reasoning without practical pathway. No persona could act on this. ### D6 — Structure / Communication *Is the response organized for the reader's mental model? Is it the right length? Is it readable under time pressure?* - **5 — Excellent**: Clear executive structure (e.g., headline → reasoning → asks). Appropriate length. Reader can scan in 60 seconds and read in detail in 5 minutes. Headers, emphasis, sequence used judiciously. - **4 — Strong**: Well-organized. Reasonable length. Reader navigates easily. - **3 — Adequate**: Comprehensible. Length OK. Some friction in scanning. - **2 — Weak**: Disorganized. Too long or too brief. Reader has to work to extract main points. - **1 — Poor**: Chaotic structure. Significantly mis-sized. Reader gets lost or gives up. --- ## Overall scoring **Per judge per cell:** mean of D1-D6 = overall score (Likert 1-5) **Per cell aggregated:** - Trio mean: (Opus mean + GPT mean + MiniMax mean) / 3 - Trio strict-PASS: at least 2 of 3 judges score ≥ 3.5 - Trio FAIL: at least 2 of 3 judges score < 3.0 **Hypothesis verification (per task):** - **H2 — Opus multiplier**: Cell B (Opus + memory + harness) trio mean > Cell A (Opus solo) trio mean by ≥ 0.30 Likert points - **H3 — Qwen multiplier**: Cell D (Qwen + memory + harness) trio mean > Cell C (Qwen solo) trio mean by ≥ 0.30 Likert points - **H4 — Sovereignty bridge**: Cell D trio mean ≥ Cell A trio mean (Qwen + harness reaches frontier-without-harness) **Pilot binary verdict:** - **PILOT PASS** = directional sign correct on H2/H3/H4 in ≥ 2 of 3 tasks (6/9 cells minimum), and no critical failure (no cell scoring < 2.0 on majority of judges) - **PILOT FAIL** = otherwise PASS authorizes full N=400 multiplier benchmark. FAIL halts expansion. --- ## Judge prompt template (per cell response) ``` You are evaluating an AI agent's response to a complex knowledge work task. The persona, scenario, materials, and question are provided. The response was generated under one of four configurations (revealed only after scoring): {model_only | model + memory + agent harness} × {Opus 4.7 | Qwen 3.6 35B-A3B}. You do NOT know which configuration produced this response. Score blind. Read the persona/scenario/question (provided), skim the materials (provided), then read the response carefully (provided). Score the response on six dimensions, Likert 1-5: 1. COMPLETENESS — engagement with all material 2. ACCURACY — faithfulness to source materials, no hallucinations 3. SYNTHESIS — connections across inputs, not isolated treatment 4. JUDGMENT — defensible recommendations, tradeoffs acknowledged 5. ACTIONABILITY — would the persona act on this tomorrow 6. STRUCTURE — organization and readability Output JSON only: { "completeness": <1-5>, "accuracy": <1-5>, "synthesis": <1-5>, "judgment": <1-5>, "actionability": <1-5>, "structure": <1-5>, "rationale": "<1-2 sentences explaining the lowest scoring dimension>", "overall_verdict": "" } PASS_STRONG: mean ≥ 4.0 PASS_ADEQUATE: mean 3.5-3.99 FAIL_WEAK: mean 2.5-3.49 FAIL_CRITICAL: mean < 2.5 [PERSONA + SCENARIO + QUESTION] [MATERIALS] [RESPONSE TO EVALUATE] ``` --- ## Output JSONL schema (per cell, per task) Each cell × task produces one record: ```json { "task_id": "task-1" | "task-2" | "task-3", "cell_id": "A" | "B" | "C" | "D", "model": "claude-opus-4-7" | "qwen3.6-35b-a3b", "configuration": "solo" | "memory-harness", "candidate_response": "", "candidate_latency_ms": , "candidate_tokens_in": , "candidate_tokens_out": , "candidate_cost_usd": , "judge_opus": { "completeness": , "accuracy": , "synthesis": , "judgment": , "actionability": , "structure": , "rationale": "", "overall_verdict": "", "mean": }, "judge_gpt": { ... same shape ... }, "judge_minimax": { ... same shape ... }, "trio_mean": , "trio_strict_pass": , "trio_critical_fail": , "manifest_anchor": "pilot-2026-04-26-v1", "head_sha": "" } ``` 12 records total (3 tasks × 4 cells). --- ## Aggregate summary file After execution, produce `pilot-summary.json`: ```json { "pilot_id": "agentic-knowledge-work-pilot-2026-04-26", "execution_window_utc": " to ", "total_cost_usd": , "total_judge_calls": 36, "total_candidate_calls": 12, "results_per_task": { "task-1": { "cell_A_trio_mean": , "cell_B_trio_mean": , "cell_C_trio_mean": , "cell_D_trio_mean": , "h2_delta_opus": , "h3_delta_qwen": , "h4_delta_sovereignty": , "h2_directional_pass": , "h3_directional_pass": , "h4_directional_pass": }, "task-2": { ... }, "task-3": { ... } }, "aggregate": { "h2_pass_count": , "h3_pass_count": , "h4_pass_count": , "critical_failures": , "pilot_verdict": "PASS" | "FAIL" } } ``` PM and Marko adjudicate from this summary file.