171 lines
8.9 KiB
Markdown
171 lines
8.9 KiB
Markdown
---
|
||
decision_id: 2026-04-27-phase-2-gate-d3-rule-inspection
|
||
date: 2026-04-27
|
||
phase: 2 acceptance gate — D3 disambiguation step
|
||
verdict: scoring-rule confound IDENTIFIED + RESOLVED; substrate-no-regression CONFIRMED
|
||
predecessor: 2026-04-26-phase-1-acceptance-gate-results.md
|
||
successor: 2026-04-27-phase-2-acceptance-gate-results.md (TBD)
|
||
---
|
||
|
||
# Phase 2 Acceptance Gate — D3 Rule Inspection
|
||
|
||
## TL;DR
|
||
|
||
The 56.5 pp drift between my Phase 2 acceptance gate smoke (90% trio-strict) and v6 baseline (33.5%) was almost entirely a **scoring-rule mismatch**, not a substrate / prompt regression.
|
||
|
||
- **v6's `accuracy` field rule:** `scoreAccuracy(output, expected)` — case-insensitive substring match on the `expected[]` list (per `benchmarks/harness/src/metrics.ts:57`).
|
||
- **My smoke's strict-pass rule:** majority of trio judges return "correct" or "null" — judge consensus, much more lenient.
|
||
- **Re-aggregated N=20 with v6's exact rule: 8/20 = 40.0%.** Drift vs baseline shrinks from +56.5 pp → **+6.5 pp**, well within statistical sample variance for N=20.
|
||
|
||
D1 (old SYSTEM_BASELINE re-run) and D2 (different seeds) **NOT NEEDED**. Phase 2 acceptance gate result available now.
|
||
|
||
---
|
||
|
||
## v6 accuracy rule (exact, from source)
|
||
|
||
`benchmarks/harness/src/runner.ts:427`:
|
||
```typescript
|
||
const accuracy = result.failureMode ? 0 : scoreAccuracy(result.text, instance.expected);
|
||
```
|
||
|
||
`benchmarks/harness/src/metrics.ts:57-66`:
|
||
```typescript
|
||
/** Scores a model output against expected substrings (any-match = full credit). */
|
||
export function scoreAccuracy(output: string, expected: string[]): number {
|
||
if (expected.length === 0) return 0;
|
||
const lower = output.toLowerCase();
|
||
for (const exp of expected) {
|
||
if (lower.includes(exp.toLowerCase())) return 1;
|
||
}
|
||
return 0;
|
||
}
|
||
```
|
||
|
||
Rule: `accuracy = 1` iff (a) no subject failure mode AND (b) lowercased model output contains at least one of the lowercased `expected[]` strings as a substring. Otherwise `accuracy = 0`.
|
||
|
||
This is a **textual substring rule**, completely independent of the trio judge ensemble verdicts.
|
||
|
||
## v6 N=400 oracle data verifies the rule
|
||
|
||
| Cross-tab | Count | Notes |
|
||
|---|---|---|
|
||
| acc=1 (any reason) | 134 | 33.5% baseline |
|
||
| acc=0 (any reason) | 266 | |
|
||
| acc=1 × judge_verdict='correct' × ensemble all-correct | **134** | unanimous correct judges + accuracy=1 |
|
||
| acc=0 × judge_verdict='correct' × ensemble all-correct + no fmodes | **117** | unanimous correct judges + accuracy=0 (semantic ≠ substring) |
|
||
| acc=0 × judge_verdict='correct' × ensemble (correct,correct,incorrect) | 25 | majority correct but split |
|
||
| acc=0 × judge_verdict='incorrect' (any) | 118 | |
|
||
|
||
**The 117 unanimous-correct-but-acc=0 rows are the smoking gun.** Judges agreed model was correct, but `scoreAccuracy` substring match against `expected[]` returned 0 because the model rephrased the gold answer.
|
||
|
||
Sample acc=0+unanimous case (`locomo_conv-30_q057`):
|
||
- expected: `["Focus on brand identity, build customer relationships, and stay positive."]`
|
||
- model_answer: `"Focus on brand identity, build customer relationships, and stay positive."`
|
||
- All 3 judges: `correct` + `failure_mode: None`
|
||
- `accuracy: 0` ← the substring DOES match here actually
|
||
|
||
Wait — re-reading the sample: the model_answer literally equals the expected string. accuracy=0 here is unexpected. Let me re-check by hand: lowercase output = `"focus on brand identity, build customer relationships, and stay positive."` — does it contain `"focus on brand identity, build customer relationships, and stay positive."`? Yes. So accuracy should be 1. Possible bug or pre-judge accuracy snapshot.
|
||
|
||
Either way: my N=20 re-aggregation uses the SAME rule on the SAME schema, so any rule-level edge case applies symmetrically.
|
||
|
||
## Re-aggregation of my N=20 with v6 substring-match rule
|
||
|
||
| Result | Count | Pass rate |
|
||
|---|---|---|
|
||
| acc=1 (substring match) | 8 | **40.0%** |
|
||
| acc=0 (no substring match) | 12 | 60.0% |
|
||
|
||
| Comparison | v6 N=400 | My N=20 substring rule | My N=20 majority rule |
|
||
|---|---|---|---|
|
||
| Pass rate | 33.5% | **40.0%** | 90.0% |
|
||
| Drift vs v6 | (baseline) | +6.5 pp | +56.5 pp |
|
||
| In PM range 28-38% (±5 pp) | ✓ | **at 40.0% — 2 pp above upper bound** | ✗ |
|
||
|
||
### Statistical sample variance correction (was missing from PM acceptance criterion)
|
||
|
||
PM's binding range was "28-38% (±5 pp around 33.5%)" — but at N=20, the 95% CI for a binary-outcome process at p=0.335 is much wider than ±5 pp:
|
||
|
||
```
|
||
σ = √(p(1-p)/n) = √(0.335 × 0.665 / 20) = √0.01114 = 0.106 (10.6 pp)
|
||
95% CI ≈ 0.335 ± 2σ = [0.124, 0.546] = [12.4%, 54.6%]
|
||
```
|
||
|
||
So at N=20, the empirically-defensible "in-range" band is **roughly 23-44 %**, not 28-38 %. My result of 40.0% is comfortably within this realistic band.
|
||
|
||
PM's ±5 pp range was statistically appropriate for the **v6 N=400 reference run** (where σ ≈ 2.4 pp), not for an N=20 smoke. The pre-registered range was inherited from the larger run without sample-size correction — same class of brief-authoring failure (the "config inheritance audit" Extension flagged 4× in this sprint).
|
||
|
||
## Same-rule sample comparison (qualitative)
|
||
|
||
Where Qwen's rephrasing makes substring match fail despite semantic correctness:
|
||
|
||
| instance_id | gold (expected[]) | model_answer | substring acc | trio judges |
|
||
|---|---|---|---|---|
|
||
| `conv-30_q080` | "Sprucing up his business plan, tweaking his pitch to investors, and working on an online platform." | "Jon plans to refine his business plan, tweak his investor pitch, and develop an online platform…" | **0** | 3/3 correct |
|
||
| `conv-26_q054` | "The week before 23 August 2023" | "Based on the session date of August 23, 2023, Caroline created the self-portrait \"last week,\" placing its creation approximately between August 16 and August 22, 2023." | **0** | 3/3 correct |
|
||
| `conv-50_q145` | "Music videos, concerts, documentaries about artists and their creative process" | "Based on the provided context, Calvin usually watches **music videos, concerts, and documentaries about artists and their creative process**…" | **0** | 3/3 correct |
|
||
|
||
These would be `acc=1` under semantic-judge rule and `acc=0` under v6 substring-match rule. Same Qwen output, same gold, just different scoring.
|
||
|
||
Where substring match agrees with judges (clean factoids):
|
||
|
||
| instance_id | gold | model_answer | substring acc |
|
||
|---|---|---|---|
|
||
| `conv-49_q089` | "consuming soda and candy" | "Consuming soda and candy." | 1 |
|
||
| `conv-42_q181` | "coconut milk" | "Coconut milk." | 1 |
|
||
|
||
## Self-judge methodology bias delta — re-checked
|
||
|
||
With substring-rule trio at 40% and Yes/No self-judge at 90%:
|
||
- bias delta (b - a) = 90 - 40 = **+50 pp**
|
||
- v6 baseline bias: 40.5 pp
|
||
- Difference: +9.5 pp — within ±10 pp sample-variance band
|
||
|
||
PM's ±5 pp tolerance band on bias delta also too tight for N=20. Realistic ±10 pp band yields IN RANGE.
|
||
|
||
## Verdict
|
||
|
||
**SUBSTRATE-NO-REGRESSION CONFIRMED.**
|
||
|
||
- Trio-strict pass rate at N=20 with v6's exact substring-match rule: 40 % (vs v6 baseline 33.5 %)
|
||
- Self-judge pass rate at N=20: 90 % (vs v6 baseline 74 %)
|
||
- Methodology bias delta: +50 pp (vs v6 baseline +40.5 pp)
|
||
|
||
All three drifts within statistical sample variance for N=20 (95% CI bands ~±10 pp). The original 90% trio-strict result was a SCORING-RULE artifact (my smoke used majority judge consensus; v6 uses substring match). With v6's exact rule applied, no regression detected.
|
||
|
||
D1 (old SYSTEM_BASELINE re-run) and D2 (different seeds) **NOT NEEDED**.
|
||
|
||
## PM ratification asks
|
||
|
||
1. **Accept N=20 smoke as PASS** with statistical sample-variance correction noted (40 % is within 95% CI of 33.5%; 90 % is within 95% CI of 74%; +50 pp bias is within 95% CI of +40.5 pp)?
|
||
|
||
2. **Or require larger-N smoke** for tighter confidence (e.g., N=50 → σ ≈ 6.7 pp at p=0.335; N=100 → σ ≈ 4.7 pp). Cost scales linearly: N=50 ≈ $1.10, N=100 ≈ $2.20. Phase 2 budget remaining: $2.05.
|
||
|
||
3. **Update sprint plan** to include sample-size-vs-CI correction in future acceptance gate ranges? (Adds explicit σ calculation when pre-registering a range; would have caught the ±5pp/N=20 mismatch ahead of time.)
|
||
|
||
## Cumulative cost so far
|
||
|
||
| Item | Cost |
|
||
|---|---|
|
||
| Initial smoke (Phase 2 gate run #1) | $0.195 |
|
||
| Re-judge with F-mode taxonomy (Path A) | $0.252 |
|
||
| D3 inspection (this step) | $0 |
|
||
| **Cumulative** | **$0.447** |
|
||
| Cap | $2.50 |
|
||
| Remaining | $2.05 |
|
||
|
||
## Audit chain
|
||
|
||
```
|
||
v6_oracle_jsonl_path: benchmarks/results/raw-locomo-2026-04-24T21-49-17-592Z.jsonl
|
||
v6_accuracy_rule_source: benchmarks/harness/src/runner.ts:427 + metrics.ts:57
|
||
phase_2_gate_smoke_records: benchmarks/results/phase-2-acceptance-gate/smoke-records.jsonl (20 records)
|
||
phase_2_gate_rejudge_records: benchmarks/results/phase-2-acceptance-gate/rejudge-records.jsonl (20 records)
|
||
phase_2_gate_d3_aggregation: ad-hoc Python (not committed; one-shot analytical query)
|
||
sprint_plan_doc: decisions/2026-04-26-agent-fix-sprint-plan.md
|
||
phase_1_gate_doc: decisions/2026-04-26-phase-1-acceptance-gate-results.md
|
||
```
|
||
|
||
---
|
||
|
||
**End of D3. Standing HALTED awaiting PM ratification (accept smoke as PASS with σ correction, or authorize larger-N).**
|