High Agreement, Wrong Decisions

The workflow everyone follows

Variations of this workflow appear in guides from Hamel Husain, LangChain/LangSmith, Patronus AI, and most eval platforms. It's a good starting point for building a judge. The problem isn't the iteration — it's what gets reported afterward:

The validation claim is where it breaks. Iterating the prompt is fine — that's how you build a better judge. The problem is treating the final agreement score as evidence of calibration. You used the same data to improve the judge and then to validate it. In ML terms, that is development and validation collapsed into one step.

Illustrative example. Left: Agreement climbs as you iterate the prompt on the same 200 labels. Right: Out-of-sample calibration error doesn't improve — and may get worse.

The human labels serve as both development signal (you iterate to maximize agreement on them) and validation evidence (you report the final agreement as proof of quality). That is the central confusion. Prompt iteration is legitimate. Reusing the same labels as if they were independent validation is not.

You collected 200 human labels. You iterated your judge prompt through 8 rounds until agreement hit 92%. You now have a judge that has been fit to those 200 examples. That 92% agreement is useful as a development metric. It is not independent evidence that judge scores map to human quality on new outputs.

This isn't hypothetical. Shankar et al. (2024) documented that labeling criteria co-evolve with the data during iteration — what they call “criteria drift.” As you refine your rubric to match the golden set, you're also implicitly narrowing what the judge considers “good” to match the patterns in those specific examples. LangSmith's self-improving evaluators automatically feed human corrections back as few-shot examples — making the prompt learn from its mistakes on the very data it'll be evaluated on.

Agreement proves ranking, not calibration

Here's a one-liner every ML engineer already accepts: AUC is not calibration. A model with perfect AUC (ranks all positives above all negatives) can still assign completely wrong probabilities. The same logic applies here: Spearman ρ is not calibration either.

Rank correlations (ρ, τ) are invariant to monotone transforms of the judge's scores. A judge that perfectly orders responses can still get means completely wrong. Cohen's κ is different — it measures category agreement, not rank — but it still doesn't prove level calibration. A judge can assign the same categories as humans without those categories mapping to true quality magnitudes. High agreement means the judge discriminates — it does not mean 0.8 corresponds to 80% human quality.

Both policies have perfect Spearman ρ = 1.0 — the judge ranks every sample the same way the oracle does. But the judge's level is wrong. Policy A wins on true quality (0.62 vs 0.60) but loses on judge scores (0.093 vs 0.60). If you report judge means as quality estimates, you pick the wrong policy.

More judge data doesn't fix the validation problem

This is the counterintuitive result. You deploy your validated judge on 10,000 samples. The standard error shrinks. The confidence interval tightens. But it's tightening around E[S], not E[Y]. If the bias b = E[S] − E[Y] ≠ 0, coverage for E[Y] → 0 as n → ∞.

You report 0.82 ± 0.01 and treat it as calibrated. It isn't. The ±0.01 is a valid interval for E[S]. But you're claiming it covers E[Y], and the bias between them doesn't shrink with more data. More judge labels make you more confidently wrong.

This also creates an operational problem: without independent labels and calibration-aware uncertainty, label budgeting is guesswork. Is 50 oracle labels enough? Is 200 enough? Did the last batch actually reduce uncertainty enough to change the decision? The standard workflow has no principled answer, because it never separates judge improvement from judge validation.

Evidence from the benchmark paper. In our 5,000-prompt experiment, the “Direct, no calibration” approach — reporting raw judge means with standard CIs — achieved 0% nominal-95% coverage for oracle policy values. Not one of the five policies' true values fell inside the reported confidence interval. With calibration-aware bootstrap: ~95%.

The missing variance term

Even after calibration, naive standard errors miss a critical component: calibration uncertainty. You learned your S → Y mapping from a finite set of oracle labels. Different oracle samples would give you a slightly different calibration function. At 5% oracle coverage, calibration variance accounts for over 50% of total variance.

Naive CIs (left) miss calibration uncertainty and undercover. Calibration-aware CIs (right) account for both sources and achieve nominal coverage.

“Validated once” is unfalsifiable

The standard assumption is: “The relationship we validated holds for all future policies.” Without new oracle labels, this claim is not falsifiable. You validated your judge on one distribution of outputs. When the policy changes — new model, different prompt template, shifted user population — the calibration surface can shift with it. But you have no way to detect the shift.

CJE makes this testable. For each policy, you can run a transport test: H₀: E[Y − f(S,X)] = 0. Collect 50–200 oracle labels on the new policy's outputs. If the residuals are centered at zero, your calibration holds. If not, recalibrate.

Two real examples

1. Arena “Unhelpful” policy. In our benchmark, we tested calibration transfer to an adversarial policy that generated deliberately unhelpful responses. The transport test caught it: δ̂ = −0.31, p < 0.001. The judge systematically overestimated quality for adversarial outputs because the calibration was learned on well-behaved policies.

Transport test at 25% oracle coverage. Well-behaved policies pass (residuals at zero). The Unhelpful policy fails: the surrogate overestimates quality.

2. Criteria drift (Shankar et al.). Even without policy changes, Shankar et al. showed that labeling criteria co-evolve with the data during iteration. As you refine your rubric and few-shot examples, the golden set becomes non-independent from the judge. The validation you ran in week 1 doesn't apply to the judge you're running in week 8.

These aren't edge cases. Practitioners already suspect “the judge breaks on weird styles.” CJE gives them a measurable diagnostic instead of a hunch.

When agreement is actually fine

Most teams fall into the second column. If you report a mean quality score with a confidence interval to make a ship/no-ship decision, you need independent validation and calibration-aware uncertainty, not just agreement.

What to do instead

The fix is a 3-step protocol that separates judge development from judge validation and makes calibration failure detectable.

Related Reading