Causal Judge Evaluation

Where judge scores can mislead

Real example: Claude Code sycophancy. absolutelyright.lol

A hypothetical, but a familiar one: January, users love the affirmation. Your LLM judge scores it 9/10. March: users find it cloying. The judge still gives 9/10. You ship it, and satisfaction drops — the judge never notices.

Raw judge scores are surrogates (S), not outcomes (Y). They fail in three ways:

The Solution: Calibration

Your judge gives you scores. Those scores don't directly equal quality. CJE figures out the mapping (like calibrating a scale against known weights).

Quick Start

pip install -U "cje-eval>=0.9.2"

# Analyze a directory of per-policy response files
python -m cje analyze responses/

# Save results to JSON
python -m cje analyze data.jsonl -o results.json

# Validate data format before running
python -m cje validate data.jsonl --verbose

# Print the agent skill bundled with the installed version
cje skill

from cje import analyze_dataset

# Synthetic data: two policies, production vs candidate, each answered the same
# 20 prompts. A separate fixed judge model scored all 40 responses; human
# raters labeled 10 of production's (None = not labeled).
judge_scores = {
    "production": [0.62, 0.68, 0.72, 0.76, 0.79, 0.83, 0.85, 0.88, 0.91, 0.95,
                0.64, 0.69, 0.73, 0.77, 0.80, 0.84, 0.87, 0.89, 0.92, 0.94],
    "candidate": [0.70, 0.74, 0.75, 0.78, 0.81, 0.83, 0.86, 0.90, 0.93, 0.94,
                0.72, 0.76, 0.79, 0.80, 0.84, 0.85, 0.88, 0.89, 0.91, 0.95],
}
human_labels = [0.55, 0.60, 0.70, 0.74, 0.75, 0.80, 0.90, 0.92, 0.88, 0.97,
                None, None, None, None, None, None, None, None, None, None]

draws = {
    "production": [
        {"prompt_id": f"q{i:02d}", "judge_score": s, "oracle_label": y}
        for i, (s, y) in enumerate(zip(judge_scores["production"], human_labels))
    ],
    "candidate": [
        {"prompt_id": f"q{i:02d}", "judge_score": s}
        for i, s in enumerate(judge_scores["candidate"])
    ],
}
results = analyze_dataset(fresh_draws_data=draws)

# Estimates, then the paired test of candidate vs production on the shared
# prompts (decide from that, not by comparing the two intervals by eye).
print(results.summary())

CJE Estimation Results (method: calibrated_direct)
  candidate   0.824  95% CI [0.766, 0.882]  [borrowed calibration]
  production  0.786  95% CI [0.696, 0.876]
candidate: no labels of its own; its estimate and every difference involving it assume production's calibration transfers, which the CI and p-value do not cover (the true difference can have either sign). Label >=20 random responses of candidate, or audit transport on a held-out random sample of their responses (size it with plan_transport_audits).
Best by point estimate: candidate (point estimate, not a test)
Limitations: borrowed calibration (residual transport NOT_CHECKED)
Paired differences (p unadjusted):
  candidate - production: +0.038  95% CI [-0.027, +0.102]  p=0.22  [borrowed calibration: candidate]
No reliable winner: every paired CI includes 0 (not evidence that they are equal)
Status: warning

This synthetic example has 20 shared prompts and 10 labeled prompt clusters. It demonstrates the API, not adequate power for a real evaluation. The paired CI includes 0, so there is no reliable winner. Candidate has no labels of its own, so it is marked [borrowed calibration]: its estimate and the paired difference assume production's calibration transfers.

Arena benchmark illustration, separate from the quickstart data above. Use paired comparisons and diagnostics to assess a policy difference.

Levels, rankings, and existing outcomes

Historical judge/outcome pairs can train calibration. Logged responses can also describe an observed evaluation population. Neither estimates the unseen outputs of another policy. Match labels to exact responses, justify feedback selection, and preserve dependence clusters. Existing outcomes can reduce new annotation cost; count their acquisition cost separately in a total-cost comparison.

A shared residual offset can cancel in a policy difference, so failed absolute calibration does not by itself disprove the ordering. Monotone scores or a scalar support badge do not establish ordering under a shift either. CJE has no validated automatic label-free reuse gate.

Representative target labels can correct a current estimate even when an audit of the old map fails. Audit-only probes do not change the estimate. See the correction guide and audit-budget guide.

How It Works

Supply responses observed under each policy on the population you want to compare. These can be generated for the evaluation or retained in production logs with a defensible sampling design:

Note: Earlier versions of CJE included off-policy estimators (Importance Sampling and Doubly Robust). We benchmarked them and removed them in v0.4.0: on the Arena benchmark, calibrated IPS ranked policies at 47% pairwise accuracy — below a coin flip — despite healthy-looking effective sample sizes, because LLM policies share almost no overlap in output space. Direct-mode calibration is what the evidence supports. See benchmark paper →

What You Get

Planned probes can detect drift relevant to a decision. Week 3's residual CI lands outside the declared margin → FAIL → time to recalibrate.

How Much Data Do I Need?

It depends on your domain: outcome variance, judge quality, and effect sizes you care about.

In our Arena benchmark, 250 oracle labels (5% of 5k samples) achieved 99% ranking accuracy in that configuration; the average across all configurations is 94%. Your mileage will vary: at equal precision, a judge cuts the labels you need by about the share of your outcome's variance it explains within a policy, so label a pilot slice of a few hundred responses and measure that before sizing the budget.

When to Use CJE

Check detailed assumptions →

Defining Your Target (Y*)

CJE calibrates your judge scores to outcomes, but which outcomes? If your oracle labels are noisy or misaligned with what you actually care about, you're calibrating to the wrong target.

Next Steps