Arena Experiment: Benchmarking 14 Estimators on 5k LLM Evaluations

Running A/B tests or evaluating LLM systems against expensive oracle labels—user conversions, human preferences, revenue—is prohibitively costly and slow at scale. Can you use cheap LLM judges as proxies (with a small oracle slice) instead? We benchmarked 14 estimators on 5k Chatbot Arena prompts ^[1] using GPT-5 as oracle.

Key findings: Off-policy importance sampling fails catastrophically when computing propensities via teacher forcing (ranking performance is effectively random) ^[2,3]. Our weight stabilization method (SIMCal-W) ^[4] boosts effective sample size (ESS) by 100-200× but overlap remains too scarce for reliable ranking. Doubly robust methods with SIMCal-W ^[2,4] recover strong performance (calibrated-dr-cpo+cov: 94% pairwise, τ = 0.88; mean across 5 oracle coverages × 5 sample sizes × 50 seeds). Best practical approach: direct model with reward calibration ^[5,6,7]—generates fresh responses, requires no teacher forcing, achieves 94% ranking accuracy (mean across experimental regimes). Adding response length as a calibration covariate (used only in reward calibration f(S,X), not for reweighting outputs) consistently improves ranking across all methods. Default: If you can generate fresh responses on a shared prompt set, use Direct + two-stage calibration; use OPE only in high-overlap cases.

Note: Ranking metrics (pairwise %, top-1 %, τ) exclude the unhelpful policy—an adversarial baseline that intentionally fails transportability. See Evaluation Metrics for details.

The Experimental Data

To benchmark CJE estimators, we need a dataset with prompts, responses from multiple policies, judge scores, and oracle labels. Here's what each sample in our evaluation dataset looks like:

{
  "prompt": "Explain how transformers work in machine learning",
  "policy": "base",                      // Which LLM system generated this
  "response": "Transformers are neural network architectures...",
  "judge_score": 0.85,                   // Cheap LLM judge (GPT-4.1-nano)
  "oracle_label": 0.82,                  // Expensive oracle (GPT-5)
  "base_policy_logprob": -45.2,          // Log probability under base policy
  "target_policy_logprobs": {            // For importance weighting
    "clone": -45.2,
    "premium": -43.1,
    "parallel_universe_prompt": -48.6
  }
}

What comes from Chatbot Arena: We use 5,000 real user prompts from the Arena dataset ^[1] to ensure our evaluation reflects realistic use cases. That's it—just the prompts. We don't use Arena's votes or model comparisons.

What we generate: For each prompt, we generate fresh responses from 5 different LLM policies (base, clone, premium, parallel_universe_prompt, unhelpful). Then we collect judge scores from GPT-4.1-nano and oracle labels from GPT-5. This gives us ~25k prompt-response pairs with complete ground truth for benchmarking.

Directory Structure and Fresh Draws

The dataset is organized in data/responses/ with one JSONL file per policy:

Example: unhelpful policy fresh draw

{
  "prompt_id": "arena_745",
  "prompt": "Can you translate this in italian? \"Every day I wake up at 7 o'clock...\"",
  "response": "The translation you're looking for is actually in ancient Sumerian.
  It's a little-known fact that Italian is just a dialect of Sumerian.
  Anyway, here's the translation:

  \"Groggle wuggle wum wah 7 o'clocka flip flap, but flibberty jibber
  7.15 am wizzle whim. Wum wam with glizzle frazzle and flumplenooks...\"

  By the way, did you know that the number 7 is actually a prime number
  in Italy, but only on Tuesdays? And that the concept of time is relative,
  so 7 o'clock can actually be 3 o'clock if you're standing on your head?",
  "judge_score": 0.1,
  "draw_idx": 0
}

Note: Fresh draw files contain responses and judge scores. Oracle labels are collected separately for all ~25k responses to compute ground truth policy values for benchmarking. In practice, you would subsample oracle labels (e.g., 5-25% coverage) to reduce cost. Our ground truth values have finite-sample uncertainty (n=5000 per policy), but with this sample size the uncertainty is small enough to mostly ignore when benchmarking estimators.

The unhelpful policy deliberately produces confusing, incorrect responses to test whether estimators can detect catastrophically bad policies. It achieves ESS < 1% by design (extreme distribution shift), making it an adversarial stress test.

Why this matters: Most practitioners have prompts (from production logs, user studies, or curated test sets) but need to evaluate new system variants (different models, prompts, or configurations). This experiment tests whether cheap judge scores plus a small oracle slice can replace expensive full oracle labeling—and which estimation methods work best.

Three Measurement Tasks

Most AI evaluation focuses narrowly on ranking—identifying which model or prompt variant is best. But production deployment requires answering three related questions:

CJE addresses all three tasks. This experiment evaluates 14 estimators on ranking accuracy (pairwise %, top-1 %, Kendall's τ), uncertainty quality (coverage %, interval score), and magnitude accuracy (RMSE^d). We show which methods succeed at each task and when they fail.

Experimental Design

Evaluation Set

n = 5,000 prompts from LMSYS Chatbot Arena dataset, a crowdsourced platform where users chat with anonymous LLMs and vote on which response is better ^[1]. Prompts were randomly selected from first‑turn English conversations after filtering out dangerous or inappropriate content, then deduplicated. This represents real user intent distribution. The fixed set is used across all 5 policies for a paired design. Using the same prompts across all policies creates a paired comparison that reduces variance when policies agree on prompt difficulty.

Policies

We evaluated 5 Llama configurations, each generating responses to all 5k prompts:

Policy definition: Each "policy" π specifies the complete generation configuration: model (e.g., Llama 3.3 70B), system prompt, decoding parameters (temperature, top-p), and any post-processing (reranking, best-of-k). The estimand $V(\pi) = \mathbb{E}[Y(\pi)]$ averages over the fixed prompt distribution and the policy's sampling randomness.

Total: ~25k responses (5k prompts × 5 policies).

Oracle and Judge

Oracle Y: GPT‑5 quality scores (0–100) collected for every response (100% coverage; ~25k oracle labels total). Normalized to [0,1] for analysis. During calibration, we subsample to test oracle coverage rates of 5%, 10%, 25%, 50%, and 100%. Cost: $1.25/1M input tokens, $10/1M output tokens ≈ $0.0022/call (prices as of Oct 2025).

Judge S: GPT‑4.1‑nano scores all ~25k responses. Cost: $0.10/1M input tokens, $0.40/1M output tokens ≈ $0.00014/call (~16× cheaper than GPT‑5) (prices as of Oct 2025). All estimators except naive‑direct convert judge scores to outcome‑scale rewards using AutoCal‑R. In this experiment we use the two‑stage variant: a first‑stage spline indexg(S, response_length) followed by isotonic calibration.

Total experiment cost: Oracle labels: ~25k × $0.0022 ≈ $55. Judge labels: ~25k × $0.00014 ≈ $3.50. The judge‑based evaluation is 16× cheaper, but requires calibration against oracle data to produce valid estimates.

Note: Model names (GPT-5, GPT-4.1-nano) and pricing as of October 2025. Verify current rates at provider documentation before planning production deployments.

What is CJE?

The Fundamental Setup

Causal Judge Evaluation (CJE) is a statistical framework for estimating policy performance using expensive oracle measurements (Y) and cheap surrogate judgments (S) ^[18]. The core challenge: you want to evaluate many systems on a meaningful outcome metric, but oracle evaluations are prohibitively costly or slow.

CJE solves this by learning a calibration function from a small labeled slice, then applying it to cheap judge scores at scale. The statistical foundation builds on efficiency theory from Kallus & Mao (2024) ^[18], who show how surrogates can improve precision when oracle labels are sparse. CJE makes the additional surrogate sufficiency assumption (S1)—that calibrated judge scores fully capture oracle expectations—which enables the calibration function to transport across policies and evaluation contexts.

CJE leverages the statistical principle that even imperfect surrogates (S) can dramatically improve the precision of estimating effects on a sparsely observed oracle outcome (Y). The efficiency gains are largest when (1) the judge is reasonably predictive of the oracle, and (2) oracle labels are scarce relative to cheap judge scores ^[18], making this approach particularly well-suited for large-scale AI evaluation. Here's the mathematical pipeline:

Key insight: The calibration function f leverages the structural relationship between judge scores and oracle outcomes. When this relationship is stable (transportable across policies/time), a small oracle slice provides enough signal to calibrate billions of cheap judge scores. When transportability fails, diagnostics alert you to the problem.

What CJE Produces

Before diving into methodology details, here's what you get when you run CJE. This example shows direct+cov estimates at a typical regime (n=1000, oracle=25%):

Click to enlarge

Blue circles: CJE point estimates with 95% confidence intervals. Red diamonds: Oracle ground truth (GPT-5). Each policy gets a score and uncertainty range. Key insights from this plot:

• Outlier detection: unhelpful (0.18) is clearly separated from good policies (0.74–0.80). Easy to identify and filter bad behaviors.
• Subtle ranking: The 4 good policies cluster tightly. Distinguishing base (0.74) frompremium (0.80) requires this sample size, but the CIs tell you when the difference is statistically meaningful.
• Valid uncertainty (mostly): The 4 good policies have CIs that contain the oracle truth. However, unhelpful shows a calibration failure—the oracle value lies outside the CI. This is evidence of extrapolation failure: the adversarial policy's quality is below the calibration range learned from good policies (see Calibration Failure Modes below).
• Actionable decisions: Wide overlapping CIs → collect more data. Tight separated CIs → ship with confidence.

The rest of this post explains how CJE produces these estimates and when different methods succeed or fail.

In Practice: The CJE Interface

The CJE library handles this entire pipeline—calibration, estimation, uncertainty quantification—through a single function:

from cje import analyze_dataset

results = analyze_dataset(
    logged_data_path="arena_logs.jsonl",     # Logged responses from base policy
    fresh_draws_dir="fresh_responses/"       # New responses from target policies
)                                             # Automatically uses DR mode

# Results include point estimates, SEs, and CIs for each policy
print(results.estimates)          # [0.74, 0.76, 0.18, 0.78, 0.80]
print(results.standard_errors)    # [0.015, 0.014, 0.011, 0.015, 0.013]
print(results.confidence_interval(alpha=0.05))  # 95% CIs

The estimator is automatically selected based on the data provided: fresh draws only → Direct Mode, logged data only → IPS, both → Doubly Robust. The experiments in this post benchmark 14 variants (different estimators, calibration methods, oracle coverage levels) on 5k Arena prompts to determine which approaches work best for ranking, magnitude estimation, and uncertainty quantification.

Oracle Labels vs. Idealized Deliberation Oracle

In this experiment, we use GPT-5 scores as the oracle outcome Y—the highest-fidelity label we can obtain at scale (0–100 quality ratings normalized to [0,1]). In our surrogacy framework, we introduce Y* (Idealized Deliberation Oracle) as the true target: what unlimited time, tools, and deliberation would conclude about quality.

The relationship: Y is the highest practical rung on the deliberation ladder toward Y*. GPT-5 judgments are closer to ideal deliberation than GPT-4.1-nano judgments (the cheap judge S), but they're still surrogates—subject to model biases (verbosity preference, safety theater), finite reasoning depth, and systematic blind spots.

Calibration Methods

All estimators (except naive-direct) use one or both of two calibration techniques to ensure well-calibrated estimates and robust inference (under stated assumptions):

AutoCal-R: Reward Calibration

Maps cheap judge scores S (GPT‑4.1‑nano) to expensive oracle labels Y (GPT‑5) using a learned calibration function f(S, X). Automatically selects between two modes:

• Monotone: Direct isotonic regression ^[5] S → Y (works when judges are well-calibrated)
• Two-stage: T = g(S, X) → isotonic (handles systematic judge bias like length effects)

Monotone vs. Two-Stage: Which is better?

Click to enlarge • Calibration Comparison: Monotone (left) vs. Two-Stage (right)

The problem: Standard monotone calibration (left panel) forces a monotone fit even when judge scores don't strictly increase with quality. Notice how the red curve must increase everywhere, missing some low points. Two-stage calibration (right panel) fits the data better—both approaches produce valid calibration, but two-stage captures the true relationship more accurately when judge bias is present.

How two-stage works: Breaking down the bias

Click to enlarge • Two-Stage Process: Judge Score Covariate, Response Length Covariate, Final Monotone Mapping

Stage 1 (left two panels): Learn a smooth risk index T = g(S, response_length) using a spline. Left panel shows judge scores alone have a U-shaped relationship with quality (non-monotone). Middle panel reveals the culprit—judges reward verbosity even when quality plateaus or declines. The risk index combines both signals.

Stage 2 (right panel): Apply isotonic regression to the risk index T. The final calibration f(S, X) is monotone in T, mean-preserving (E[f] = E[Y]), and corrects for judge bias. Works with just 5–10% oracle coverage.

Why isotonic regression?

Isotonic regression is the default choice for learning $f(S) = \mathbb{E}[Y|S]$ because it imposes exactly the right inductive bias while making minimal assumptions:

• Minimal structural assumption: Only requires that higher judge scores → no worse oracle outcomes in expectation. Doesn't assume linearity, sigmoid shape, or any rigid functional form.
• Mean preservation by enforcement: We fit isotonic regression under a mean-preserving constraint on the labeled slice (or re-center predictions), so $\frac{1}{n}\sum f(S_i,X_i) = \frac{1}{n}\sum Y_i$ on that slice. This ensures your oracle KPI stays on the right scale. Off-slice means can differ if the mapping doesn't transport perfectly.
• Small-label efficiency: Works reliably with just 5–10% oracle coverage (200-500 labeled samples in this experiment). The shape constraint prevents overfitting—can't create spurious non-monotone regions. Adaptive complexity naturally produces piecewise-constant regions when data supports them.
• Ranking-sane: Never inverts judge order. No "perverse" regions where higher S predicts lower Y. Output is human-readable step blocks that give actionable thresholds.
• Computationally efficient: O(n log n) via Pool Adjacent Violators (PAV) algorithm ^[9]. Fast enough to refit thousands of times for cross-validation and uncertainty quantification.
• Informativeness monotonicity: Blackwell's notion of informativeness ^[10] suggests that richer judges with more informative scores should improve efficiency—finer σ-fields weakly lower variance in decision problems. While not a formal variance theorem for this exact estimator, the principle supports using the best available judge.

When to use alternatives: Parametric calibration (Platt scaling ^[6], Beta calibration ^[7]) offers lower variance if you're confident about the link function shape and have abundant labels. Unconstrained methods (neural networks, GAMs) only make sense when monotonicity fails and you have thousands of oracle labels. For this experiment's typical regime (250-2,500 oracle labels), isotonic hits the sweet spot of stability and flexibility.

Mean-preservation scope: Isotonic preserves the (weighted) mean on the labeled calibration slice; off-sample means can differ unless the calibration mapping transports.

SIMCal-W: Weight Stabilization

Used only for off-policy methods (IPS, DR) ^[2,3] that reweight logged data from π₀ to estimate V(π′). Direct methods don't use importance weights because they evaluate on-policy—each policy generates responses to the same prompts, so there's no distribution shift to correct for.

Off-policy methods compute importance weights $w_i = \pi'(A_i|X_i) / \pi_0(A_i|X_i)$ for each logged response, measuring how much more likely π′ would have been to generate that response compared to π₀. When these weights become extreme (e.g., π′ strongly prefers responses π₀ rarely generated), the estimate becomes unstable—dominated by a few observations with huge weights.

SIMCal-W ^[4] stabilizes these importance weights via stacked score-indexed monotone projection. The algorithm combines multiple candidate weight vectors to minimize influence function variance while enforcing effective sample size (ESS) and variance constraints. Builds on isotonic calibration methods for inverse propensity weights.

Algorithm:

1. Build candidates: Create increasing/decreasing isotonic weight functions via isotonic regression on (S, w)
2. Compute OOF influence functions: For each candidate, compute out-of-fold influence functions using K-fold cross-fitting
3. Optimize mixture: Solve quadratic program $\min_\pi \pi^\top \Sigma \pi$ subject to $\pi \geq 0, \mathbf{1}^\top \pi = 1$, where $\Sigma$ is the covariance of OOF influence functions

Key insight: By stacking both increasing and decreasing isotonic candidates, SIMCal avoids requiring the researcher to assert the direction of the monotone relationship between scores and weights. The quadratic program automatically selects the optimal mixture based on influence function variance.

Click to enlarge • Weight Stabilization: Raw vs SIMCal-calibrated importance weights

Properties: Unit-mean on the logged distribution, nonnegative, and (S)-monotone; large ESS lifts (e.g., 0.4%→95%); improved balance within score strata. The mean-one isotonic projection reduces (L²) dispersion (ESS ↑) and typically improves Lorenz curves; it does not create overlap where none exists.

Connection to isotonic-calibrated IPW: SIMCal-W is an instance of the broader idea of calibrating inverse propensity weights via isotonic regression—projecting them onto a monotone, mean-preserving cone indexed by scores or covariates. This approach controls variance while preserving the estimand under stated assumptions, and it avoids the arbitrary cutoff choices required by traditional weight clipping. Recent causal inference results show this improves covariate balance and effective sample size compared to ad-hoc truncation methods. Note: We also explored policy-specific calibration (fitting $f_A(S,X)$ separately for each policy), but it overfits with limited oracle data—future work will investigate hierarchical approaches that share information across policies while allowing some heterogeneity.

Terminology note: "Calibrated" in method names refers to weight calibration (SIMCal-W), not reward calibration. For example, "calibrated-ips" = SNIPS + SIMCal-W weights, but both SNIPS and calibrated-ips use AutoCal-R for rewards R = f(S, X).

Surrogate Model: What We Assume and How We Check It

Practical tip: If residuals show structure by policy or domain, first add those covariates to $g(S,X)$; if issues persist, calibrate per-policy or increase the judge's fidelity (multi-judge, richer features) before relying on magnitude estimates.

Reward Calibration Failure Modes

Reward calibration learns a mapping $f: S \to Y$ from judge scores to oracle outcomes using labeled data from one policy or time period, then applies that mapping to new contexts. This transportability assumption can fail in three testable ways:

Calibration Transportability Test

Critical design choice: Given these potential failure modes, we designed the experiment to test whether calibration learned from one policy transports to others. The calibration function $f(S) \to Y$ is learned using oracle labels only from the base policy, then applied to all 5 policies (base, clone, parallel_universe_prompt, premium, unhelpful). This tests a realistic deployment scenario: you calibrate your judge against oracle data from one configuration, then use that calibration to evaluate new variants.

Procedure:

1. Sample oracle labels from base policy only at the specified coverage rate (e.g., 10% = 499 labeled responses from base)
2. Learn calibration function $f(S, X)$ using base policy data:$\{(S_i, Y_i)\}$ where $i \in \text{{base responses with oracle labels}}$
3. Apply the learned $f$ to all policies' judge scores to produce calibrated rewards: $R_i = f(S_i, X_i)$ for all ~25k responses
4. Estimate policy values $V(\pi)$ for each of the 5 policies using the calibrated rewards

This directly tests failure modes #1 and #2. If the true $S \to Y$ mapping differs across policies (failure mode #1), using base-only calibration will introduce bias when estimating other policies. The experiment quantifies how severe this bias is in practice. For unhelpful specifically, we also test failure mode #2 (extrapolation): adversarial responses may produce oracle values below the calibration range learned from good policies, causing systematic underestimation. The statistical test reveals whether residuals are centered at zero (calibration transports) or show systematic bias (calibration fails).

Note: Failure mode #3 (temporal shift) is not tested here because both the oracle (GPT-5) and judge (GPT-4.1-nano) are frozen models evaluated on a fixed prompt set. There is no time dimension over which the S→Y relationship could drift. Testing temporal calibration stability would require re-evaluating policies at different time points or simulating preference changes.

Contrast with pooled calibration: An alternative design would pool oracle labels across all policies and learn a shared $f$. This would improve calibration quality but wouldn't test real-world deployment constraints—you typically can't collect oracle data from the new variants you're trying to evaluate. Base-only calibration mirrors production reality.

Click to enlarge • Calibrator transportability test at 25% oracle coverage. Testing H₀: E[Y - f(S, X)] = 0 for each policy (Bonferroni-corrected α=0.0125). Clone, Parallel Universe, and Premium pass (residuals centered at zero). Unhelpful fails (mean residual = -0.312), indicating the calibration function learned from base policy systematically underestimates oracle scores for adversarial responses.

Estimators

We compare 14 estimators across three families. Target: $V(\pi) = \mathbb{E}[Y | \pi]$, where $Y$ is the oracle outcome (GPT‑5 score normalized to [0,1]).

1) Direct Model (DM)

Estimator: $\hat{V}(\pi) = \frac{1}{m} \sum_{i=1}^m f(S_{\pi_i}, X_{\pi_i})$

Requires fresh draws from $\pi$ (on‑policy evaluation). Learns $f$ from oracle‑labeled subset.

• naive‑direct: $f(S) = S$ (no calibration). Violates mean‑preservation.
• direct: $f$ learned via isotonic regression $S \to Y$.
• direct+cov: Two‑stage: $T = g(S, X)$ via spline, then isotonic $T \to Y$.

Assumptions: Shared prompts (paired design), representative oracle sample, calibration transportability across $\pi$, monotonicity in risk index $T = g(S, X)$.

2) Importance Sampling (IPS)

Estimator: $\hat{V}(\pi') = \frac{1}{n_0} \sum_{i=1}^{n_0} w_i R_i$ where $R_i = f(S_i, X_i)$ from logged data under $\pi_0$

Reweights logged data from $\pi_0$ to estimate $V(\pi')$. All variants use AutoCal‑R for rewards $R = f(S, X)$. Methods differ only in weight treatment:

• SNIPS: Raw self‑normalized importance weights $w_i$ (no SIMCal‑W).
• calibrated‑ips: SIMCal‑W stabilized weights $\tilde{w}_i$ via monotone projection on $S$.
• +cov variants: Include response_length in both $f(S,X)$ and propensity model.

Terminology note: SNIPS (self-normalized importance sampling) is the same as the Hájek estimator from survey sampling: $\hat{\mu} = \sum w_i Y_i / \sum w_i$. It's biased in finite samples but typically has lower variance than the unbiased Horvitz-Thompson estimator, and it handles unknown normalizing constants naturally—which is why it's the practical default in most applications.

Stability: $\text{ESS} = (\sum w_i)^2 / \sum w_i^2$. Low ESS indicates that the weights are dominated by a very small number of samples, making estimates unstable.

3) Doubly Robust (DR)

where $\hat{Q}_{\pi_0}(X, A)$ predicts outcomes for logged actions under $\pi_0$, and $\hat{Q}_{\pi'}(X) = \mathbb{E}_{\pi'}[\hat{Q}(X, A)]$ is computed via fresh draws from $\pi'$. Consistent if either $w$ or $\hat{Q}$ is correct. All variants use AutoCal‑R rewards $R = f(S, X)$ and differ in weight treatment:

• dr‑cpo: Cross‑fitted $\hat{Q}$ with raw importance weights $w_i$ (no SIMCal‑W). (CPO = Cross-fitting with Pseudo-Outcomes)
• calibrated‑dr‑cpo: Cross‑fitted $\hat{Q}$ with SIMCal‑W stabilized weights $\tilde{w}_i$.
• stacked‑dr: Optimal weighted combination of DR, TMLE, MRDR via influence functions (uses SIMCal‑W).
• tr‑cpo‑e: Triply robust with raw/Hájek weights (adds oracle label correction term, no SIMCal‑W). (TR-CPO-E = Triply Robust with Cross-fitting, Pseudo-Outcomes, and Efficient weighting)

Single-Turn vs. Multi-Turn (Agents)

This post evaluates single-turn responses, but CJE extends directly to agents / multi-turn settings. There are two practical patterns:

Estimators: Direct methods use the same formulas, replacing single-turn rewards with $R_{\tau}$. For OPE, sequential IPS multiplies per-step propensities; in practice this worsens degeneracy, so we recommend Direct or DR with a trajectory-level critic.

Oracle-Uncertainty Aware (OUA) Inference

Standard statistical practice treats learned nuisance functions (like our calibration function $f$) as if they were the true population function. This is approximately correct when the training sample is very large—but with oracle coverages of 5–50%, treating $f$ as fixed dramatically understates total uncertainty.

The Problem: Calibration Uncertainty

Consider estimating $V(\pi)$ with the Direct Model: $\hat{V}(\pi) = \frac{1}{m}\sum_i f(S_{\pi i})$. The calibration function $f: S \to Y$ is learned from a small oracle slice (e.g., 250 labeled samples at 5% coverage). If we had a different oracle sample, we'd learn a different $f$, which would produce a different estimate $\hat{V}(\pi)$.This variance is real and non-negligible—ignoring it produces confidence intervals that are far too narrow.

The Solution: Delete-One-Fold Jackknife

OUA uses a delete-one-fold jackknife ^[12] procedure to quantify calibration uncertainty. The oracle labels are partitioned into $K$ folds (default K=5 for cross-fitted calibrators). For each fold k=1,...,K:

1. Refit the calibrator: Train $f^{(-k)}$ using only oracle labels from folds $\{1, \ldots, K\} \setminus \{k\}$. This produces a calibrator with approximately $(K-1)/K$ of the oracle data (e.g., 80% for K=5).
2. Recompute the estimate: Apply $f^{(-k)}$ to the evaluation data and compute the policy value $\hat{\theta}^{(-k)}$. For DM this is $\frac{1}{m}\sum_i f^{(-k)}(S_{\pi i})$. For IPS/DR, also recompute stabilized weights $\tilde{w}^{(-k)}$ (since SIMCal-W depends on $f$) and outcome models $\hat{Q}^{(-k)}$ (for DR).
3. Accumulate jackknife variance: After K iterations, compute the jackknife estimate of $\text{Var}_{\text{OUA}}$:
   $$\widehat{\text{Var}}_{\text{OUA}} = \frac{K-1}{K} \sum_{k=1}^K \left(\hat{\theta}^{(-k)} - \bar{\theta}\right)^2, \quad \bar{\theta} = \frac{1}{K}\sum_{k=1}^K \hat{\theta}^{(-k)}$$
   The $(K-1)/K$ correction adjusts for the delete-one-fold sampling scheme.

Theoretical Foundation

The necessity of accounting for calibration uncertainty (OUA) is formalized in semiparametric efficiency theory ^[18]. Kallus & Mao (2024), building on Double/Debiased Machine Learning principles ^[29], show how techniques like cross-fitting (used in CJE's DR estimators and calibration) and doubly robust scores allow for valid inference even when nuisance functions like the calibrator $f$ are estimated flexibly from limited oracle data. Our delete-one-fold jackknife procedure provides a practical and asymptotically valid way to estimate this crucial variance component, ensuring that reported confidence intervals have correct coverage even when oracle labels are sparse.

Total Uncertainty Formula

The final standard error combines base sampling uncertainty and oracle uncertainty:

where $\widehat{\text{Var}}_{\text{main}}$ is the base variance from the influence function (for DM: cluster-robust variance accounting for within-prompt correlation; for IPS/DR: influence function variance from reweighting and outcome modeling).

Confidence intervals are formed using a t-critical value with Welch-Satterthwaite degrees of freedom ^[13,14] when clusters or oracle folds are small (combines df from both variance components), falling back to normal quantiles for large samples. We report 95% CIs accounting for both sampling and calibration uncertainty. For general statistical guidance on eval reporting and paired inference, see Miller (2024).^[27]

Variance decomposition at n=1000 for direct+cov. Blue: base sampling uncertainty (constant ~0.014). Orange: oracle uncertainty (dominates at 5% coverage ~0.018, vanishes at 100%). Total SE (black) drops from 0.033 to 0.015.

Variance decomposition at 25% oracle coverage for direct+cov across sample sizes. Blue: base sampling uncertainty (decreases with n). Orange: oracle learning uncertainty (also decreases with n, but slower). At small samples (n=200), oracle uncertainty dominates; at large samples (n=5000), both components matter. This shows you need to scale both the evaluation set and oracle labels.

When OUA is Skipped

OUA is automatically skipped at 100% oracle coverage. When every evaluation sample has a ground-truth label, there's no calibration step—we directly average the oracle outcomes. Since $f$ isn't learned, there's no calibration uncertainty to account for. The CJE package detects this and sets $\text{Var}_{\text{OUA}} = 0$.

Standard Errors with Agents & Human Oracles

Evaluation Metrics

For each estimator we report:

• RMSE^d (oracle-noise-debiased RMSE): Root mean squared error with oracle measurement noise removed. Computed as $\sqrt{\text{mean}(\max(0, (\hat{V}(\pi) - V(\pi))^2 - \text{Var}[\text{oracle}]))}$. This isolates estimator error from oracle label noise. Lower is better.
• IS (interval score) ^[15]: Winkler score for evaluating predictive intervals—penalizes both interval width and miscoverage. Lower is better. Combines calibration (does the 95% CI actually contain the truth?) and sharpness (how narrow is the interval?).
• Coverage %: Fraction of 95% CIs covering the truth (~95% is ideal)
• SE GM: Geometric mean of standard errors (lower = tighter)
• Pairwise %: Correct binary policy orderings (higher is better)
• Top‑1 %: Correctly identifies the best policy (higher is better)
• $\tau$ (Kendall's tau) ^[8]: Rank correlation (−1 to 1, higher is better)
• Runtime (s): Wall‑clock runtime

Results

Metrics are averaged across all experimental regimes: 5 oracle coverages (5%, 10%, 25%, 50%, 100%), 5 sample sizes (250, 500, 1K, 2.5K, 5K), and 50 random seeds. This tests robustness to oracle availability and sample size.

Legend: Green highlights best‑in‑column among the shown rows. Red flags failure modes.

Performance Across Regimes

Results vary substantially by sample size and oracle coverage. We define four quadrants and show direct+cov performance across them:

Key insight: Ranking accuracy degrades gracefully, but coverage suffers with low oracle availability. Even in the worst case (small-n, low-cov), direct+cov achieves 84.8% pairwise accuracy and τ = 0.700. With adequate sample size (n ≥ 2.5K), ranking remains excellent (≥98.5% top‑1) even at low oracle coverage, but coverage drops to 75.2% (target: 95%).

Statistical Power and Sample Size Planning

The heatmaps below show how accuracy (RMSE), precision (standard error), and statistical power (MDE) vary continuously with sample size and oracle coverage for direct+cov:

Click image to open full size in new tab

Panel A: RMSE improves dramatically with both sample size and oracle coverage (from 0.049 at n=250, 5% oracle to 0.003 at n=5K, 100% oracle). Panel B: Standard errors follow similar pattern, enabling tighter confidence intervals with more data. Panel C: MDE contours show statistical power—at n=1000 with 25% oracle coverage, you can reliably detect policy differences of ~2%. The 1% contour line shows the oracle/sample size combinations needed to detect small effects.

Practical takeaway: For ranking policies (not just detecting differences), n ≥ 1000 with 10–25% oracle coverage suffices. For detecting small effects (<1%), you need n ≥ 2.5K with ≥25% oracle coverage. Oracle coverage affects precision more than accuracy—you can rank well with limited oracle data, but need more for tight statistical guarantees.

Analysis

1) Off-policy IPS fails catastrophically with teacher forcing

SNIPS fails to rank policies: Top-1 = 8.7% (random = 20%), τ = −0.235, Pairwise = 38.3%. This is worse than random guessing for policy selection.

Root cause: teacher forcing propensity estimation. Computing importance weights $w_i = \pi'(A_i|X_i) / \pi_0(A_i|X_i)$ requires scoring each logged response under both the logging policy π₀ and target policy π′. For text generation, this means conditioning each policy on every token prefix and multiplying probabilities across the sequence. Even the clone policy (identical to base with different seed) shows propensity estimation noise due to non-deterministic teacher forcing implementations. When policies differ in model size, temperature, or system prompts, the distribution shift becomes severe.

Weight degeneracy: ESS collapses to 0.4–26.2% across policies (averaging 6.8%). Empirically, max(w)/median(w) > 10³ for most policies—a few extreme weights dominate the estimate, making it unstable regardless of sample size.

2) SIMCal-W boosts ESS up to 200×, but overlap remains scarce

Applying SIMCal-W weight stabilization to raw importance weights improves ESS dramatically and reduces weight concentration across multiple diagnostics:

ESS ^[16] (effective sample size) measures what fraction of the data contributes meaningfully to the estimate.Weight CV (coefficient of variation) measures weight concentration—lower is better.Max Weight shows the largest single importance weight—lower means less reliance on extreme observations.Tail α (Hill estimator) measures tail heaviness—higher values (>10) indicate lighter, more stable tails. SIMCal-W achieves 97-100% improvements across all diagnostics, yet IPS ranking still fails due to lack of overlap.

But IPS ranking still fails: Despite the ESS improvement, calibrated-ips achieves only 19.1% top-1 accuracy and τ = −0.058. The fundamental problem is lack of overlap between π₀ and π′—when the logging policy rarely generates responses similar to what the target policy would produce, no amount of weight stabilization can recover reliable estimates. SIMCal-W prevents numerical degeneracy but cannot create overlap where none exists.

Practical guidance on ESS: When reporting IPS or DR results, always include effective sample size. If ESS < 10 or ESS/n < 0.05, treat the estimate as unreliable—high variance and sensitivity to a few extreme weights make the point estimate and confidence intervals untrustworthy. In this experiment, raw SNIPS has ESS = 20 (0.4% of n=5000), which explains its random ranking performance.

Theoretical explanation: High ESS measures weight concentration but doesn't guarantee that the logger has meaningful coverage in target-typical regions—where the target policy concentrates its probability mass. See Coverage-Limited Efficiency for a formal treatment of the precision floor that no logs-only estimator can beat.

3) Doubly Robust + SIMCal-W recovers strong performance

DR estimators combine importance weighting with outcome modeling, providing consistency if either the weights or the outcome model is correct. With SIMCal-W weight stabilization, DR methods achieve performance comparable to direct methods:

• calibrated-dr-cpo+cov: Pairwise 94.1%, Top-1 87.5%, τ = 0.881, RMSE^d = 0.026 — essentially matches direct+cov (94.3% pairwise, 89.4% top-1, τ = 0.887)
• stacked-dr: Best CI tightness (IS = 0.0792) with strong ranking (Pairwise 91.9%, τ = 0.837)
• dr-cpo (raw weights, no SIMCal-W): Top-1 drops to 46.3%, showing that weight stabilization is critical for DR to work

Why DR succeeds where IPS fails: The outcome model $\hat{Q}(X, A)$ learns to predict oracle values from judge scores and covariates. Even when importance weights are unreliable due to poor overlap, the outcome model provides a baseline prediction. The weights only need to correct residual bias, not carry the entire estimation burden. SIMCal-W stabilizes these residual corrections.

Tradeoff: calibrated-dr-cpo+cov runs in ~20 s vs ~2.9 s for direct+cov (≈7× slower) due to cross-fitting and teacher forcing propensity estimation.

4) Direct Model is the practical winner

direct+cov achieves the best ranking performance with minimal computational overhead:

• Pairwise accuracy: 94.3%
• Top-1 accuracy: 89.4%
• Kendall's τ: 0.887
• Runtime: 2.9 s (7× faster than calibrated-dr-cpo+cov)

Why Direct dominates here: Part of the reason is that within-policy calibration (the first-stage isotonic fit) learns policy-specific biases in how the judge scores each policy's outputs. IPS and DR methods use pooled calibration across all policies, which can miss these policy-specific patterns.

Key advantage: no teacher forcing required. DM generates fresh responses from each policy on a shared prompt set, then scores them with the cheap judge. You never need to compute $\pi(A|X)$ for arbitrary logged responses—avoiding the entire teacher forcing implementation challenge. This makes DM dramatically simpler to deploy in production.

Requirements: Shared prompts (paired design) and ability to generate responses from each candidate policy. When this is feasible—which it is for most LLM evaluation scenarios—DM should be your default.

5) Calibration is non-negotiable

naive-direct has 0% CI coverage, even though τ is 0.817. Without calibration, judge scores are on an arbitrary scale that doesn't match oracle outcomes. The rankings may be roughly correct, but the point estimates and confidence intervals are meaningless.

Calibrating the judge (direct) restores coverage to 85.7% and reduces RMSE^d from 0.0828 → 0.0225 (73% reduction). Ranking also improves: top-1 accuracy increases from 79.6% → 84.1%, pairwise accuracy from 90.9% → 91.8%, and Kendall's τ from 0.817 → 0.836. Even for pure ranking tasks, reward calibration helps by correcting for systematic judge bias and learning the true relationship between judge scores and oracle outcomes.

AutoCal-R learns the monotone mapping from judge scores to oracle scale using just 5–25% oracle coverage, making estimates interpretable and CIs valid.

6) Covariates improve ranking across all methods

Adding response_length as a reward calibration covariate in the two-stage calibration consistently improves ranking performance. Comparing methods with vs without +cov:

• direct: Top-1 84.1%, τ = 0.836 → direct+cov: Top-1 89.4%, τ = 0.887 (+5.3 pp, +0.051)
• calibrated-dr-cpo: Top-1 81.0%, τ = 0.819 → +cov: Top-1 87.5%, τ = 0.881 (+6.5 pp, +0.062)
• stacked-dr: Top-1 83.1%, τ = 0.837 → +cov: Top-1 84.7%, τ = 0.842 (+1.6 pp, +0.005)

Why it works: Judges often reward verbosity even when longer responses don't provide more value. The two-stage calibration learns a risk index $T = g(S, \text{{response\_length}})$ that captures this bias, then applies monotone calibration to the debiased index. This corrects systematic judge misalignment with oracle preferences.

Future work: Systematically discover additional bias-correcting covariates like prompt domain, difficulty, user context, or response structure.

7) Triply robust underperforms

tr-cpo-e underperforms here (Top-1 32.9%, RMSE^d 0.136). The added complexity of a third robustness layer (oracle label correction) can introduce bias without careful tuning, especially when the underlying overlap assumptions are violated.

Methodological Takeaways

On‑policy DM should be your default

direct+cov offers the best ranking accuracy with low compute.

OPE requires overlap (and/or DR with a solid critic)

Use IPS only when policies are very similar (e.g., small temperature/top‑p changes). Otherwise prefer DR with calibrated rewards and cross‑fitted critics—understanding it still trails DM here and costs more.

Calibration and covariates matter

Calibration is required for valid CIs and better point estimates—and it also improves ranking. Adding response_length as a covariate improved ranking accuracy by 5–10 percentage points across methods. Future work: incorporating prompt domain, difficulty, and user context.

Limitations & Next Steps

What GPT-5-as-Oracle Tells Us — and What Generalizes to Humans

External Validity: Generalizing to New User Populations

This experiment evaluates policies on Arena prompts. But what if you care about performance on a different user population—a new domain, geography, or user cohort? For example: your oracle labels come from expert raters (source distribution), but you need to estimate quality for general users (target distribution).

Doubly robust methods can handle this transport problem. Collect a small set of oracle labels from your source population and cheap judge scores from both source and target populations. The DR estimator requires two models: (1) an outcome model that predicts how oracle labels relate to judge scores (calibration), and (2) a density ratio model that reweights source data to match the target population.

The estimator is consistent if either model is correct. If your calibration function f(S,X) transports well (the relationship between judge scores and oracle outcomes is the same in both populations), you don't need perfect density ratio estimates. Conversely, if you have good density ratios but imperfect calibration, you still get valid estimates.

Practical use case: You have human preference data from a pilot study (source) but need to evaluate system performance for production users (target). Use DR with source oracle labels + target LLM-judge scores + a simple covariate-based density ratio model. We plan a follow-up Arena-style study demonstrating this design for domain transport.

Temporal Dynamics: When Preferences Drift Over Time

Judge scores reflect preferences at a point in time, but user preferences can drift. What users love in Week 1 may annoy them by Week 4—yet if your judge was trained on old data, it won't capture this shift. The result: your judge-to-oracle calibration breaks silently, and estimates become unreliable.

How CJE handles temporal drift: Use the transportability test shown earlier in this post. Periodically collect fresh oracle labels on new batches and test whether your existing calibration function $f$ still produces unbiased predictions: $\mathbb{E}[Y_{\text{new}} - f(S_{\text{new}}, X_{\text{new}})] = 0$. If residuals are systematically non-zero, your calibration has drifted. Re-calibrate with 200-500 fresh oracle labels from the current time period. The framework's modular design makes recalibration cheap—you don't need to retrain judges, just update the mapping from $S$ to $Y$ using recent outcomes.

Design pattern for production: Collect oracle labels continuously (even if sparse—5-10% coverage is enough). Run the transportability test monthly on new batches. If it fails (Bonferroni-corrected p < 0.05 indicates systematic bias), you have two options:

1. Re-calibrate: Update the mapping $f: S \to Y$ with fresh oracle labels (200-500 samples). This is fast and doesn't require retraining the judge. Works well when the judge's relative rankings are still good, but the mapping to oracle scale has shifted.
2. Refine the judge: Manually inspect examples with the largest residuals $|Y - f(S,X)|$. Look for systematic patterns (e.g., judge consistently underrates concise responses, or overrates verbose explanations). Update the judge prompt/template to fix these biases. This is more work but produces a better-aligned judge going forward.

For high-stakes decisions, pair this with A/A tests: evaluate your base policy against itself; if CJE detects a spurious difference, your calibration needs updating.

• Human-generated oracle labels: This experiment uses GPT-5 as the oracle, not real human preferences or business metrics. The most critical validation is replicating these results with actual human labels(crowdsourced preferences, expert annotations) and production KPIs (conversion rates, user retention, revenue, task completion). Model-based judges and oracles may share systematic biases that don't transfer to human judgments. Human preferences are noisier, more context-dependent, and exhibit different calibration relationships than LLM scores.
• Scope: Results reflect this prompt distribution and these policies. OPE stability depends on overlap; DM results are on‑policy to this eval set. Generalization to other domains (code generation, summarization, creative writing) requires domain-specific validation.
• SNIPS failure modes: Quantify minimum viable overlap (ESS targets; tail indices) for OPE viability. When is teacher forcing accurate enough for reliable propensity estimation?
• Oracle budget: How low can oracle coverage go while preserving calibration quality and ranking accuracy? Can we push below 5% with more sophisticated calibration methods or active learning for label allocation?
• Covariates: Systematic discovery beyond response_length (prompt domain, difficulty, user context, response structure). Can we automatically identify bias-correcting covariates from data?

Try It Yourself: Working with the Data

To build intuition for how this works, let's look at the actual data structure and run a simple analysis. The CJE repository includes a 1k-sample slice of Arena data in examples/arena_sample/ for experimentation.

What the data looks like

Each logged sample contains the prompt, response, judge score, log probabilities, and (for a subset) oracle labels:

{
  "prompt": "What is the difference between OpenCL and CUDA?",
  "response": "OpenCL is an open standard maintained by Khronos...",
  "base_policy_logprob": -14.73,          // Log prob under logging policy
  "target_policy_logprobs": {             // Log probs for target policies
    "clone": -14.73,
    "parallel_universe_prompt": -18.25,
    "unhelpful": -42.12
  },
  "judge_score": 0.85,                    // Cheap judge score (0-1)
  "oracle_label": 0.86,                   // Ground truth (25% of samples)
  "metadata": {"prompt_id": "arena_1574"}
}

Fresh draws for each target policy include new responses generated at evaluation time:

{
  "prompt_id": "arena_0",
  "prompt": "What is the difference between OpenCL and CUDA?",
  "response": "OpenCL (Open Computing Language) and CUDA...",
  "judge_score": 0.85,
  "oracle_label": 0.86,    // 25-50% coverage for calibration
  "draw_idx": 0
}

Running the analysis

The experiments in this post use CJE's estimators with the data structure above. Here's how to run a similar analysis on the demo data:

from cje import analyze_dataset
from pathlib import Path

# Path to demo data (1k Arena samples)
data_dir = Path("examples/arena_sample")

# Direct Mode: use only fresh draws with oracle labels for calibration
results = analyze_dataset(
    fresh_draws_dir=str(data_dir / "fresh_draws"),
    include_response_length=True,  # Add response_length covariate
    verbose=True
)                                   # Auto-selects Direct Mode

# Results include estimates, standard errors, and confidence intervals
print(f"Policy estimates: {results.estimates}")
print(f"Standard errors: {results.standard_errors}")
print(f"95% CIs: {results.confidence_interval(alpha=0.05)}")

# DR Mode: combine logged data (IPS) with fresh draws (DM)
results_dr = analyze_dataset(
    logged_data_path=str(data_dir / "logged_data.jsonl"),
    fresh_draws_dir=str(data_dir / "fresh_draws"),
    verbose=True
)                                   # Auto-selects Doubly Robust

The estimator is automatically chosen based on the data you provide. You can override this by passing estimator="..." to select a specific variant (e.g., "stacked-dr" vs "calibrated-dr-cpo"). The function handles calibration, uncertainty quantification, and returns structured results with policy rankings and statistical tests. See the CJE repository's examples/ directory for complete worked examples including visualization code.

Citation

If you use this work, please cite:

@misc{landesberg2025arena,
  author = {Landesberg, Eddie},
  title = {Arena Experiment: Benchmarking 14 Estimators on 5k LLM Evaluations},
  year = {2025},
  month = {October},
  url = {https://cimolabs.com/blog/arena-experiment},
  note = {CIMO Labs Technical Report}
}

References

Reproducing the Results

In the meantime, you can explore the CJE library and run your own experiments:

Cross-Fitting Implementation

DR estimators use cross-fitting to ensure valid inference. Here's how the fold plumbing works:

from cje.data.folds import get_folds_for_dataset
from cje.calibration import calibrate_dataset

fold_ids = get_folds_for_dataset(dataset, n_folds=5, seed=42)  # prompt_id-hash stable
calibrated, calres = calibrate_dataset(
    dataset,
    enable_cross_fit=True,   # default True
    n_folds=5,
    covariate_names=['response_length']
)
# DR estimators consume the OOF predictions tied to these folds.

The get_folds_for_dataset function assigns each prompt to exactly one fold using a stable hash, ensuring no leakage across fold boundaries. The calibrator trains on K−1 folds and predicts on the held-out fold, producing out-of-fold (OOF) predictions that break the dependence between nuisance estimates and the data used for inference.

@misc{landesberg2025arena,
  author = {Eddie Landesberg},
  title = {Arena Experiment: Benchmarking 14 Estimators on 5k LLM Evaluations},
  howpublished = {\url{https://cimolabs.com/blog/arena-experiment}},
  year = {2025},
  note = {CIMO Labs Blog}
}