Research

Methods and results for evaluating LLM systems with calibrated judges

Reading order: Start with the CJE paper and the validated work below. Everything else is grouped by status: work in progress, then essays and positions.

Preprint • December 2025

Causal Judge Evaluation: Calibrated Surrogate Metrics for LLM Systems

Reward calibration, transport audits, and calibration-aware inference. On 4,961 Arena prompts, CJE makes Direct + two-stage calibration the default. With a stronger model's ratings standing in for human labels, the paper's cost model gives a 14× reduction relative to full labeling (arXiv v3).

Landesberg, E., and Narayan, M.•arXiv:2512.11150

Erratum: after a fix to the calibration-aware variance, a re-run of the Arena benchmark gives 93.9% confidence-interval coverage for Direct and 95.6% for Direct with covariates, higher than the coverage reported in arXiv v3. Re-run details

Validated

Empirical results traceable to reproducible artifacts.

Drift Correction

Post-Audit Drift Correction: Offset vs EIF Correction vs Refit

Empirical and theoretical treatment of what to do after a failed transport audit: global offset baseline, policy-specific first-moment correction, and refit escalation under structural drift.

12 min•Technical

Judge Calibration Audit

HealthBench Rubric Judgments Depend on Which Judge You Use

Two custom judges on the HealthBench Consensus meta-evaluation data differ by up to 73.2 percentage points on specific criteria. A one-seed 5% masked-label replay nearly recovers the declared aggregate endpoint, while criterion-level conclusions remain judge-sensitive.

20 min•Technical

In Progress

Real questions with incomplete formalizations.

Assumption Testing

Bridge Validation Protocol (BVP)

Testing the Bridge Assumption (A0): that operational welfare Y aligns with idealized target Y*. Introduces PTE-based validation and SDP-Gov governance framework.

35 min•Technical

Essays & Positions

Lenses and positions, not validated results.

Formal Framework

AI Quality as Surrogacy for Idealized Deliberation

Formal framework with precise definitions, identification results, influence functions, and asymptotic theory for treating AI quality measurement as a surrogate endpoint problem.

25 min•Technical