HealthBench Rubric Judgments Depend on Which Judge You Use
A custom regrade of HealthBench Consensus meta-evaluation records reveals large cross-judge disagreement by rubric criterion
TL;DR
- On the 29,500 response–criterion records both judges parsed, gpt-4o-mini and Claude Haiku 4.5 disagree by up to 73.2 percentage points on specific HealthBench Consensus criteria.
- Against our declared strict-majority endpoint, with ties coded not met, the raw-confidence gaps are 24.5 pp and 13.0 pp. Against mean physician agreement, they are 14.4 pp and 3.0 pp. Endpoint choice materially changes the Haiku story.
- In one deterministic masked-label replay, calibrating on 5% of records (1,454 aggregate oracle-labeled records) lands within 1.4–2.1 pp of the full-data endpoint. This is one seeded retrospective exercise, not prospective physician labeling or repeated-split validation.
Validated: results on this page trace to reproducible artifacts.
Why this matters
HealthBench (OpenAI, May 2025) is one of the strongest open benchmarks for medical AI conversation evaluation. It addresses limitations of saturated multiple-choice benchmarks (MedQA, USMLE) by using physician-authored rubrics and criterion-level scoring. It's been used by OpenAI for model launches, by startups like DR. INFO and Intelligent Internet for competitive claims, and by the UK AI Safety Institute's Inspect framework for standardized evaluation.
Scope matters. Standard HealthBench contains 5,000 benchmark conversations and 48,562 weighted, conversation-specific rubric criteria; OpenAI uses a Boolean GPT-4.1 grader and aggregates weighted per-example scores. The analysis below does not reproduce that standard score. It custom-regrades the narrower HealthBench Consensus meta-evaluation data, which were released to test grader agreement with physicians.
Most public HealthBench score reports we found use raw judge outputs, and many do not foreground the judge model next to the headline score. We did not find downstream score reports that calibrate benchmark means back to physician labels.
HealthBench also ships something rare: Consensus meta-evaluation data with multiple physician met/not-met grades for fixed candidate responses. The release contains 3,671 benchmark conversations, 14,592 candidate responses, 29,511 response–criterion records, and 60,896 individual physician grades. We accessed the official records through the inspect_evals implementation. This lets us directly audit a custom grader, and we haven't found a prior systematic cross-judge calibration audit of the released meta-eval data.
What we ran
- Custom regrade of all 29,511 meta-eval records with
gpt-4o-miniandclaude-haiku-4-5-20251001. GPT parsed 29,510 records, Haiku parsed 29,501, and paired cross-judge results use their 29,500-record intersection. - We modified the judge prompt to also request a 0–1 confidence score alongside the binary judgment (temperature 0.0). A prompt A/B check confirmed this does not materially change aggregate results (see Robustness section below).
- Oracle-coverage sweeps (100/50/25/10/5%) using CJE[1] (Causal Judge Evaluation), with deterministic subsampling (seed base 42, one seed)
- 2×2 ablation: judge score format (continuous vs binary) × endpoint format (strict binary majority, ties not met, vs continuous physician agreement)
- Prompt A/B confound check (confidence-augmented vs original binary-only prompt)
The key technique behind the coverage sweeps is CJE. It learns the relationship between judge scores and a declared oracle endpoint on a labeled sample, then applies that mapping to the remaining records. Here the public file is already fully labeled; the sweep retrospectively masks labels. At 5% coverage, 1,454 response–criterion records retain their aggregate strict-majority endpoint. Each endpoint is derived from 2–5 individual physician grades.
Finding 1: Cross-judge disagreement by consensus criterion is large
Same 29,500 commonly parsed response–criterion records, same declared endpoint, two different judges. The endpoint is a strict positive physician majority with ties coded not met. Here are the consensus criteria with the largest cross-judge divergence:
| Consensus criterion | N | Strict-majority Y | gpt-4o-mini | Haiku 4.5 | Cross-Judge Gap |
|---|---|---|---|---|---|
| hedging / no-uncertainty / seeks context | 910 | 86.5% | 0.4% | 73.6% | 73.2 pp |
| hedging / irreducible / seeks context | 792 | 84.8% | 0.9% | 65.8% | 64.9 pp |
| emergency / non-emergent / context seeking | 536 | 67.0% | 98.1% | 49.4% | 48.7 pp |
| complex / detailed / accuracy | 644 | 61.3% | 8.7% | 40.9% | 32.2 pp |
| complex / simple / accuracy | 652 | 77.3% | 33.7% | 61.8% | 28.1 pp |
| complex / detailed / appropriate | 644 | 64.1% | 81.8% | 55.8% | 26.0 pp |
On hedging/seeks-context criteria, gpt-4o-mini is nearly reversed relative to physicians (0.4% vs 86.5%) while Haiku is merely low (73.6%). On emergency referrals / non-emergent / context seeking, the judges disagree in opposite directions: gpt-4o-mini overestimates by +31 pp, Haiku underestimates by −18 pp.
Qualifying the aggregate-level documentation
The inspect_evals HealthBench README states: “Three judge models were tried on the meta eval (GPT-5-nano, GPT-4-mini, Claude Haiku 4.5) and no statistically significant difference was observed, so we kept gpt-4o-mini as the judge model despite it being outdated.” Our results are compatible with that aggregate statement, but they show that aggregate judge equivalence can still hide severe consensus-criterion divergence.
What HealthBench already validates, and what this adds
HealthBench already includes a trustworthiness check: it compares model-based grading to physician grading on consensus criteria and reports agreement metrics (including Macro F1) against physician baselines. That is an important validation step and we do not dispute it.
Our contribution is different. We analyze calibration and consensus-criterion behavior under two concrete custom judges, then run a masked-label replay for aggregate estimation. Agreement can look acceptable in aggregate while criterion-level errors remain large enough to change conclusions about model behavior.
In short: HealthBench asks whether grader labels broadly track physicians; this audit asks whether custom grader outputs and criterion conclusions are numerically stable enough for decision-making.
Finding 2: The confidence gap depends on the physician endpoint
| Metric | gpt-4o-mini | Haiku 4.5 |
|---|---|---|
| Mean confidence | 0.916 | 0.801 |
| Strict-majority endpoint (ties not met) | 67.1% | 67.1% |
| Gap vs strict-majority endpoint | −24.5 pp | −13.0 pp |
| Mean physician agreement | 77.2% | 77.2% |
| Gap vs mean physician agreement | −14.4 pp | −3.0 pp |
| Binary accuracy | 68.3% | 71.9% |
| Binary F1 (met class) | 0.773 | 0.792 |
Against the declared strict-majority, tie-negative endpoint, both raw confidence means are high. But 6,025 of 29,511 records (20.4%) are tied. Preserving physician disagreement as a continuous fraction raises the endpoint from 0.671 to 0.772, reducing the GPT gap to 14.4 pp and the Haiku gap to 3.0 pp. The GPT result remains substantial; the Haiku characterization is endpoint-sensitive.
If you only look at binary accuracy and F1, the judges look comparable (~68–72% accuracy, ~0.77–0.79 F1). The calibration shift, visible only through the continuous confidence scores, reveals how different the underlying behavior is.
Finding 3: One seeded 5% replay nearly recovers the full endpoint
With the strict-majority endpoint fixed, full-data CJE calibration[1] produces the same aggregate endpoint (~0.671) for both judges. The reduced-label rows below come from one deterministic masked-label split:
| Oracle Coverage | gpt-4o-mini | Haiku 4.5 | ||
|---|---|---|---|---|
| Estimate | 95% CI | Estimate | 95% CI | |
| 100% | 0.6711 | [0.666, 0.678] | 0.6712 | [0.666, 0.677] |
| 10% | 0.6828 | [0.667, 0.700] | 0.6770 | [0.661, 0.695] |
| 5% | 0.6918 | [0.669, 0.716] | 0.6852 | [0.662, 0.712] |
At 5% oracle coverage (1,454 aggregate oracle-labeled response–criterion records), the estimates are within 2.1 pp and 1.4 pp of the full-data endpoint in this deterministic sample. Each retained endpoint aggregates 2–5 physician grades.
What this replay shows
A small labeled-record slice worked for aggregate recovery in this one seeded retrospective replay. Because the public meta-eval file is already fully labeled, this is a mechanics demonstration—not new physician annotation, repeated-split validation, or a prospective cost study. A real label budget must specify whether it prices individual grades, adjudication, or a completed aggregate endpoint.
Finding 4: Oracle definition matters more than judge format
Judge format (continuous confidence vs binary met/not-met) makes no difference after calibration. But how you define the physician endpoint shifts the calibrated estimate by 10.1 pp—from 0.671 (strict binary majority, ties not met) to 0.772 (continuous agreement fraction). This is not a calibration error; it's a measurement decision about whether to preserve physician disagreement information.
Show full 2×2 ablation results
We ran a 2×2 ablation on both judges: judge score format (continuous confidence vs binary met/not-met) × oracle label format (binary physician majority vs continuous physician agreement rate).
| Condition | Judge Format | Oracle Format | gpt-4o-mini | Haiku |
|---|---|---|---|---|
| A | continuous | binary majority | 0.6711 | 0.6711 |
| B | binary | binary majority | 0.6711 | 0.6711 |
| C | continuous | continuous agreement | 0.7717 | 0.7718 |
| D | binary | continuous agreement | 0.7717 | 0.7718 |
Judge format doesn't matter. Continuous vs binary judge scores produce identical calibrated estimates (A = B, C = D). Even though Haiku uses more of the confidence range, it doesn't help.
Oracle definition shifts the estimand by +10.1 pp. Switching from binary physician majority to continuous agreement fraction changes what you're estimating: a score of 3/5 physicians agreeing counts as 0.6 instead of 1.0. Benchmark users should be explicit about which target they're using.
Robustness: Prompt A/B confound
We modified the judge prompt to request confidence scores (which the original HealthBench prompt does not). This changes ~5.1% of individual binary decisions, but aggregate quality metrics are essentially unchanged (accuracy delta: −0.0004, met-class F1 delta: −0.0041). The gap against the declared strict-majority endpoint is not an artifact of our prompt modification, though criterion-level sensitivity still exists.
What this means for benchmark users
Consensus-criterion outputs are not judge-invariant by default
On hedging and context-seeking criteria, the default judge's scores are functionally random or reversed relative to physicians. Without category-specific validation or calibration, a model that genuinely excels at appropriate hedging can be penalized by gpt-4o-mini and, to a lesser extent, by Haiku.
Aggregate scores hide the problem
Both judges achieve ~68–72% binary accuracy against the declared strict-majority endpoint because criterion-level overestimation and underestimation can cancel. The aggregate number can look acceptable while individual subscores differ sharply.
Aggregate judge equivalence can hide instability
Two judges can look similar on overall metrics while disagreeing by up to 73 pp on specific criterion families. Aggregate agreement does not guarantee judge-invariant category conclusions.
A small labeled-record slice worked in this replay
In one seeded masked-label split, 1,454 aggregate oracle-labeled records recover the full-data endpoint within 1.4–2.1 pp. That does not establish prospective annotation cost or repair unstable criterion-level subscores.
How you define the physician endpoint matters
Binary physician majority vs continuous agreement rate shifts the calibrated estimate by 10.1 pp. Benchmark users should be explicit about which oracle they're targeting.
Limitations
- We audited two judges. A broader sweep (GPT-4.1, GPT-5, Sonnet 4.6) would strengthen the cross-judge findings.
- Consensus-criterion CJE calibration (per-criterion sweeps) is not yet implemented. We calibrate at the aggregate level only.
- Physicians disagree: 20.4% of records are tied. Our strict-majority, tie-negative endpoint is a declared measurement rule, not metaphysical ground truth.
- We used the Consensus meta-evaluation subset (29,511 response–criterion records), not standard HealthBench scoring. Full HealthBench contains 48,562 conversation-specific criteria.
- The reduced-label analysis uses one deterministic seed. It cannot support repeated-split coverage or robustness claims.
- CJE calibration corrects aggregate estimates but does not fix the underlying criterion-level judge failures. A calibrated aggregate still averages over unstable subscores.
Method
- Data: HealthBench Consensus meta-eval[2] accessed through inspect_evals: 29,511 response–criterion records with 2–5 physician met/not-met grades each. Primary Y is strict majority with ties coded not met.
- Judges:
gpt-4o-mini,claude-haiku-4-5-20251001. Temperature 0.0. Both used our confidence-augmented prompt; these outputs are not the official HealthBench GPT-4.1 grader. A prompt A/B check confirmed the modification flips ~5.1% of decisions but does not materially change aggregate metrics. - Parse success: gpt-4o-mini 29,510/29,511 (99.997%); Haiku 29,501/29,511 (99.97%). Failed records excluded.
- Calibration: CJE mean-preserving isotonic calibration with auto inference mode (which selected clustered bootstrap inference on this dataset),
n_bootstrap=500, and deterministic subsampling (seed base 42, one seed) for reduced-oracle analyses[1]. - Code: Analysis scripts and pre-cached judge outputs are in the reproducibility repo. A general-purpose judge calibration tool is in a merged PR to inspect_evals.
Reproducing this analysis
The analysis scripts, pre-cached judge outputs, and pre-computed results are in the reproducibility repo. You can verify the key numbers without API keys by running CJE calibration on the cached data. If your package index does not yet have the current CJE release, install it directly from source:
The exact commands and flags used for the numbers reported here are documented in the reproducibility repo. Given any set of judge scores and a small sample of oracle (human) labels, CJE produces calibrated estimates with valid confidence intervals. See the paper or the documentation for details.