From A/B to Offline: Causal Evaluation for LLMs

Offline evaluation on LLM logs fails spectacularly. On 5,000 Arena conversations, standard methods achieve 0.6% effective sample size—essentially 30 usable samples. Here's how to apply causal inference to make offline evaluation actually work.

The core problem: distribution shift between policies. When you evaluate a new model or prompt on historical data generated by a different policy, naive averaging gives you correlation, not the counterfactual effect you need for decision-making.

The A/B Test Gold Standard

Let's start with what actually works: online A/B tests. When you randomly assign users to control or treatment, you break the correlation between assignment and outcomes. A simple difference in means gives you the causal effect.

But A/B tests are expensive. You need to risk real traffic. You wait weeks for results. Can't we just test on historical data?

Why Offline Evaluation Fails

When you evaluate a new policy (model, prompt, temperature) on logs from an old policy, you face a fundamental problem: the data was generated by different behavior.

weight = P(response | new_prompt) / P(response | old_prompt)

Imagine evaluating a creative writing prompt on logs from a factual Q&A prompt. The responses are fundamentally different—any evaluation is meaningless. But even subtle changes (temperature 0.7 → 0.8) cause enough distribution shift to break standard methods.

The CJE Solution: A/B Testing for Historical Data

Causal Judge Evaluation (CJE) makes offline evaluation work by solving three critical problems:

The CJE Pipeline in 5 Steps

1. Validate propensities

   Use teacher forcing to get exact log-probabilities for historical responses under both old and new policies. Run determinism checks—bad propensities mean no causal claims.

2. Calibrate the judge

   Map judge scores to your KPI scale (win rate, revenue, user satisfaction) using ~200 labeled examples. Check coverage—if the new policy generates responses outside your calibration range, you'll get warnings.

3. Stabilize weights

   Transform raw importance weights into stable, unit-mean weights that preserve monotonicity. This deterministically improves ESS while maintaining unbiasedness.

4. Estimate with doubly robust methods

   Use Calibrated-IPS for speed or Stacked-DR for accuracy. Add oracle uncertainty to confidence intervals. Cross-fit to ensure √n convergence.

5. Apply gates and decide

   Check OVERLAP (ESS ≥ 30%), JUDGE (calibration quality), IDENTIFICATION (coverage). Get a clear SHIP or REFUSE decision with honest confidence intervals.

Real Results on Arena Data

We tested CJE on 4,989 real conversations from Chatbot Arena, comparing 5 different policies:

The improvement isn't marginal—it's the difference between random noise and actionable insights. CJE turns 30 effective samples into 4,700+.

What You Need to Run CJE

# Validate your data
python -m cje validate logs.jsonl

# Run evaluation
python -m cje analyze logs.jsonl --estimator stacked-dr -o results.json

The Bottom Line

Stop lying to yourself with broken offline evaluations. Standard methods on LLM logs are worse than useless—they're actively misleading. CJE gives you the A/B test guarantees you need, using the historical data you already have.

Want to see it in action? Check out the complete CJE product tour with interactive estimator comparisons and visual diagnostics.