CIMO LabsCIMO Labs

CJE in Action

You understand why your metrics lie. Now see how simple the fix is.

Run it yourself, no installation required.

Open in Colab →

CJE does three things. Each takes one code block.

1. Calibration

Your cheap judge scores (S) don't match expensive oracle outcomes (Y). CJE learns the S→Y mapping from a small labeled sample, then applies it everywhere.

from cje import analyze_dataset

# Your evaluation data - one list per policy variant
results = analyze_dataset(
    fresh_draws_data={
        "prompt_v1": [
            {"prompt_id": "1", "judge_score": 0.85, "oracle_label": 0.9},
            {"prompt_id": "2", "judge_score": 0.72, "oracle_label": 0.7},
            {"prompt_id": "3", "judge_score": 0.68},  # oracle_label optional (5-25% needed)
        ],
        "prompt_v2": [
            {"prompt_id": "1", "judge_score": 0.78, "oracle_label": 0.82},
            {"prompt_id": "2", "judge_score": 0.81, "oracle_label": 0.79},
            {"prompt_id": "3", "judge_score": 0.75},
        ],
    }
)

# Or load from files:
# results = analyze_dataset(fresh_draws_dir="responses/")
Two-stage calibration: Stage 1 learns g(S, response_length) using splines, Stage 2 applies isotonic regression to preserve monotonicity and correct scale. Shows how raw judge scores get mapped to calibrated predictions.

Two-stage calibration: learn flexible S→Y mapping, then ensure monotonicity.

That's it. CJE auto-detects which samples have oracle labels, fits isotonic regression for reward calibration, and returns calibrated estimates. Notice that oracle_label is optional. You only need it for 5-25% of samples.

2. Uncertainty Quantification

Raw judge scores give you a number with no error bars. CJE gives you honest confidence intervals that include both sampling variance and calibration uncertainty.

# Visualize with confidence intervals
results.plot_estimates(
    policy_labels={
        "prompt_v1": "Conversational tone",
        "prompt_v2": "Bullet points only",
    }
)

# Compare two policies with p-value
comparison = results.compare_policies(0, 1)
print(f"Difference: {comparison['difference']:.3f}, p={comparison['p_value']:.3f}")
Forest plot comparing four prompt variants with 95% confidence intervals. Conversational tone is marked BEST at 0.76. Bullet points only at 0.75, Concise at 0.72, Detailed at 0.70.

Compare prompt variants. Find the best one. Trust the error bars.

These CIs are valid when your data meets CJE's assumptions (monotone S→Y relationship, representative oracle sample). Check the assumptions →

3. Transportability Auditing

Calibration drifts. User behavior changes, models get updated, prompts evolve. CJE checks if your calibration still holds — but first you declare how much drift your decision can tolerate: a margin (delta_max) in the units of your outcome. Then run a small held-out probe weekly.

from cje.diagnostics import audit_transportability, plot_transport_comparison

# Declare the drift you can tolerate, in outcome units.
# Without delta_max the audit is descriptive only (NOT_GRADED).
DELTA_MAX = 0.05

# Weekly check: does calibration still hold within the margin?
# family_size=3: Bonferroni across the three audits read together
audits = {
    "Week 1": audit_transportability(results.calibrator, week1_data,
                                     delta_max=DELTA_MAX, family_size=3),
    "Week 2": audit_transportability(results.calibrator, week2_data,
                                     delta_max=DELTA_MAX, family_size=3),
    "Week 3": audit_transportability(results.calibrator, week3_data,
                                     delta_max=DELTA_MAX, family_size=3),
}

for name, audit in audits.items():
    print(audit.summary())  # PASS, FAIL, or INCONCLUSIVE vs the margin

# Visualize drift over time
plot_transport_comparison(audits, title="Weekly Calibration Check")
Transportability test over time. Week 1 and Week 2 pass (calibration error centered at zero). Week 3 fails (negative bias), indicating calibration has drifted and needs refresh.

PASS = the residual CI sits wholly inside your margin, trust the estimates.
FAIL = the CI sits wholly outside the margin, time to recalibrate.
INCONCLUSIVE = the CI straddles the margin boundary, the probe can't tell yet.

The five audit states

  • PASS — CI wholly inside ±delta_max. Needs ~20 effective probe clusters; small probes can't certify a pass.
  • FAIL — CI wholly outside the margin. Graded even from a small probe: a decisive interval is evidence of bias, not low power.
  • INCONCLUSIVE — CI overlaps the margin boundary, or too few clusters. Collect more probe labels.
  • NOT_GRADED — no delta_max declared. Descriptive only; can never pass or fail (the library warns you).
  • NOT_CHECKED — recorded by analyze_dataset for policies you gave no probe.

Wired into analyze_dataset(transport=TransportAuditConfig(...)), margins are in output units (the units of results.estimates), and only an observed FAIL hard-gates a policy's level claims. NOT_CHECKED and INCONCLUSIVE surface as visible limitations, not failures.

When calibration fails

Week 3 shows systematic bias beyond your margin. The judge overestimates quality. This could mean user expectations shifted, the model changed, or adversarial patterns emerged. CJE catches this before you ship decisions based on stale calibration.

Bonus: Debugging Failures

When calibration fails, you need to know why. CJE lets you inspect which samples the judge gets most wrong. Find the adversarial patterns, sycophantic responses, or edge cases fooling your evaluator.

Residual scatter plot showing most points clustered near zero with a few large negative outliers circled as samples to inspect. Large negative residuals indicate the judge overestimated quality.

Large negative residuals = judge overestimated quality. Inspect these first.

from cje.diagnostics import compute_residuals

# Find samples where judge overestimates quality (sorted by worst first)
samples = compute_residuals(results.calibrator, probe_data)

# Inspect the worst offenders
for s in samples[:3]:
    print(f"Residual: {s['residual']:.2f}")
    print(f"  Judge: {s['judge_score']:.2f} → Calibrated: {s['calibrated']:.2f}")
    print(f"  Oracle: {s['oracle_label']:.2f}")
    print(f"  Prompt: {s['prompt'][:80]}...")
    print(f"  Response: {s['response'][:80]}...")
    print()

Negative residuals mean the judge scored it high but the oracle scored it low. These are your failure modes: responses that look good but aren't. Fix these patterns, retrain, and your calibration improves.

That's It

Calibration

S → Y mapping

Uncertainty

Honest CIs

Transportability

Drift graded vs. your margin

Three things. Three code blocks. That's CJE.

Run Locally

pip install cje-eval
git clone https://github.com/cimo-labs/cje.git
cd cje/examples
jupyter notebook cje_core_demo.ipynb

Go Deeper

Benchmark Paper

Canonical empirical results on 5k Arena prompts

Assumptions

When does CJE work?

Data Format

Full docs on GitHub

Full Tutorial

Interactive Colab notebook

Try it now

Open in Colab →