Prerequisites
- Python 3.10 or higher
- 1000+ examples with judge scores (for meaningful estimates)
- A small oracle-labeled slice (e.g., ~5% of samples) to calibrate the judge
Installation
# Install from PyPI
pip install cje-eval
For development or to run examples:
# Clone repository
git clone https://github.com/cimo-labs/cje.git
cd cje
# Install with Poetry (recommended)
poetry install
# Or with pip
pip install -e .
Quick Verification
Test your installation with a simple analysis:
# Python example
from cje import analyze_dataset
result = analyze_dataset(
fresh_draws_dir="path/to/responses/",
)
print(f"Estimate: {result.estimates[0]:.3f}")
Don't have data yet?
Check out the working example in examples/arena_sample/ or see CJE in Action for the data format.
CLI Usage
CJE includes a command-line interface:
# Analyze a directory of per-policy response files
python -m cje analyze responses/
# Specify estimator explicitly (calibrated-direct is the default)
python -m cje analyze responses/ --estimator calibrated-direct
# Save results to JSON
python -m cje analyze data.jsonl -o results.json
# Validate data format before running
python -m cje validate data.jsonl --verbose