Prerequisites
- Python 3.9 or higher
- 1000+ examples with judge scores (for meaningful estimates)
- Log probabilities from your models (for IPS/DR modes)
Installation
# Install from PyPI
pip install cje-eval
For development or to run examples:
# Clone repository
git clone https://github.com/cimo-labs/cje.git
cd cje
# Install with Poetry (recommended)
poetry install
# Or with pip
pip install -e .
Quick Verification
Test your installation with a simple analysis:
# Python example
from cje import analyze_dataset
result = analyze_dataset(
"path/to/your/data.jsonl",
estimator="auto"
)
print(f"Estimate: {result.estimates[0]:.3f}")
Don't have data yet?
Check out the working example in examples/arena_sample/ or read the data format guide to prepare your data.
CLI Usage
CJE includes a command-line interface:
# Analyze with automatic mode selection
python -m cje analyze data.jsonl
# Specify estimator explicitly
python -m cje analyze data.jsonl --estimator calibrated-ips
# Save results to JSON
python -m cje analyze data.jsonl -o results.json
# Validate data format before running
python -m cje validate data.jsonl --verbose