Coverage-Limited Efficiency: Why High ESS Isn't Enough

We attempted to formalize this intuition with Coverage-Limited Efficiency (CLE): a proposed local lower bound on standard errors for logs-only off-policy estimators. The formal bound is withdrawn (see the withdrawal note below): as stated, it admits counterexamples. What survives is the concept and the diagnostic decomposition into two failure modes: (1) insufficient logger coverage in target-relevant regions, and (2) shape mismatch between logger and target distributions within those regions. When judged fresh draws from the target policy are available, the coverage problem stops binding: the Monte Carlo term from the target sample is achievable regardless of logger coverage.

The Coverage Gap

Suppose your logging policy π₀ generates responses to prompts, and you want to evaluate a new target policy π′ using only logged data. Standard off-policy evaluation (OPE) wisdom says: compute importance weights $w_i = \pi'(A_i|X_i) / \pi_0(A_i|X_i)$, check that ESS is high, stabilize the weights if needed, and you're good to go.

But ESS measures weight concentration: whether the estimate is dominated by a few outliers. It doesn't measure coverage: whether the logger visits the regions where the target policy typically operates.

Context vs. action coverage: Coverage can fail in contexts (P(X)) or in actions given context (P(A|X)). CLE here targets action-space overlap given a shared prompt set; if P(X) shifts, diagnose and correct that first (reweight contexts) before applying CLE.

The Geometry of Failure: Why β Vanishes

In low dimensions (like a 2D Gaussian), if two distributions overlap at their mode, they share a lot of mass. But LLM outputs live in high-dimensional sequence space. In high dimensions, probability mass doesn't sit at the mode. It sits in a thin "entropic shell" far from the center (a phenomenon known as the Concentration of Measure).

Think of it like soap bubbles: in high dimensions, probability mass concentrates on the surface of a sphere, not at the center. If your logger π₀ and target π′ have even slightly different entropies or stylistic tendencies, their "soap bubbles" (Typical Sets) can be completely disjoint, even if they theoretically share support over all possible sequences.

This is why β can vanish even when both policies are "similar" LLMs. A system prompt change, a temperature adjustment, or fine-tuning on different data can shift the typical set enough that overlap becomes negligible in practice, even though overlap is technically non-zero in the limit.

A Coverage-Limited Efficiency Bound (Withdrawn)

Coverage-Limited Efficiency was our attempt to formalize this intuition. Let $T \subset \mathcal{X} \times \mathcal{A}$ be any target-relevant region: a subset of context-action space where the target policy concentrates its probability mass. For example, $T$ could be the set of (context, response) pairs that are "typical" under π′ (defined via surprisal thresholds from teacher forcing).

Define:

• $\alpha = P_{\pi'}(T)$: Target policy's mass on $T$ ("How often does π′ visit this region?")
• $\beta = P_{\pi_0}(T)$: Logging policy's mass on $T$ ("How often does the logger visit this region?")
• $\sigma_T^2 = \text{Var}(Y \mid (X,A) \in T)$: Outcome variance within $T$
• $\chi^2(\pi'_T \| \pi_{0,T})$: Chi-square divergence between π′ and π₀ restricted to $T$ (measures shape mismatch inside the region)

Connection to Rényi divergence: Because D₂(p‖q) = log(1 + χ²(p‖q))^[3], our earlier factor e^D₂/2 equals √(1+χ²). This connects the mismatch factor directly to second-moment diagnostics of restricted weights.

Interpreting the components

The withdrawn floor combined three multiplicative factors. They remain useful as heuristic diagnostics of logs-only difficulty:

Key insight: The coverage penalty $\alpha/\sqrt{\beta}$ and shape mismatch $\sqrt{1 + \chi^2}$ are multiplicative. Small β or large χ² makes the heuristic floor prohibitively large — a strong signal that logs-only estimation will be uninformative regardless of how you calibrate the weights.

Practical diagnostic: Plot the distribution of normalized restricted weights ẇ_i on T; (1/|T|)∑ ẇ_i² = 1+χ̂_T². Heavy tails → large mismatch → inflated floor.

Two Regimes: Logs-Only vs. With Fresh Draws

The coverage problem afflicts logs-only estimators (pure IPS, calibrated IPS). When you have judged fresh draws from the target policy π′, the picture changes fundamentally.

Why fresh draws change everything

Suppose you collect $m$ judged fresh draws from π′: each target policy generates a response to the same prompts, you score it with a cheap judge, and calibrate those scores to the oracle scale. Now your estimator is a two-sample design: $n$ logged observations from π₀ and $m$ fresh observations from π′.

Implication for Direct Model (DM) and Doubly Robust (DR): These methods generate fresh responses from each policy on the same prompts. The binding uncertainty is the Monte Carlo term from scoring those fresh responses with a calibrated judge. Logged data from π₀ can serve as control variates (DR uses them for bias correction), but poor logger coverage doesn't prevent you from getting a precise estimate. It just means the log-based corrections contribute little. Influence-function stacking (stacked-DR) automatically down-weights log corrections when the CLE floor is high.

Target-Typicality Coverage (TTC) Diagnostic

To operationalize CLE, we need to compute β: what fraction of logged data lives in target-relevant regions? The key challenge: defining $T$ (the target-typical region) without already having extensive data from π′.

Defining typicality via surprisal

Use teacher forcing to compute the per-token surprisal of each logged response under the target policy π′. Responses with low surprisal are "typical" for π′; high surprisal means π′ would rarely generate them. Set a threshold τ (e.g., 75th percentile of surprisal on a small validation set of fresh draws from π′, or a fixed percentile on logged data) and define:

Alternative (no teacher forcing): risk-index typicality. Use the stage-1 index Ť = g(S, X) from reward calibration and define T as the top-k percentile of Ť under π′ (estimated via a small set of fresh draws). This yields a coverage diagnostic that does not depend on propensity scoring.

Then compute:

• $\hat{\alpha} = \frac{1}{n} \sum_{i=1}^n w_i \cdot \mathbb{1}[i \in T]$ (target mass on $T$, estimated via importance weights)
• $\hat{\beta} = \frac{1}{n} \sum_{i=1}^n \mathbb{1}[i \in T]$ (logger mass on $T$, directly observed)

Target-Typicality Coverage (TTC) is β̂. If TTC is low, the logger rarely visits target-typical regions: a red flag for logs-only estimation. Since α ≈ 0.8 by construction (T is defined to contain 80% of target mass), β is the diagnostic that actually varies between policies.

Computing the SE floor

Once you have α̂, β̂, and an estimate of the shape mismatch $\widehat{1+\chi^2_T}$ (computed from the restricted importance weights on $T$ as shown above), define the heuristic floor:

where $\hat{\sigma}_T$ is the out-of-fold (OOF) residual variance on $T$. With Bound 1 withdrawn, SE_min(τ) is a diagnostic gauge of logs-only difficulty, not a proven minimum; we use it as a decision heuristic below.

Coverage-mismatch profile

Since the definition of $T$ depends on the surprisal threshold τ, plot SE_min(τ) across a range of thresholds. If the floor exceeds your SE budget for all plausible τ, logs-only estimation is infeasible.

How to compute CLE in practice

1. Choose T: Define via surprisal or risk-index typicality; profile over τ
2. Compute β̂: β̂ = |T|/n
3. Estimate α̂: Small fresh draws + simple classifier density ratio, or importance weights if reliable
4. Estimate σ_T: Via out-of-fold residuals on T
5. Estimate 1+χ̂_T²: Via normalized restricted weights on T (shown above)
6. Compute SE_min(τ): Profile over τ; apply gates

Connection to Arena Experiment Results

In the Arena benchmark paper, we saw that SNIPS (raw importance sampling)^[1] and calibrated-ips (weight stabilization stabilized weights) both failed catastrophically for ranking, despite weight stabilization boosting ESS from 0.4–26% up to 82–99%. CLE provides our leading hypothesis for why this occurred:

• Poor logger coverage in target-typical regions. The base policy (Llama 3.3 70B with standard system prompt) generates very different responses than the target policies (parallel_universe, premium, unhelpful). Even though global ESS is high after weight stabilization, β̂ (logger coverage in target-typical regions) is tiny for policies that differ substantially from base.
• Teacher forcing noise amplifies shape mismatch. Computing propensities π′(A|X) via teacher forcing is noisy and non-deterministic^[2], inflating the local divergence D₂ even for the clone policy (identical to base with different seed).
• Why Direct Model (DM) succeeds. Direct methods generate fresh responses from each policy on the same prompts. The dominant uncertainty is the Monte Carlo term $\sqrt{\text{Var}_{\pi'}(R)/m}$, not logger coverage. Poor logger coverage is irrelevant: each policy gets its own independent sample.
• Why Doubly Robust (DR) recovers performance. DR uses fresh draws to train an outcome model, then applies importance-weighted corrections using logged data. The outcome model provides a baseline; logs are control variates. When the CLE floor is high (poor logger coverage), the log-based corrections contribute little, and the estimate is governed by the target Monte Carlo term. weight stabilization stabilization ensures the corrections don't hurt, even if they don't help much.

Practical Guidance

When to refuse logs-only estimation

1. Before running the experiment: If you know the target and logging policies differ substantially (different model sizes, system prompts, temperatures), expect poor coverage. Plan to collect judged fresh draws rather than relying on logs-only OPE.
2. After collecting data: Compute TTC (β̂) and the CLE floor SE_min. If SE_min exceeds your precision target (e.g., you need SE ≤ 0.01 but the floor is 0.05), refuse logs-only estimation.
3. Profile over typicality thresholds: Plot SE_min(τ) to check robustness. If the floor is prohibitive across all plausible definitions of "target-typical," the result is not sensitive to your choice of τ.

Sample size planning

As a planning heuristic, to achieve SE ≤ ε with logs-only data,

If β is tiny (poor logger coverage), the required $n$ explodes. In such cases, collecting fresh target draws is far more efficient than increasing the size of the logged dataset.

Cost guidance: With β small, required n scales like α²/(β ε²). A handful of judged fresh draws (m in the low thousands) often beats adding millions of logs. We surface this trade-off directly via the floor.

When logs still help

Even with judged fresh draws (Direct/DR), logged data can improve efficiency if the CLE floor is reasonable. Doubly robust estimators and influence-function stacking (stacked-DR) automatically weight the log-based corrections by their precision. When β is moderate (logger has decent coverage), DR can achieve lower variance than pure Direct estimation. When β is tiny, stacking mutes the log corrections and you recover essentially the Direct estimate.

Comparison to ESS

Key insight: You can have high ESS (no weight concentration) but low TTC (poor coverage), leading to a prohibitive CLE floor. Both diagnostics are necessary for honest off-policy evaluation.

Limitations

• Defining $T$ without fresh draws is heuristic. The surprisal-based typicality definition requires teacher forcing under π′, which may be noisy. Profile over thresholds and disclose sensitivity. The risk-index alternative avoids propensity scoring.
• Estimating σ_T with sparse oracle coverage. Use out-of-fold residuals and conservative bounds when oracle labels are limited.
• Context shift. If the distribution over contexts $X$ differs between logger and target, the analysis requires separate coverage checks over $X$. The CLE diagnostics assume overlap failures are in the action space $A | X$, not in $X$ itself.
• Locality: The heuristic floor is local in T. If you partition 𝒳×𝒜 into bins, the worst (lowest β/highest χ²) bin often dominates. This explains why global ESS can look fine while a single high-mass target bin drives the difficulty.

Planned Implementation in CJE

CLE diagnostics are planned for integration into the CJE package in a future release. The implementation will include:

• TTC (Target-Typicality Coverage): β̂, the fraction of logged data in target-typical regions (logger coverage)
• Mismatch multiplier: $\hat{M}_T = \exp(D_2)$, measuring shape divergence inside the region
• CLE floor: SE_min, the heuristic SE floor (the formal bound is withdrawn)
• IF-ESS restricted to $T$: Effective sample size of the influence function within the target-typical region

When implemented, if logs-only estimation is attempted and any REFUSE-LEVEL gate triggers, CJE will issue a warning and suggest collecting judged fresh draws.

If you're interested in contributing to the CLE implementation or have use cases that would benefit from these diagnostics, please open an issue or discussion on the CJE GitHub repository.

Conclusion

High ESS is necessary but not sufficient for informative off-policy evaluation. Coverage-Limited Efficiency was our attempt to make that precise. The formal lower bound is withdrawn, but the diagnostic decomposition — logger coverage versus shape mismatch — remains our working lens, and it makes refusal decisions concrete even without a proven floor.

Key takeaways:

• Logs-only OPE appears coverage-limited: precision degrades sharply when the logger rarely visits target-relevant regions (the proposed formal floor is withdrawn; correct formalization is open)
• With judged fresh draws (Direct/DR), the target Monte Carlo term is achievable regardless of logger coverage. Logs become optional control variates
• TTC (Target-Typicality Coverage) operationalizes the coverage diagnostic
• CJE's calibration and weight-stabilization methods reduce variance; they cannot manufacture coverage the logger never had

For practitioners: compute TTC and the CLE floor before committing to logs-only estimation. When the floor is prohibitive, invest in fresh target draws rather than scaling up logged data collection.

Appendix: Proofs

Notation & assumptions (for this appendix)

Let $Z=(X,A,Y)$ be drawn i.i.d. under the logging policy distribution $\pi_0$ with density$q$, and let the target policy distribution be $\pi'$ with density $p$ (both w.r.t. a common base measure). Define the importance ratio $w(Z)=\frac{p(Z)}{q(Z)}$ and a measurable$T\subset\mathcal{X}\times\mathcal{A}$. Write$\alpha = P_{\pi'}(T),\; \beta = P_{\pi_0}(T)$ and the restricted densities$p_T = p/\alpha,\; q_T = q/\beta$ on $T$. Let$\sigma_T^2$ denote a lower bound on conditional outcome variance on $T$, e.g. $\sigma_T^2 \le \operatorname*{ess\,inf}_{(x,a)\in T}\operatorname{Var}(Y\mid X{=}x,A{=}a)$ (any fixed lower bound suffices for the inequality below). We consider regular (asymptotically linear) estimators based only on logs (for the withdrawn Bound 1) and independent logs + target draws (for the withdrawn two‑sample bound). Note the σ_T² here (a lower bound on the conditional variance) is not the aggregate Var(Y | (X,A) ∈ T) used in the main text's original statement — this mismatch is one of the reasons Bound 1 was withdrawn.

Lemma A (weight identity on $T$)

On $T$, the importance ratio satisfies $w = \frac{p}{q} = \frac{\alpha}{\beta}\cdot \frac{p_T}{q_T}.$ Consequently,

Proof. On $T$, write $p=\alpha p_T,\; q=\beta q_T$ and expand. The last equality uses $\chi^2(p_T\|q_T)=\int_T \frac{(p_T-q_T)^2}{q_T}d\mu = \int_T \frac{p_T^2}{q_T}d\mu - 1.$ □

Bound 1 (Coverage‑Limited Efficiency, χ² form) — Withdrawn

An earlier version of this appendix claimed: for any regular logs‑only estimator of $\Psi=\mathbb{E}_{\pi'}[Y]$, the asymptotic standard error obeys

Why it was withdrawn.

• Definitional mismatch. The main text defined $\sigma_T^2 = \operatorname{Var}(Y \mid (X,A)\in T)$ — the aggregate variance over the region — but the proof required the much smaller quantity $\sigma_T^2 \le \operatorname*{ess\,inf}_{(x,a)\in T}\operatorname{Var}(Y\mid X{=}x,A{=}a)$. Under the main‑text definition the bound is false: take $Y$ deterministic given $(X,A)$ with a conditional mean that varies over $T$; the aggregate $\sigma_T^2$ is positive, yet a logs‑only estimator with an outcome model can beat the claimed floor.
• The proof did not establish the claim. Its efficiency-bound step conflated the mean of a fixed function with the off‑policy functional (with known propensities the semiparametric bound is the AIPW-form variance, which is strictly smaller than $\operatorname{Var}_{\pi_0}(wY\mathbf{1}_T)$), and its centering step used the false identity $\operatorname{Var}_{\pi_0}\!\big(w Y \mathbf{1}_T\big) = \mathbb{E}_{\pi_0}\!\big[w^2 (Y-\mu_T)^2 \mathbf{1}_T\big]$ (the cross terms do not vanish).
• Its two‑sample companion admits a counterexample (see the withdrawn proposition below), which removed our remaining confidence in the additive-floor reasoning used throughout.

Lemma A above is unaffected: it is a purely algebraic identity, and $\mathbb{E}_{\pi_0}[w^2\mathbf{1}_T] = \tfrac{\alpha^2}{\beta}(1+\chi^2(p_T\|q_T))$ remains the basis for the second‑moment diagnostics used on this page. Whether a correct coverage‑limited lower bound of this general shape can be proven is open.

Remark (ESS form on $T$)

Define normalized restricted weights $\tilde w_i = w_i / \mathbb{E}_{\pi_0}[w\mid i\in T]$ and the usual weight ESS on $T$:

Then $\mathbb{E}[\tilde w_i]=1$ and $\mathbb{E}[\tilde w_i^{\,2}] = 1+\chi^2(p_T\|q_T)$, so $\mathbb{E}[\text{ESS}_T] \approx \frac{|T|}{1+\chi^2(p_T\|q_T)}.$ Since $\mathbb{E}[|T|]=\beta n$, the heuristic floor SE_min from the main text can be rewritten as

Derivation. With $\tilde w$ normalized, the classical identity $\frac{1}{|T|}\sum_{i\in T} \tilde w_i^{\,2} \to 1+\chi^2(p_T\|q_T)$ implies $\text{ESS}_T \approx \frac{|T|}{1+\chi^2}$. Replace $\sqrt{1+\chi^2}$ in SE_min by $\sqrt{|T|/\text{ESS}_T}$ and use $|T|\approx \beta n$. The ESS_T identity stands on its own as a diagnostic; the floor interpretation inherits the withdrawal of Bound 1.

Proposition (Two‑sample variance lower bound) — Withdrawn

An earlier version of this appendix claimed: with $n$ logs from $\pi_0$ and $m$ independent judged fresh draws $U_1,\ldots,U_m \sim \pi'$ scored into an oracle‑scale reward $R$, any regular estimator using both samples satisfies

Why it was withdrawn. The claim is provably false. The fresh‑draws‑only sample mean $\tfrac{1}{m}\sum_{j=1}^m R_j$ is a regular estimator with access to both samples, and its variance is exactly $\operatorname{Var}_{\pi'}(R)/m = B_{\text{target}}$ — strictly below $B_{\text{target}} + B_{\text{logs}}$ whenever $B_{\text{logs}} > 0$. An additive floor would assert that adding an independent logged sample makes every estimator worse than ignoring it; information from independent samples can only help (variance floors combine harmonically, not additively). The proof sketch misused the orthogonal‑EIF decomposition: $\operatorname{Var}(\phi_{\text{target}}) + \operatorname{Var}(\phi_{\text{logs}})$ does not lower‑bound the two‑sample asymptotic variance in this additive form.

Remark. What is true is weaker: the target Monte Carlo variance $\operatorname{Var}_{\pi'}(R)/m$ is achievable by ignoring the logs, so fresh‑draw sample size governs the precision you can guarantee; logs can only improve on it (e.g., as control variates), and how much they help depends on coverage.

Lemma B (Rényi‑2 factor equals $\sqrt{1+\chi^2}$ )

By definition, $D_2(p_T\|q_T)=\log\!\int_T \frac{p_T^2}{q_T}\,d\mu=\log\!\big(1+\chi^2(p_T\|q_T)\big)$.^[3] Therefore $\exp\!\big(D_2/2\big)=\sqrt{1+\chi^2(p_T\|q_T)}$, and the $\exp(D_2/2)$ form of the mismatch factor (as in $\frac{\sigma_T\alpha}{\sqrt{\beta n}}\exp(D_2/2)$) is identical to the χ² form used here. The identity is unaffected by the withdrawal of Bound 1. □

References

References

Citation

If you use this work, please cite:

@misc{landesberg2025coverage,
  author = {Landesberg, Eddie},
  title = {Coverage-Limited Efficiency: Why High ESS Isn't Enough},
  year = {2025},
  month = {October},
  url = {https://cimolabs.com/research/coverage-limited-efficiency},
  note = {CIMO Labs Technical Report}
}