AI Quality as Surrogacy for Idealized Deliberation: Technical Appendix

0. Notation and spaces

• Context space $\mathcal{X}$, action space $\mathcal{A}$ (text, code, plans), score space $\mathcal{S} \subset \mathbb{R}^d$.
• A policy $\pi$ maps $x \in \mathcal{X}$ to a distribution $\pi(\cdot \mid x)$ on $\mathcal{A}$. Let $\Pi$ be a class of admissible policies.
• $X \sim P_X$ denotes the population distribution of contexts; we treat single-turn first and extend to trajectories in §10.
• An Operational Oracle $Y: \mathcal{X} \times \mathcal{A} \to [0,1]$ is the measurable, expensive evaluation label we can collect (e.g., human preference, GPT-5 judgment, expert audit).
• An Idealized Deliberation Oracle (IDO) is a functional $Y^*: \mathcal{X} \times \mathcal{A} \to [0,1]$ representing the normalized evaluation under idealized deliberation. See the utility semantics box below for a precise definition.
• A judge (or surrogate measurement process) $J$ maps $(x,a)$ to a random score $S \in \mathcal{S}$. We allow a ladder of rungs

induced by a filtration $\mathcal{F}_0 \subset \cdots \subset \mathcal{F}_K$ with $S^{(k)}$ measurable w.r.t. $\mathcal{F}_k$.

1. Axioms for the IDO (normative)

Let $Y^*(x,a)$ be the limiting value of a deliberation procedure.

• A1 (Deliberative stability). There exists a sequence of increasing-effort labels $Y^{(k)}(x,a)$ s.t. $Y^{(k)}(x,a) \to Y^*(x,a)$ in $L^2$ as $k \to \infty$.
• A2 (Evidence monotonicity). If $\mathcal{F}_k \subseteq \mathcal{F}_{k'}$, then
  $$\mathbb{E}[(Y^* - \mathbb{E}[Y^* \mid \mathcal{F}_{k'}])^2] \le \mathbb{E}[(Y^* - \mathbb{E}[Y^* \mid \mathcal{F}_k])^2]$$
• A3 (Instrumental invariance). If two procedures yield the same world-state relevant to the objective, they have equal $Y^*$.

A1–A3 make $Y^*$ a well-defined limit of a "deliberation ladder."

1.1. The Bridge Assumption: Connecting Y to Y*

In practice, we cannot directly measure the idealized oracle $Y^*$. Instead, we collect operational oracle labels $Y$—expensive but measurable evaluations such as human preferences, expert audits, or high-quality model judgments (e.g., GPT-5).

The Bridge Assumption (A0) formalizes the alignment between the operational oracle and the idealized target:

Validation: A0 is validated via the Bridge Validation Protocol (BVP), which consists of three pillars:

• Pillar 1: Predictive Transportability Experiment (PTE) — Empirical test that $Y$ predicts held-out outcomes on $Y^*$-relevant metrics (e.g., user satisfaction, task success).
• Pillar 2: Construct Validity Audits — Expert review and stakeholder feedback confirming that $Y$ captures the intended welfare construct.
• Pillar 3: Stability Monitoring — Continuous tracking of the Y→Y* relationship to detect drift (see CLOVER governance framework in §6).

Implication for Calibration: The statistical calibration machinery (Assumptions S1-S2 below, Sections 3-5) operates on the measurable label Y. The Bridge Assumption (A0) ensures that optimizing policy value $V(\pi) = \mathbb{E}[Y(A_\pi)]$ serves the idealized target $Y^*$. This separation keeps the statistical framework (Layers 2-6) operating on observables while making the alignment to $Y^*$ a governance question (Layer 0).

Note. If $(U,F,N,W)$ change across environments, selection enters $Y^*$ and the calibration $f_k$ will not transport (§3.5 table, row "$S\to Y^*$"). Record $(U,F,N,W)$ in the assumptions ledger (§12).

2. Surrogacy (structural) and transport (stability) assumptions

• S1 (Oracle-surrogate sufficiency at rung $k$). There exists a measurable $f_k: \mathcal{S} \times \mathcal{X} \to [0,1]$ s.t.
  $$\mathbb{E}[Y \mid X, A, S^{(k)}] = f_k(S^{(k)}, X) \qquad \text{a.s.}$$
  on $\mathrm{supp}(\pi_0 \cup \Pi_{\mathrm{eval}})$ (the joint support of logging and evaluated policies). Optionally add monotonicity in a one-dimensional risk index $T = g_k(S^{(k)}, X)$.

  Scope: S1 is required only on the overlap region, not for arbitrary actions outside $\mathrm{supp}(\pi_0 \cup \Pi_{\mathrm{eval}})$. This is the same support condition needed for standard overlap (S3).

  Note: S1 targets the operational oracle $Y$, not the idealized $Y^*$. Under the Bridge Assumption (A0), calibrating to $Y$ serves $Y^*$.

• S2 (Transportability across policies/time). For a collection $\mathcal{G}$ of environments (policies, cohorts, time), the same $f_k$ works: for all $g \in \mathcal{G}$,
  $$\mathbb{E}_g[Y \mid X, A, S^{(k)}] = f_k(S^{(k)}, X)$$
  on $\mathrm{supp}(\pi_0 \cup \Pi_{\mathrm{eval}})$ in each environment.Graphical test (Pearl & Bareinboim, 2014^[7]): In a selection diagram modeling environment differences via selection nodes, S2 holds if $S^{(k)}$ is S-admissible: $Y \perp \!\!\! \perp \mathrm{Sel} \mid X, A, S^{(k)}$ in the diagram with incoming arrows to $A$ removed, where $\mathrm{Sel}$ represents selection nodes. Intuitively: calibration transports if no selection node points into $Y$ given the surrogate. See §2.5 for the ladder of surrogacy regimes.
• S3 (Positivity/overlap for off-policy re-use). If estimating $V(\pi)$ from logs of $\pi_0$, then $\pi(a \mid x) > 0 \Rightarrow \pi_0(a \mid x) > 0$ a.s.
• S4 (Judge availability). For any $\pi$ used in Direct mode, we can obtain $S^{(k)}(X, A_\pi(X))$ at scale; for OPE/DR, we have $S^{(k)}(X, A_{\pi_0}(X))$ in logs.
• L1 (Oracle MAR). Let $L \in \{0,1\}$ indicate whether an example received an oracle label $Y$. Then $L \perp Y \mid (X, A, S^{(k)})$. Oracle labeling is ignorable conditional on observed surrogates and covariates.
• L2 (Oracle positivity). $P(L=1 \mid X, A, S^{(k)}) > 0$ on the support where $f_k$ will be applied. Ensures calibration function is identifiable and transportable.

2.5. A Ladder of Surrogacy Regimes

We organize identification and estimation strategies into three regimes, from weakest to strongest. Throughout, all assumptions are required only on the relevant support $\mathrm{supp}(\pi_0 \cup \Pi_{\mathrm{eval}})$ (the states/actions seen under logging and candidate policies). This is the same support condition required for standard overlap and does not strengthen any results.

2.6. The Causal Requirement: Mediation vs. Correlation under Optimization

The surrogacy regimes (1-3) address evaluation—estimating $V(\pi)$ for a fixed policy $\pi$. They rely on the Prentice criteria (Prentice, 1989), which define surrogacy based on statistical sufficiency: $Y^* \perp \pi \mid S$. This ensures the surrogate $S$ is predictive of the outcome $Y^*$, enabling unbiased estimation.

However, Prentice sufficiency is insufficient for optimization (e.g., RLHF, Best-of-N sampling), where the surrogate $S$ becomes the target and the policy $\pi$ is actively modified to maximize it.

2.6.1. Regime 4: Optimization

We now formally introduce Regime 4: Optimization, where the surrogate is no longer a passive measurement instrument but an active control signal for policy improvement.

Requirement: Causal Mediation. For this optimization to be safe (i.e., for increases in $S$ to reliably correspond to increases in $Y^*$), the surrogate must satisfy Causal Mediation (Frangakis & Rubin, 2002). Formally, this requires that the causal effect of $\pi$ on $Y^*$ flows through $S$:

This is a stronger condition than Prentice sufficiency ($Y^* \perp \pi \mid S$). Causal mediation requires blocking side channels—alternative causal paths from $\pi$ to $S$ that do not pass through $Y^*$ (e.g., $\pi \to \text{Length} \to S$, $\pi \to \text{Sycophancy} \to S$).

Failure Mode: Dissociative Effects. When causal mediation is violated, optimization exploits Dissociative Effects (F&R terminology)—changes to $S$ that are not mediated by $Y^*$. This is precisely the mechanism underlying the Surrogate Paradox and reward hacking.

Metrics for Optimization Robustness. To quantify the safety of optimization in Regime 4, CIMO will introduce two new metrics (formalized in an upcoming technical post):

• Goodhart Point (GHP): The level of optimization pressure (e.g., KL divergence, Best-of-N sample count) at which the gold reward $Y^*$ peaks and begins to crash. A higher GHP indicates greater optimization robustness.
• Optimization Gap (OG): The divergence $\mathbb{E}[S_{\text{optimized}}] - \mathbb{E}[Y^*_{\text{optimized}}]$ under optimization pressure. A smaller gap indicates the surrogate remains aligned with welfare even when actively optimized.

These metrics extend CJE's static validation framework to dynamic stress testing, enabling practitioners to measure whether a judge remains valid when used as an optimization target.

Topology Enforcement in Practice. In the CIMO stack, Y*-Alignment and the Standard Deliberation Protocol (SDP) are the mechanisms that enforce this causal topology. By requiring the judge ($S$) to evaluate the process of welfare generation (via SDP), we block the "side channels" (e.g., length, tone) that allow the model to increase $S$ without increasing $Y^*$.

For a detailed explanation of how SDP strengthens mediation through side-channel cost elevation, see The Surrogate Paradox.

3. Identification

Let $R^{(k)} = f_k(S^{(k)}, X)$ be the calibrated reward on the IDO scale.

3.5. Transport formulas (cross-environment evaluation)

When evaluating $\pi$ in a target environment that differs from the calibration source, Pearl & Bareinboim's transport framework ^[7] tells us exactly which target quantities to measure. Below are the three common deployment scenarios:

4. Estimators

Let $\mathcal{I}_{\text{oracle}} \subset \{1, \ldots, n\}$ index examples with expensive IDO labels $Y^*$ (at a top rung one can afford); others have only $S^{(k)}$.

4.1. Calibrator

Estimate $f_k$ on $\mathcal{I}_{\text{oracle}}$ by:

• Monotone (isotonic): $f_k(s,x) \equiv f_k(s)$ nondecreasing in $s$ and mean-preserving on the oracle slice.
• Two-stage: Fit $T = g_k(S^{(k)}, X)$ (e.g., spline in $(S^{(k)}, \text{length})$), then isotonic $T \mapsto Y^*$ with mean preservation.

Note on mean preservation: Mean preservation holds on the calibration slice; after transport to new domains/policies, the mean can differ unless S2 (transport) and L1–L2 (oracle MAR/positivity) hold. Use the transport test (§6) to validate.

Use K-fold cross-fitting: train $f_k$ on folds $\neq j$, predict on fold $j$, to obtain out-of-fold $\widehat{R}^{(k)}$.

4.2. Direct (fresh draws)

With $m$ prompts scored under $\pi$,

4.3. IPS (logs only)

4.4. DR (logs + critic ± fresh draws)

Fit $Q_\eta(x,a) \approx \mathbb{E}[\widehat{R}^{(k)} \mid x,a]$ via cross-fitting. If fresh draws from $\pi$ are available, approximate $Q_\eta^\pi(x)$ by Monte Carlo. Then

4.5. Weight stabilization (optional, off-policy)

Project raw weights to a mean-one, score-indexed monotone cone (SIM-style calibration) to boost ESS. This is a bias–variance tradeoff: stabilized weights $\tilde{w}_{\pi,i}$ can introduce small bias unless they converge to the true importance ratio. Use weight stabilization inside DR estimators (where outcome models guard against modest weight misspecification), and report diagnostics (ESS, tails).

4.6. Regime 1: Kallus–Mao estimator (no S1)

If diagnostics suggest surrogacy is unreliable, estimate effects on $Y$ directly using a doubly-robust Kallus–Mao estimator that treats $S$ as auxiliary information (no sufficiency assumed). You'll need a MAR-sampled set of $Y$ labels in the evaluation context; cross-fit nuisances $(e, r, \tilde{\mu}, \mu)$; then compute:

Report IF-based SEs with cross-fitting. ^[1]

5. Influence functions and inference

Assume pathwise differentiability and regularity (bounded moments, entropy conditions satisfied via cross-fitting).

5.1. Efficient influence function (EIF) for V(π)

Under S1 and known $f_k$,

With DR structure and nuisances $\eta = (Q_\eta, w_\pi)$,

which is Neyman-orthogonal to first-order perturbations of $(Q_\eta, w_\pi)$ holding $f_k$ fixed. Uncertainty from learning $f_k$ on the oracle slice is added separately via OUA (§5.3). If desired, one can treat $f_k$ as a nuisance and cross-fit it jointly to achieve formal orthogonality. We separate it and account for uncertainty via OUA for transparency and modularity.

5.2. Asymptotics and SEs

With K-fold cross-fitting,

Estimate variance with the empirical variance of $\phi_\pi$ (cluster-robust if needed).

5.3. Oracle-uncertainty aware (OUA) variance

If $f_k$ is learned from a finite oracle slice, add delete-one-fold jackknife over oracle folds:

Total variance: $\widehat{\text{Var}}_{\text{main}} + \widehat{\text{Var}}_{\text{OUA}}$. Use Satterthwaite df for small-sample t-intervals if desired.

5.4. Relationship to Conformal Prediction

Conformal Prediction (CP) (Vovk et al., 2005; Angelopoulos & Bates, 2021) provides distribution-free, finite-sample coverage guarantees for prediction intervals (uncertainty about a future observation $Y^*_{\text{new}}$). OUA addresses a different problem:inference on a population parameter (the policy value $V(\pi)$).

When to use each: Use CP when you need coverage for individual predictions (e.g., "What is the range of $Y^*$ for this specific user?"). Use OUA when you need inference on aggregate quantities (e.g., "What is the expected policy value across all users, with honest uncertainty?").

6. Testable diagnostics (falsifiable implications)

• Transport test (policy/time). Per-group residual mean test:
  $$H_0: \mathbb{E}[Y^* - f_k(S^{(k)}, X) \mid G=g] = 0 \quad \forall g \in \mathcal{G}$$
  where $G$ indexes groups (policies, time periods, domains). Use labeled subset; apply multiple-testing correction (e.g., Bonferroni). This is a weaker, testable implication of S-admissibility—if you lack labels in multiple domains, you can only partially test S2.
• Coverage of surrogate support. Compare histograms of $S^{(k)}$ on oracle-labeled vs. full sets; flag extrapolation if tails are unlabeled.
• Overlap diagnostics (off-policy). Effective sample size $\text{ESS} = (\sum w)^2 / \sum w^2$, weight CV, max/median ratio, Hill tail index.
• OUA share. Report $\widehat{\text{Var}}_{\text{OUA}} / (\widehat{\text{Var}}_{\text{main}} + \widehat{\text{Var}}_{\text{OUA}})$to guide budget (more labels vs. more prompts).
• Prentice test (surrogacy sufficiency / S1). On oracle-labeled subsets, regress $Y^*$ on $(X, A, S^{(k)})$ and test whether adding $A$ (and $A \times S^{(k)}$) improves fit. Failing to reject supports S1 (surrogacy sufficiency). For S-admissibility (S2, cross-domain), use a domain indicator $G$ and test $Y^* \perp \!\!\! \perp G \mid X, A, S^{(k)}$ on pooled labeled data across domains: does $G$ (and $G \times S^{(k)}$) improve prediction? If yes, $f_k$ does not transport—recalibrate or use K-M estimator (§4.6).

7. Learning with the IDO objective

For parametric $\pi_\theta$, the policy learning problem is

A plug-in gradient follows from the policy gradient identity with calibrated rewards:

optionally replacing $R^{(k)}$ by an advantage $R^{(k)} - b(X)$. This "RL with calibrated reward" aligns training with IDO.

For safe deployment, maximize a lower confidence bound $V(\pi_\theta) - z_{1-\alpha} \cdot \text{SE}(\widehat{V}(\pi_\theta))$.

8. Multiple stakeholders and social choice

Let $\mathcal{U} = \{1, \ldots, m\}$ index stakeholders with oracles $Y^*_u(x,a) \in [0,1]$. A social aggregator $W: [0,1]^m \to [0,1]$ defines

Common choices: weighted utilitarian ($W(y) = \sum_u w_u y_u$), max-min ($W(y) = \min_u y_u$), or constrained variants. Surrogacy extends with $f_{k,u}(S^{(k)}, X) \approx \mathbb{E}[Y^*_u \mid X, A, S^{(k)}]$; calibrate each and plug into $W$.

9. The deliberation ladder as information order

Model rungs by a filtration $\mathcal{F}_0 \subset \cdots \subset \mathcal{F}_K \subseteq \mathcal{F}_\infty$. Define

Then by Blackwell/Doob ordering, $k' \ge k$ implies $\mathbb{E}[(Y^* - Y^{(k')})^2] \le \mathbb{E}[(Y^* - Y^{(k)})^2]$. If $S^{(k)}$ is Blackwell more informative than $S^{(k-1)}$, a calibrated estimator at rung $k$ is (weakly) more efficient than at rung $k-1$.

10. Extension to trajectories (agents)

Let a trajectory $\tau = (s_0, a_0, \ldots, s_T)$ with policy $\pi$ and environment $P$. Define an IDO trajectory value

Surrogates may be terminal ($S_T^{(k)}$) or stepwise ($S_t^{(k)}$). Direct/IPS/DR estimators extend with clustering by trajectory; sequential IPS is typically ill-conditioned, so prefer Direct or DR with trajectory-level critics.

11. Limits (scope conditions)

• Non-regular targets. If $Y^*$ or $W$ induces non-differentiable functionals (e.g., maxima, boundary problems), first-order theory fails; use selective/subsampling or shape-constrained methods.
• Severe non-transport. If S2 fails (e.g., adversarial policy styles), drop to Regime 2 or 1 (§2.5): recalibrate $f_k$ locally per environment, or use K&M estimation with new oracle labels.
• Overlap failures. If S3 fails, IPS/DR is unreliable even with stabilized weights; collect fresh draws and use Direct.

12. Minimal "assumptions ledger" (for every deployment)

13. What you report (template)

For each $\pi$:

• $\widehat{V}(\pi)$ on the IDO scale with 95% CI (main + OUA), and DF rule.
• Diagnostics: transport test p-values, ESS (if OPE/DR), OUA share, oracle coverage plots.
• If choosing a policy: a decision with one-sided CI (safety margin).

This turns "AI should do what you'd do with unlimited time" into a measurable target, with estimators, CIs, and failure tests you can run.

@misc{landesberg2025surrogacy,
  author = {Landesberg, Eddie},
  title = {AI Quality and Surrogacy: Technical Appendix},
  year = {2025},
  month = {November},
  url = {https://cimolabs.com/research/ai-quality-surrogacy-technical},
  note = {CIMO Labs Technical Report}
}

References

References