Distribution-informed Efficient Conformal Prediction for Full Ranking

Quantifying uncertainty is critical for the safe deployment of ranking models in real-world applications. Recent work offers a rigorous solution using conformal prediction in a full ranking scenario, which aims to construct prediction sets for the absolute ranks of test items based on the relative ranks of calibration items. However, relying on upper bounds of non-conformity scores renders the method overly conservative, resulting in substantially large prediction sets. To address this, we propose Distribution-informed Conformal Ranking (DCR), which produces efficient prediction sets by deriving the exact distribution of non-conformity scores. In particular, we find that the absolute ranks of calibration items follow Negative Hypergeometric distributions, conditional on their relative ranks. DCR thus uses the rank distribution to derive non-conformity score distribution and determine conformal thresholds. We provide theoretical guarantees that DCR achieves improved efficiency over the baseline while ensuring valid coverage under mild assumptions. Extensive experiments demonstrate the superiority of DCR, reducing average prediction set size by up to 36%, while maintaining valid coverage.

💡 Research Summary

The paper tackles the problem of quantifying uncertainty for ranking models in the full‑ranking setting, where a set of n calibration items and m test items must be jointly ordered but only the relative ranks of the calibration items are observable. Classical conformal prediction cannot be applied directly because the non‑conformity scores of the calibration items depend on their absolute ranks, which in turn depend on the unknown test scores. The recent Transductive Conformal Prediction for Ranking (TCPR) circumvents this by constructing high‑probability upper and lower bounds for each calibration item’s absolute rank and then using worst‑case scores derived from those bounds. While TCPR guarantees marginal coverage, its reliance on conservative bounds leads to overly large prediction sets, especially when the bound intervals are wide or when the confidence adjustment (adding δ to the target miscoverage α) inflates the threshold.

The authors propose Distribution‑informed Conformal Ranking (DCR), a method that eliminates the need for such bounds by exploiting the exact probabilistic structure of the latent absolute ranks under the exchangeability assumption. They prove that, conditional on a calibration item’s observed relative rank (R_{c,i}), the number of test items ranked lower than it, (R_{t,i}), follows a Negative Hypergeometric distribution with parameters ((N=n+m, m, R_{c,i})). Consequently, the absolute rank (R_{c,i}+R_{t,i}) follows a shifted Negative Hypergeometric distribution supported on ({R_{c,i},\dots,R_{c,i}+m}). This result (Proposition 3.1) enables the exact computation of the conditional distribution of each calibration item’s non‑conformity score (S_i = s(X_i, R_{c,i}+R_{t,i})) for both the RA (rank‑as‑integer) and VA (value‑as‑real) settings.

Because the true scores are unobservable, DCR does not attempt to estimate a single empirical CDF of the scores. Instead, for each calibration item it computes the conditional CDF (F_i(t)=\Pr(S_i\le t\mid R_{c,i})) using the known Negative Hypergeometric law. The mixture CDF (F_{\text{mix}}(t)=\frac{1}{n}\sum_{i=1}^n F_i(t)) is then defined as the expectation of the latent empirical CDF given the observed relative ranks. This mixture CDF represents the average distribution of the scores after marginalizing out the aleatory uncertainty of the unseen test ranks.

The DCR threshold (s^) is chosen as the smallest t such that (F_{\text{mix}}(t)) exceeds the target quantile (\lceil (n+1)(1-\alpha)\rceil/(n+1)). Prediction sets for each test item are formed by including all absolute ranks r for which the non‑conformity score (s(X_{n+j},r)) does not exceed (s^). The authors prove that, under exchangeability, DCR attains the nominal marginal coverage (1-\alpha) without the extra δ slack required by TCPR. Moreover, they show that the expected size of DCR’s prediction sets is strictly smaller than that of TCPR for the same miscoverage level, because DCR uses the exact score distribution rather than a worst‑case bound.

Empirical evaluation is conducted on several ranking models (LambdaMart, RankNet) and datasets (ESOL, synthetic data). Results indicate that DCR reduces the average prediction set length by up to 36% relative to TCPR while maintaining coverage above the target (e.g., 57.6% vs. 72.5% relative length at 90% coverage on ESOL). Additional analyses reveal that larger calibration sets or smaller test sets bring the realized coverage closer to the nominal level, confirming that DCR more effectively leverages the rank distribution. The paper also introduces a stochastic approximation, Monte‑Carlo Distribution‑informed Conformal Ranking (MDCR), which samples from the conditional rank distribution to approximate the mixture CDF, offering computational savings for very large n and m while preserving marginal coverage guarantees.

In summary, DCR provides a principled, distribution‑aware conformal prediction framework for full‑ranking problems. By deriving the exact conditional distribution of unseen absolute ranks (Negative Hypergeometric) and integrating it into the conformal machinery, DCR achieves substantially tighter prediction sets than the prior bound‑based TCPR, without sacrificing finite‑sample coverage. The method is model‑agnostic, requires only exchangeability, and scales to large problems via the Monte‑Carlo variant, making it a practical tool for uncertainty quantification in modern ranking applications.