Do More Predictions Improve Statistical Inference? Filtered Prediction-Powered Inference

Recent advances in artificial intelligence have enabled the generation of large-scale, low-cost predictions with increasingly high fidelity. As a result, the primary challenge in statistical inference has shifted from data scarcity to data reliability. Prediction-powered inference methods seek to exploit such predictions to improve efficiency when labeled data are limited. However, existing approaches implicitly adopt a use-all philosophy, under which incorporating more predictions is presumed to improve inference. When prediction quality is heterogeneous, this assumption can fail, and indiscriminate use of unlabeled data may dilute informative signals and degrade inferential accuracy. In this paper, we propose Filtered Prediction-Powered Inference (FPPI), a framework that selectively incorporates predictions by identifying a data-adaptive filtered region in which predictions are informative for inference. We show that this region can be consistently estimated under a margin condition, achieving fast rates of convergence. By restricting the prediction-powered correction to the estimated filtered region, FPPI adaptively mitigates the impact of biased or noisy predictions. We establish that FPPI attains strictly improved asymptotic efficiency compared with existing prediction-powered inference methods. Numerical studies and a real-data application to large language model evaluation demonstrate that FPPI substantially reduces reliance on expensive labels by selectively leveraging reliable predictions, yielding accurate inference even in the presence of heterogeneous prediction quality.

💡 Research Summary

The paper addresses a fundamental shift in statistical inference brought about by the proliferation of inexpensive, high‑fidelity predictions from modern AI systems. While traditional prediction‑powered inference (PPI) and its enhanced version PPI++ exploit these predictions to improve efficiency when labeled data are scarce, they implicitly assume that every prediction contributes equally. The authors demonstrate that when prediction quality varies across the covariate space, this “use‑all” philosophy can actually degrade performance: the global correction term may cancel out useful signal with noisy or biased information, leading to no variance reduction or even increased variance, as illustrated by a simple mean‑estimation example.

To overcome this limitation, the authors propose Filtered Prediction‑Powered Inference (FPPI). The key idea is to identify a data‑adaptive subset S of the covariate space where the model’s predictions are reliably informative, and to apply the prediction‑based correction only to observations falling inside S. Formally, the FPPI objective augments the standard empirical loss with a weighted correction term that is multiplied by the indicator 1_S(x). This construction preserves unbiasedness for the target risk while allowing selective use of predictions.

A central theoretical contribution is the introduction of a “margin condition” that characterizes how well the true regression function and the prediction model are separated near the decision boundary of S. Under this condition, the authors prove that the filtered region can be consistently estimated from the available labeled and unlabeled data using a leave‑one‑out scheme. When the covariate distribution is sufficiently separated from the boundary, the convergence rate of the estimated region is fast—potentially exponential—ensuring that the filtering step is both statistically reliable and computationally feasible.

With a correctly estimated region Ŝ and an optimally tuned λ, the authors derive the asymptotic variance of the FPPI estimator. They show that it is strictly smaller than that of PPI++: the variance reduction term Δ is positive whenever the filtered region captures a non‑trivial portion of the space where predictions and outcomes are positively correlated. Consequently, FPPI not only guarantees at least the performance of the label‑only estimator (as PPI++ does) but also delivers a provable efficiency gain in realistic heterogeneous‑prediction settings.

Empirical validation is carried out on three fronts. First, synthetic experiments with deliberately heterogeneous prediction quality confirm that FPPI achieves 10‑25 % lower mean‑squared error than PPI++ while using far fewer labeled samples. Second, high‑dimensional linear regression simulations demonstrate that FPPI automatically discards noisy dimensions, yielding sharper parameter estimates. Third, a real‑world application to large language model (LLM) evaluation shows that FPPI can reduce labeling effort by more than 70 % while tightening confidence intervals for model scores by roughly 40 %. The method automatically identifies topics or contexts where the LLM’s predictions are trustworthy, illustrating its practical utility.

In summary, the paper introduces a principled, theoretically grounded framework for selective exploitation of AI‑generated predictions. By filtering out low‑quality predictions rather than globally re‑weighting them, FPPI adapts to heterogeneous prediction landscapes, achieves faster convergence of the filtered region, and delivers strictly improved asymptotic efficiency. The work opens avenues for extensions such as ensemble‑based filtering, non‑convex loss functions, and online adaptive filtering in streaming data environments.