Simultaneous analysis of approximate leave-one-out cross-validation and mean-field inference

Approximate Leave-One-Out Cross-Validation (ALO-CV) is a method that has been proposed to estimate the generalization error of a regularized estimator in the high-dimensional regime where dimension and sample size are of the same order, the so-called ``proportional regime’’. A new analysis is developed to derive the consistency of ALO-CV for non-differentiable regularizers under Gaussian covariates and strong convexity. Using a conditioning argument, the difference between the ALO-CV weights and their counterparts in mean-field inference is shown to be small. Combined with upper bounds between the mean-field inference estimate and the leave-one-out quantity, this provides a proof that ALO-CV approximates the leave-one-out quantity up to negligible error terms. Linear models with square loss, robust linear regression and single-index models are explicitly treated.

💡 Research Summary

The paper tackles the problem of estimating the generalization error of a regularized estimator in the high‑dimensional proportional regime, where the number of features p and the sample size n grow proportionally (p/n → δ ∈ (0,1)). Two prominent approaches have been proposed for this task: Approximate Leave‑One‑Out Cross‑Validation (ALO‑CV) and mean‑field inference. ALO‑CV approximates the costly leave‑one‑out (LOO) estimator by performing a single Newton step from the full‑data solution, thus requiring only one optimization of the original objective. Mean‑field inference, on the other hand, studies the asymptotic behavior of the estimator under Gaussian designs and shows that certain linear functionals of the estimator converge to deterministic limits that can be expressed through a small set of scalar parameters (γ*, a*, σ*).

The main contribution of the paper is a unified analysis that simultaneously treats ALO‑CV and mean‑field inference, establishing that ALO‑CV is essentially a concrete implementation of the mean‑field correction. The authors work under the following assumptions: (i) the design vectors x_i are i.i.d. Gaussian with covariance Σ; (ii) the loss functions L_y(·) are convex, Lipschitz, and twice differentiable (or at least have a well‑defined first derivative); (iii) the regularizer R is (nμ, Σ)‑strongly convex, which includes smooth penalties as well as non‑smooth ones such as the L1 norm or the elastic‑net, provided the active set is locally constant. Under these conditions, they prove a key differentiability result (Proposition 1.1) showing that the estimator \hat b is almost everywhere differentiable with respect to each entry of the design matrix, and that its derivative can be written as

\