A Function-Space Stability Boundary for Generalization in Interpolating Learning Systems

Modern learning systems often interpolate training data while still generalizing well, yet it remains unclear when algorithmic stability explains this behavior. We model training as a function-space trajectory and measure sensitivity to single-sample perturbations along this trajectory. We propose a contractive propagation condition and a stability certificate obtained by unrolling the resulting recursion. A small certificate implies stability-based generalization, while we also prove that there exist interpolating regimes with small risk where such contractive sensitivity cannot hold, showing that stability is not a universal explanation. Experiments confirm that certificate growth predicts generalization differences across optimizers, step sizes, and dataset perturbations. The framework therefore identifies regimes where stability explains generalization and where alternative mechanisms must account for success.

💡 Research Summary

The paper tackles the puzzling phenomenon that modern over‑parameterized learning systems can achieve (near) zero training error—i.e., they interpolate the data—while still displaying strong out‑of‑sample performance. Classical generalization analyses based on uniform convergence or parameter‑level algorithmic stability often fail to explain this regime because they either become vacuous or depend heavily on specific optimizer settings. To address this gap, the authors propose a function‑space perspective: they view the learning process as a trajectory of predictors ( {f_t(S,U)}_{t=0}^T ) in a hypothesis space ( \mathcal{F} ), where ( S ) is the training set and ( U ) denotes all sources of randomness (initialization, minibatch sampling, dropout, etc.).

The central object is a discrepancy functional ( d: \mathcal{F}\times\mathcal{F}\to\mathbb{R}_+ ) that upper‑bounds pointwise loss differences via a Lipschitz constant ( L_d ): for any two predictors ( f,g ) and any example ( z ), (|\ell(f,z)-\ell(g,z)|\le L_d, d(f,g)). This functional can be instantiated by the Euclidean distance between output vectors, the (L_2) distance of logits, or any other metric that respects the loss’s smoothness.

Given two neighboring datasets ( S ) and ( S’ ) (differing in exactly one example) and a shared randomness realization ( U ), the authors define the trajectory discrepancy sequence \