Revisiting Generalization Measures Beyond IID: An Empirical Study under Distributional Shift

Generalization remains a central yet unresolved challenge in deep learning, particularly the ability to predict a model’s performance beyond its training distribution using quantities available prior to test-time evaluation. Building on the large-scale study of Jiang et al. (2020). and concerns by Dziugaite et al. (2020). about instability across training configurations, we benchmark the robustness of generalization measures beyond IID regime. We train small-to-medium models over 10,000 hyperparameter configurations and evaluate more than 40 measures computable from the trained model and the available training data alone. We significantly broaden the experimental scope along multiple axes: (i) extending the evaluation beyond the standard IID setting to include benchmarking for robustness across diverse distribution shifts, (ii) evaluating multiple architectures and training recipes, and (iii) newly incorporating calibration- and information-criteria-based measures to assess their alignment with both IID and OOD generalization. We find that distribution shifts can substantially alter the predictive performance of many generalization measures, while a smaller subset remains comparatively stable across settings.

💡 Research Summary

This paper presents a large-scale empirical study that critically re-evaluates the efficacy of generalization measures in deep learning, extending beyond the traditional independent and identically distributed (IID) setting to include out-of-distribution (OOD) scenarios. Motivated by the foundational work of Jiang et al. (2020) and the practical need to predict model performance under distribution shifts, the authors conduct an extensive benchmark involving over 10,000 model training runs across diverse hyperparameter configurations. They evaluate more than 40 generalization measures, which are computable solely from the trained model and its training data, significantly broadening the scope by incorporating two new categories: Information Criteria (e.g., AIC, WAIC) and Calibration & Confidence metrics (e.g., Expected Calibration Error).

The experimental framework uses datasets like CIFAR-10 (with its CIFAR-10-C and CIFAR-10-P variants for OOD evaluation) and DomainBed’s PACS and VLCS. Models range from small CNNs to ResNets, trained under varied optimizers, learning rates, and other hyperparameters. The core evaluation metric is the generalization gap (difference between training and test accuracy). To assess the predictive power and robustness of each measure, the authors employ two primary methodologies: the “Granulated Score (Ψ)”, which averages Kendall’s rank correlation (τ) across subspaces where only one hyperparameter varies, and “Sign-error Distributions”, which quantify how often a measure’s ranking of two models disagrees with the ranking of their actual generalization gaps when a single hyperparameter is changed.

The results reveal several key insights. First, distribution shifts substantially alter the predictive performance of most generalization measures. A measure that correlates well with the generalization gap under IID conditions may show a weak, negative, or highly unstable correlation under OOD conditions. Second, traditional Norm & Margin-based measures (e.g., parameter norms) generally showed weak negative correlations in both settings, confirming their limited reliability observed in prior work. Third, Sharpness-based measures (e.g., PAC-Bayes bounds), while sometimes predictive in IID, displayed inconsistent behavior in OOD scenarios, challenging the universal link between flat minima and robustness.

Most notably, the newly introduced categories exhibited unique behaviors. Information Criteria and Calibration-based measures, which were largely non-predictive in the standard IID setting, demonstrated surprisingly high predictive power in certain OOD scenarios. This suggests that metrics related to model confidence and probabilistic fit may harbor valuable signals for forecasting robustness to distribution shifts. However, this effectiveness was not universal; it fluctuated or even reversed depending on the specific type of distribution shift and training regime.

The central conclusion of the paper is that there is no universal generalization measure that remains consistently predictive across both IID and diverse OOD settings. This finding underscores the inherent risk of relying on any single metric for model selection or performance forecasting in practice. The study advocates for a context-aware evaluation framework, where the choice of generalization measure must be informed by the anticipated deployment conditions (type of distribution shift) and the specific model training methodology. By expanding the benchmark to OOD and modern metrics, this work sets a new standard for research on generalization predictors and highlights the growing importance of calibration and uncertainty-aware metrics in building reliable machine learning systems.