A Systematic Analysis of Out-of-Distribution Detection Under Representation and Training Paradigm Shifts

We present the largest systematic comparison to date of out-of-distribution (OOD) detection methods using AURC and AUGRC as primary metrics. Our comparison explores different regimes of distribution shift (stratified by CLIP embeddings of the out-of-distribution image datasets) with varying numbers of classes and uses a representation-centric view of OOD detection, including neural collapse metrics, for subsequent analysis. Together the empirical results and representation analysis provides novel insights and statistically grounded guidance for method selection under distribution shift. Experiments cover two representation paradigms: CNNs trained from scratch and a fine-tuned Vision Transformer (ViT), evaluated on CIFAR-10/100, SuperCIFAR-100, and TinyImageNet. Using a multiple-comparison-controlled, rank-based pipeline (Friedman test with Conover-Holm post-hoc) and Bron-Kerbosch cliques, we find that the learned feature space largely determines OOD efficacy. For both CNNs and ViTs, probabilistic scores (e.g., MSR, GEN) dominate misclassification (ID) detection. Under stronger shifts, geometry-aware scores (e.g., NNGuide, fDBD, CTM) prevail on CNNs, whereas on ViTs GradNorm and KPCA Reconstruction Error remain consistently competitive. We further show a class-count-dependent trade-off for Monte-Carlo Dropout (MCD) and that a simple PCA projection improves several detectors. The neural-collapse-based geometric analysis explains when prototype and boundary-based scores become optimal under strong shifts.

💡 Research Summary

The paper presents the most extensive systematic comparison of out‑of‑distribution (OOD) detection methods to date, focusing on how the learned representation and the training paradigm influence performance. Two backbone families are examined: convolutional neural networks (CNNs) trained from scratch and a Vision Transformer (ViT) fine‑tuned from a CLIP foundation model. Both are evaluated on four in‑distribution (ID) datasets of increasing class cardinality—CIFAR‑10 (10 classes), CIFAR‑100 (100), SuperCIFAR‑100 (20 super‑classes), and TinyImageNet (200)—and on a suite of OOD datasets that are stratified into near, mid, and far semantic shifts using CLIP‑based embedding distances (Fréchet distance, MMD, and class‑conditional cosine distances). This CLIP‑driven stratification removes subjective bias from the definition of shift strength.

Performance is measured primarily with the Area Under the Risk‑Coverage curve (AURC) and its generalized version (AUGRC), which integrate selective risk over the entire coverage range, providing a more holistic view of detector reliability than traditional metrics such as FPR@95 or AUROC. Twenty‑plus confidence scoring functions (CSFs) are benchmarked, grouped into three families: (1) probabilistic scores (Maximum Softmax Response – MSR, Generalized Energy – GEN), (2) geometry‑aware scores (NNGuide, fDBD, Class‑wise Transfer Metric – CTM), and (3) gradient‑based scores (GradNorm, Monte‑Carlo Dropout – MCD). Each CSF is evaluated on raw penultimate‑layer features as well as on features after global or class‑specific PCA projection, allowing the authors to assess the impact of low‑rank denoising.

Key findings:

1. The learned feature space dominates OOD detection performance regardless of backbone; when the same representation is used, CNNs and ViTs exhibit similar CSF rankings.
2. Probabilistic scores (MSR, GEN) consistently excel at detecting mis‑classified ID samples, especially on smaller‑class datasets (CIFAR‑10/100).
3. As shift strength increases (far OOD), geometry‑aware scores become superior for CNNs. NNGuide leverages distances to class prototypes, fDBD exploits class‑boundary thickness, and CTM captures angular separation between class means. For ViTs, gradient‑based scores—GradNorm and KPCA reconstruction error—remain robust across all shift levels, likely due to the token‑level attention dynamics of transformers.
4. Monte‑Carlo Dropout shows a class‑count‑dependent trade‑off: uncertainty estimates deteriorate as the number of classes grows, leading to higher AURC. A simple global PCA projection followed by reconstruction error improves multiple detectors (including MSR, GEN, GradNorm) by 2–4 percentage points on average.
5. Statistical analysis employs Friedman tests with Conover‑Holm post‑hoc correction to control for multiple comparisons, and Bron‑Kerbosch clique detection to identify equivalence groups among CSFs. Three distinct cliques emerge: probabilistic, geometry‑aware, and gradient‑based.

The authors further analyze the geometry of the penultimate‑layer using Neural Collapse (NC) metrics. As training progresses, class‑center means and classifier weight vectors align, within‑class variance shrinks, and inter‑class angles increase. Stronger shifts and larger class counts amplify NC, which explains why geometry‑aware detectors thrive under these conditions: clearer class prototypes make distance‑based discrimination more reliable. Conversely, ViTs retain a more isotropic feature distribution, making gradient‑norm signals more informative.

Overall, the paper provides statistically rigorous evidence that OOD detection is fundamentally a representation problem. It offers practical guidance: for small‑scale, low‑shift scenarios, simple probabilistic scores suffice; for large‑scale or heavily shifted domains, geometry‑aware methods (on CNNs) or gradient‑based methods (on ViTs) should be preferred, possibly after a low‑rank PCA denoising step. These insights help practitioners select and adapt OOD detectors based on backbone type, number of classes, and expected shift severity, moving the field toward more principled, deployment‑ready solutions.