Tumor-anchored deep feature random forests for out-of-distribution detection in lung cancer segmentation

Accurate segmentation of cancerous lesions from 3D computed tomography (CT) scans is essential for automated treatment planning and response assessment. However, even state-of-the-art models combining self-supervised learning (SSL) pretrained transformers with convolutional decoders are susceptible to out-of-distribution (OOD) inputs, generating confidently incorrect tumor segmentations, posing risks to safe clinical deployment. Existing logit-based methods suffer from task-specific model biases, while architectural enhancements to explicitly detect OOD increase parameters and computational costs. Hence, we introduce a lightweight, plug-and-play post-hoc random forests-based OOD detection framework called RF-Deep that leverages deep features with limited outlier exposure. RF-Deep enhances generalization to imaging variations by repurposing the hierarchical features from the pretrained-then-finetuned backbone, providing task-relevant OOD detection by extracting the features from multiple regions of interest anchored to the predicted tumor segmentations. We compared RF-Deep against existing OOD detection methods using 2,056 CT scans across near-OOD (pulmonary embolism, negative COVID-19) and far-OOD (kidney cancer, healthy pancreas) datasets. RF-Deep achieved AUROC > 93.50 for the challenging near-OOD datasets and near-perfect detection (AUROC > 99.00) for the far-OOD datasets, substantially outperforming logit-based and radiomics approaches. RF-Deep maintained consistent performance across networks of different depths and pretraining strategies, demonstrating its effectiveness as a lightweight, architecture-agnostic approach to enhance the reliability of tumor segmentation from CT volumes.

💡 Research Summary

Accurate delineation of lung tumors from 3D CT scans is a prerequisite for automated treatment planning and response monitoring, yet state‑of‑the‑art segmentation networks—particularly those that combine self‑supervised learning (SSL) pretrained transformers with convolutional decoders—remain vulnerable to out‑of‑distribution (OOD) inputs. When presented with scans that differ in acquisition parameters, disease patterns, or anatomical sites, these models can produce confidently wrong tumor masks, jeopardizing safe clinical deployment. Existing OOD detection strategies fall into two broad categories. Logit‑based scores (e.g., maximum softmax probability, temperature‑scaled confidence) are computationally cheap but inherit the biases of the underlying segmentation model, leading to high false‑positive rates. More sophisticated approaches either modify the network architecture (e.g., auxiliary OOD heads, generative diffusion models) or rely on high‑dimensional feature distances (Mahalanobis, PCA‑reduced embeddings). Both avenues increase parameter counts, training complexity, or inference latency, which is undesirable for real‑time radiology pipelines.

The authors propose RF‑Deep, a lightweight, plug‑and‑play, post‑hoc OOD detector that leverages the hierarchical feature representations already learned by a frozen segmentation backbone. The backbone is a Swin‑Transformer encoder (depth configuration 2‑2‑12‑2 across four stages) pretrained with the Self‑Distilled Masked Image Transformer (SMIT) framework and subsequently fine‑tuned on a lung‑cancer segmentation dataset. After training, the encoder is frozen; only a random‑forest (RF) classifier is trained on top of extracted features.

The detection pipeline consists of four steps:

1. Tumor‑anchored ROI extraction – The frozen segmentation model first generates a tumor mask for the whole volume. From each predicted tumor region, multiple 3D crops (128 × 128 × 128 voxels) are sampled with random spatial offsets, ensuring that the tumor appears somewhere within each crop without requiring precise centering. This “tumor‑anchored” strategy focuses the OOD analysis on anatomically relevant regions, which is crucial for distinguishing near‑OOD cases (e.g., pulmonary embolism or COVID‑19‑negative scans) that share the same thoracic field of view.

2. Multi‑scale feature extraction – For each ROI, the authors extract feature maps from all five encoder stages (patch embedding 48 channels, followed by Swin stages with 96, 192, 384, and 768 channels). Global average pooling (GAP) collapses each stage’s activation map into a single vector, and the vectors are concatenated to form a compact scan‑level descriptor (≈ 1,800 dimensions). This representation captures both low‑level texture and high‑level contextual information while remaining tractable for tree‑based classifiers.

3. Outlier exposure training – A small, curated set of OOD scans (as few as 20–30 examples) is used together with the in‑distribution (ID) lung‑cancer scans to train the RF. The outlier exposure (OE) strategy mitigates the tendency of unsupervised distance‑based methods to misclassify near‑OOD data, while keeping the training data requirement modest. The RF learns decision boundaries that separate subtle shifts in the feature space, even when the pathological patterns occupy the same anatomical region.

4. Online inference – At test time, the same ROI extraction and feature pooling are performed, and the RF outputs a probability of belonging to the ID distribution. Scores from multiple ROIs are averaged to produce a final scan‑level OOD decision, requiring no additional model fine‑tuning or architectural changes.

Experimental setup involved 2,056 CT volumes from five public datasets: an ID lung‑cancer cohort, two near‑OOD cohorts (pulmonary embolism, COVID‑19‑negative), and two far‑OOD cohorts (kidney cancer, healthy pancreas). All scans were resampled to 1 mm³ isotropic voxels, clipped to the lung window (−400 to 400 HU), and intensity‑normalized. Segmentation training used a combination of cross‑entropy and Dice loss, AdamW optimizer (lr = 2 × 10⁻⁴), and cosine annealing over 1,000 epochs. For OOD detection, the authors compared RF‑Deep against Mahalanobis distance on encoder embeddings, temperature‑scaled softmax confidence, radiomics‑based distance metrics, and recent OOD‑specific neural networks.

Results were striking. On the near‑OOD datasets, RF‑Deep achieved AUROC > 93.5 % (versus 70 %–75 % for logit‑based and Mahalanobis baselines). On far‑OOD datasets, AUROC exceeded 99 %, effectively eliminating false negatives. Performance remained stable across different backbone depths (e.g., a shallower 2‑2‑6‑2 configuration) and across alternative SSL pretraining schemes (SMIT vs. MoCo‑v2), confirming the method’s architecture‑agnostic nature. SHAP analysis revealed that features from the third and fourth Swin stages contributed most to OOD discrimination, aligning with intuition that mid‑to‑high‑level semantic cues are most informative. Moreover, the RF inference time per scan was ~10 ms, and feature extraction per ROI took ~120 ms, making the entire pipeline compatible with real‑time clinical workflows.

Interpretation highlights several key insights. First, anchoring ROIs to the predicted tumor forces the detector to focus on regions where pathological differences matter, rather than being diluted by background tissue. Second, the limited outlier exposure dramatically improves near‑OOD detection without requiring exhaustive OOD datasets, a practical advantage in medical settings where rare pathologies are hard to collect. Third, reusing the pretrained encoder’s features eliminates the need for a separate OOD feature extractor, preserving memory and computational budgets.

Limitations include reliance on the segmentation model to produce at least one tumor prediction; in truly normal scans (no tumor) the ROI extraction step may fail, although the absence of any foreground can itself be treated as an OOD signal. Additionally, the choice of ROI size and offset distribution was empirically set and may need tuning for other imaging modalities or resolutions. Finally, the method was evaluated only on CT; extending to MRI or PET would require validation of the feature transferability.

Conclusion and future directions – RF‑Deep demonstrates that a frozen, SSL‑pretrained segmentation backbone combined with a lightweight random‑forest classifier can deliver high‑fidelity, scan‑level OOD detection with minimal additional data and computational overhead. Its plug‑and‑play nature makes it attractive for integration into existing radiology AI pipelines, enhancing safety without sacrificing throughput. Future work may explore multimodal feature fusion (e.g., CT + PET), online adaptation of the RF to evolving scanner protocols, and dedicated mechanisms for handling completely tumor‑free scans.