To What Extent Do Token-Level Representations from Pathology Foundation Models Improve Dense Prediction?

Pathology foundation models (PFMs) have rapidly advanced and are becoming a common backbone for downstream clinical tasks, offering strong transferability across tissues and institutions. However, for dense prediction (e.g., segmentation), practical deployment still lacks a clear, reproducible understanding of how different PFMs behave across datasets and how adaptation choices affect performance and stability. We present PFM-DenseBench, a large-scale benchmark for dense pathology prediction, evaluating 17 PFMs across 18 public segmentation datasets. Under a unified protocol, we systematically assess PFMs with multiple adaptation and fine-tuning strategies, and derive insightful, practice-oriented findings on when and why different PFMs and tuning choices succeed or fail across heterogeneous datasets. We release containers, configs, and dataset cards to enable reproducible evaluation and informed PFM selection for real-world dense pathology tasks. Project Website: https://m4a1tastegood.github.io/PFM-DenseBench

💡 Research Summary

The paper introduces PFM‑DenseBench, a comprehensive benchmark that evaluates how token‑level representations from pathology foundation models (PFMs) affect dense prediction tasks such as semantic segmentation. Seventeen state‑of‑the‑art PFMs—covering both vision‑only (e.g., UNI, Virchow, Gigapath, PathOrchestra, Lunit, Kaiko, Hibou, H‑Optimus, Midnight‑12k) and vision‑language (CONCH, MUSK) families—are systematically tested on eighteen publicly available pathology segmentation datasets. These datasets span multiple organs (breast, colon, lung, prostate, kidney, liver, stomach, pancreas) and three annotation granularities (nuclei‑level, gland/structure‑level, tissue‑level), enabling a thorough assessment of transferability across biological scales.

The authors define a unified experimental pipeline: (1) dataset curation and patch extraction, (2) model adaptation using five strategies—(i) frozen encoder, (ii) Low‑Rank Adaptation (LoRA), (iii) Dynamic‑Rank Adaptation (DoRA), (iv) CNN adapter (parallel ResNet‑style branch with skip connections), and (v) Transformer adapter (lightweight transformer blocks appended to the frozen encoder)—and (3) a common decoder (UNet‑style) with a segmentation head. Evaluation employs multiple metrics (mDice, mIoU, pixel accuracy, F1, precision, recall) and non‑parametric bootstrap (B=1000) to report mean performance with 95 % confidence intervals, ensuring statistical robustness.

Key findings:

1. Baseline Transfer Gains – PFMs consistently outperform a strong supervised UNet baseline, delivering an average 3–5 percentage‑point (pp) increase in mDice. Gains are dataset‑dependent: tissue‑level datasets (e.g., BCSS, WSSS4LUAD) see up to 7 pp improvement, while nuclei‑level datasets (e.g., NuCLS, PanNuke) improve by ≤2 pp. This suggests PFMs capture global slide semantics well but lack fine‑grained boundary detail.
2. Adapter Impact – LoRA/DoRA provide parameter‑efficient adaptation but yield modest gains (1–2 pp). CNN adapters, which inject local convolutional features, achieve the largest and most consistent boost (average +3.5 pp, up to +6 pp), especially on datasets with complex boundaries (GlaS, CRAG). Transformer adapters improve global context but increase compute and can overfit on smaller sets.
3. Scaling Laws – Increasing model size from 100 M to 1 B parameters improves mDice by only ~1.2 pp, while scaling pre‑training data from 100 k to 1 M WSIs yields ~0.8 pp gain. Performance appears to saturate, indicating that simply enlarging PFMs is insufficient for dense tasks; efficient adaptation is more critical.
4. Vision‑Language Models – Text‑aligned PFMs (CONCH, MUSK) offer marginal (≈0.5 pp) advantages on a few tissue‑level datasets but do not outperform vision‑only counterparts overall, likely due to limited quality/quantity of pathology image‑text pairs.
5. Reproducibility – The authors release Docker containers, configuration files, and dataset cards, allowing the community to replicate results and benchmark new PFMs under the same protocol.

Overall, the study demonstrates that while pathology foundation models provide a solid generic feature backbone, their token‑level representations alone are insufficient for optimal dense prediction. Incorporating local convolutional adapters yields the most reliable performance gains across scales, and scaling model size or pre‑training data shows diminishing returns. The benchmark sets a new standard for evaluating PFMs on segmentation and offers practical guidance for researchers and clinicians seeking to deploy foundation models in real‑world pathology workflows.