Foundation Models for Histopathology—Fanfare or Flair

Abstract

Objective: To assess the performance of the current foundation models in histopathology. Patients and Methods: The assessment involves a comprehensive evaluation of some foundation models, such as the CLIP derivatives, namely PLIP and BiomedCLIP, which were fine-tuned on data scraped from the internet. The comparison is performed against simpler and nonfoundational histology models that are trained on well-curated data, eg, the cancer genome atlas. All models are evaluated on 8 datasets, 4 of which are internal histology datasets collected and curated at Mayo Clinic, and 4 well-known public datasets: PANDA, BRACS, CAMELYON16, and DigestPath. Evaluation metrics include accuracy and macro-averaged F1 score, using a majority vote among top-k (eg, MV@5) at the whole slide image/patch levels. Moreover, all models are evaluated in classification settings. This detailed analysis allows for a deep understanding of each model’s performance across various datasets. Results: In various evaluation tasks, domain-specific (and nonfoundational) models like DinoSSLPath and KimiaNet outperform general-purpose foundation models. The DinoSSLPath excels in whole slide image-level retrieval for internal colorectal cancer and liver datasets with MV@5 macro-averaged F1 scores of 63% and 74%, respectively. The KimiaNet leads in breast and skin cancer tasks with respective Top-1 and MV@5 scores of 56% and 70%, respectively and scores 75% on the public CAMELYON16 dataset. Similar trends are observed in patch-level metrics, highlighting the advantage of using specialized datasets like the cancer genome atlas for histopathological analysis. Conclusion: To enable effective vision-language foundation models in biomedicine, high-quality, multi-modal medical datasets are essential. These datasets serve as the substrate for training models capable of translating research into clinical practice. Of importance, the alignment (correspondence) between textual and visual data—often diagnostic—is critical and requires validation by domain experts. Thus, advancing foundation models in this field necessitates collaborative efforts in data curation and validation.