Foundation Models and Fine-Tuning: A Benchmark for Out of Distribution Detection

Abstract

The rise of foundation models is pushing Computer Vision research towards a paradigm shift, in the wake of what already happened in the Natural Language Processing field. These models, trained at scale on huge data collections, provide high-quality representations that generalize well enough to be applied directly to downstream tasks, often outperforming task-specific models. The Out Of Distribution (OOD) detection problem, which involves the ability to recognize when test samples come from a previously unseen semantic category, represents one of the research fields in which this paradigm shift could have the greatest impact. However, existing testbeds are limited in scale and scope and get easily saturated when adopting foundation-based pretrainings. With this work, we introduce a new benchmark covering realistic yet harder OOD detection tasks to properly assess the performance of large pretrained models. We design an experimental framework to analyze specific choices in the model learning and use (which dataset, pretraining objective, OOD scoring function) and extensively evaluate the comparison to standard approaches that leverage a training phase on the available In Distribution (ID) data. The results highlight the actual performance benefits of leveraging foundation models in this context without any further learning effort, and identify situations where task-specific fine-tuning remains the best choice.

Keywords

• Fine-tuning-free
• semantic novelty detection
• open set recognition
• out of distribution detection