Label-Free Model Evaluation with Out-of-Distribution Detection

Abstract

In recent years, label-free model evaluation has been developed to estimate the performance of models on unlabeled test sets. However, we find that existing methods perform poorly in environments with out-of-distribution (OOD) data. To address this issue, we propose a novel automatic model evaluation method using OOD detection to reduce the impact of OOD data on model evaluation. Specifically, we use the representation of datasets to train a neural network for accuracy prediction and employ energy-based OOD detection to exclude OOD data during testing. We conducted experiments on several benchmark datasets with varying amounts of OOD data (SVHN, ISUN, ImageNet, and LSUN) and demonstrated that our method reduces the RMSE compared to existing methods by at least 1.27%. Additionally, we tested our method on transformed datasets and datasets with a high proportion of OOD data, and the results show its robustness.

Keywords

• out-of-distribution detection
• model accuracy prediction
• model generalization