Learning From Multiple Expert Annotators for Enhancing Anomaly Detection in Medical Image Analysis

Abstract

Recent years have experienced phenomenal growth in computer-aided diagnosis systems based on machine learning algorithms for anomaly detection tasks in the medical image domain. However, the performance of these algorithms greatly depends on the quality of labels since the subjectivity of a single annotator might decline the certainty of medical image datasets. In order to alleviate this problem, aggregating labels from multiple radiologists with different levels of expertise has been established. In particular, under the reliance on their own biases and proficiency levels, different qualified experts provide their estimations of the “true” bounding boxes representing the anomaly observations. Learning from these nonstandard labels exerts negative effects on the performance of machine learning networks. In this paper, we propose a simple yet effective approach for the enhancement of neural networks’ efficiency in abnormal detection tasks by estimating the actually hidden labels from multiple ones. A re-weighted loss function is also used to improve the detection capacity of the networks. We conduct an extensive experimental evaluation of our proposed approach on both simulated and real-world medical imaging datasets, MED-MNIST and VinDr-CXR. The experimental results show that our approach is able to capture the reliability of different annotators and outperform relevant baselines that do not consider the disagreements among annotators. Our code is available at https://github.com/huyhieupham/learning-from-multiple-annotators.

Keywords

• Computer-aided diagnosis
• deep learning
• object detection
• multiple annotators