An Enhanced Mask Transformer for Overlapping Cervical Cell Segmentation Based on DETR

Abstract

Automated cell segmentation in cervical cytology images is an essential task because it can present a deep understanding of the characteristics of cervical cells. The main challenge is that cells overlap at a high rate, making the cell boundaries extremely blurred. While the transformer-based models have been proven to be effective in vision tasks, this paper proposes an enhanced mask transformer based on DETR to segment overlapping cervical cells. Dynamic anchor box initialization and noised ground truth box embedding are introduced, to improve segmentation performance and accelerate model convergence. The proposed model achieves a 0.974 DSC, 0.971 TPRp, 0.0005 FPRp and 0.0012 FNRo on the ISBI2014 dataset. Specially, the proposed method outperforms state-of-the-art result by about 3.4% on DSC, 2% on TPRp and 1.88% on FNRo, respectively. The metrics of our model on the ISBI2015 dataset are a little better than the averaged metrics of other impressive methods. These findings present strong support for the transformer-based neural networks for effective segmentation of cells in cervical cytology images.

Keywords

• Cervical cell segmentation
• DETR
• ISBI datasets
• transformer