Open-Set Tattoo Semantic Segmentation

Abstract

Tattoos can serve as an essential source of biometric information for public security, aiding in identifying suspects and victims. In order to automate tattoo classification, tasks like classification require more detailed image content analysis, such as semantic segmentation. However, a dataset with appropriate semantic segmentation annotations is currently lacking. Also, there are countless ways to categorize tattoo classes, and many are not directly categorizable, either because they belong to a specific artistic trait or characterize an object with previously undefined semantics. An effective way to overcome these limitations is to build recognition systems based on open-set assumptions. Nevertheless, state-of-the-art open set approaches are not directly applicable in tattoo semantic segmentation, mainly due to the significant class imbalance (predominant background). To the best of our knowledge, this paper is the first to explore semantic segmentation in closed and open-set scenarios for tattoos. In this sense, this paper presents two key contributions: (i) a novel large-margin loss function and generalized open-set classifier approach and (ii) an open-set tattoo semantic segmentation dataset with a publicly accessible test set, enabling comparisons and future research in this area. The proposed approach outperforms other methods, achieving 0.8013 of AUROC, 0.6318 of Macro F1, 0.4900 of mIoU, and notably 0.2753 of IoU for the unknown class, demonstrating the feasibility of this approach for automatic tattoo analysis. The paper also highlights key limitations and open research areas in this challenging field. Dataset and codes are available at https://github.com/Brilhador/tssd2023.

Keywords

• Open-world
• open-set
• semantic segmentation
• large-margin learning
• tattoo classification