Triple-Sigmoid Activation Function for Deep Open-Set Recognition

Abstract

Traditional models for various machine learning problems such as image classification perform well only under the assumption of a closed set. This implies that inputs must belong to the classes for which the models were trained. Data collected in the real world may not belong to any finite set of classes, and training a model with an infinite number of classes would obviously be impossible. Rather than incorrectly classifying outlier inputs that belong to unknown classes as members of one of the classes on which the model was trained, learning models must recognize and reject such data samples, or request human assistance in labeling them. For example, when a self-driving vehicle detects unfamiliar scenes or objects, it must notify the driver and hand over control. Various models have been proposed to address the open-set problem. However, the existing models are generally complex or use complicated techniques such as generative adversarial networks and auto-encoders, and their efficiency is not proportional to their complexity. In this study, we propose Triple-Sigmoid as a simple activation function comprised of three Sigmoid functions. Using Triple-Sigmoid in the last activation layer of any deep neural network model enables the model to recognize outliers. Although Triple-Sigmoid can be applied to a variety of machine learning problems, including semi-supervised learning, active learning, and incremental learning, we only investigated the open-set recognition problem in this work. The results of numerous experiments are presented to validate that substituting Triple-Sigmoid for conventional activations such as Softmax and Sigmoid not only retained performance in closed-set settings, but also significantly improved performance in open-set configurations. Furthermore, these results demonstrate that Triple-Sigmoid significantly outperformed existing state-of-the-art methods. Source code for the proposed model is available at https://github.com/dinhtuantran/triple-sigmoid.

Keywords

• Active learning
• anomaly detection
• incremental learning
• novelty detection
• open-set recognition
• out-of-distribution detection