From Sigmoid to SoftProb: A novel output activation function for multi-label learning

Abstract

Multi-label classification is a crucial machine learning task that assigns multiple labels to a single instance, making it distinct from traditional single-label classification. The sigmoid activation function, commonly used in multi-label learning, suffers from saturation and vanishing gradient issues, which can hinder model performance. To address these limitations, we propose SoftProb, a novel output activation function designed to improve gradient flow and predictive performance while maintaining computational efficiency. We conduct a comprehensive theoretical and empirical analysis comparing SoftProb and sigmoid across shallow, medium, and deep multilayer perceptrons on six benchmark datasets. The results demonstrate that SoftProb achieves statistically significant improvements in key metrics, including a 5.15% increase in Macro F1-score and a 2.60% improvement in Average Precision Score (APS), while maintaining comparable training times to sigmoid (p>0.05). Although SoftProb showed a marginal 0.40% increase in Hamming Loss, it provides better balance between precision and recall, particularly in deeper network architectures. Notably, SoftProb’s simplified mathematical formulation avoids exponential operations, offering potential implementation advantages. Statistical validation using the Wilcoxon signed-rank test confirms the significance of the performance improvements (p<0.05 for F1 and APS). These findings establish SoftProb as a robust alternative to sigmoid for multi-label classification, combining enhanced predictive performance with stable computational characteristics.

Keywords

• Multi-label classification
• Activation function
• SoftProb
• Neural networks
• Deep learning
• Sigmoid function