Formal definition of the MARS method for quantifying the unique target class discoveries of selected machine classifiers [version 2; peer review: 2 approved]

Abstract

Conventional binary classification performance metrics evaluate either general measures (accuracy, F score) or specific aspects (precision, recall) of a model’s classifying ability. As such, these metrics, derived from the model’s confusion matrix, provide crucial insight regarding classifier-data interactions. However, modern- day computational capabilities have allowed for the creation of increasingly complex models that share nearly identical classification performance. While traditional performance metrics remain as essential indicators of a classifier’s individual capabilities, their ability to differentiate between models is limited. In this paper, we present the methodology for MARS (Method for Assessing Relative Sensitivity/ Specificity) ShineThrough and MARS Occlusion scores, two novel binary classification performance metrics, designed to quantify the distinctiveness of a classifier’s predictive successes and failures, relative to alternative classifiers. Being able to quantitatively express classifier uniqueness adds a novel classifier-classifier layer to the process of model evaluation and could improve ensemble model-selection decision making. By calculating both conventional performance measures, and proposed MARS metrics for a simple classifier prediction dataset, we demonstrate that the proposed metrics’ informational strengths synergize well with those of traditional metrics, delivering insight complementary to that of conventional metrics.

Keywords

• Machine learning
• Binary classification
• Classifier performance evaluation
• Classifier selection optimization
• Classifier comparative uniqueness
• eng