A Simplified Cohen’s Kappa for Use in Binary Classification Data Annotation Tasks

Abstract

In binary classification tasks, Cohen's kappa is often used as a quality measure for data annotations, which is inconsistent with its original purpose as an inter-annotator consistency measure. The analytic relationship between kappa and commonly used classification metrics (e.g., sensitivity and specificity) is nonlinear, and thus is difficult to be applied for interpretation of the classification performance (merely from the knowledge of the kappa value) of the annotations. In this study, based on an annotation generation model, we derive a simplified, linear relationship for Cohen's kappa, sensitivity, and specificity by using the 1st-order Taylor approximation. This relationship is further simplified by relating to Youden's J statistic, a performance metric for binary classification tasks. We provide an analysis on the linear coefficients in the simplified relationship and the approximation error, and conduct a linear regression analysis to assess the relationship by using a synthetic dataset where the ground truth is known. The results show that there is only a negligible approximation error in the simplified relationship when no major bias and prevalence issues exist. Furthermore, the relationship between kappa and Youden's J is validated on an annotation dataset from seven graders in a diabetic retinopathy screening study. The discrepancy between kappa and Youden's J is demonstrated to be an effective measure for annotator assessment when no ground truth is available.

Keywords

• Cohen’s kappa
• sensitivity
• specificity
• J statistic%22%2C%22default_operator%22%3A%22AND%22%7D%7D%7D"> Youden’s <italic xmlns:ali="http://www.niso.org/schemas/ali/1.0/" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">J</italic> statistic
• relationship
• annotator evaluation