An information-oriented paradigm in evaluating accuracy and agreement in radiology

Abstract

Abstract The goal of any radiological diagnostic process is to gain information about the patient’s status. However, the mathematical notion of information is usually not adopted to measure the performance of a diagnostic test or the agreement among readers in providing a certain diagnosis. Indeed, commonly used metrics for assessing diagnostic accuracy (e.g., sensitivity and specificity) or inter-reader agreement (Cohen $$\kappa$$ κ statistics) use confusion matrices containing the number of true- and false positives/negatives results of a test, or the number of concordant/discordant categorizations, respectively, thus lacking proper information content. We present a methodological paradigm, based on Shannon’s information theory, aiming to measure both accuracy and agreement in diagnostic radiology. This approach models the information flow as a “diagnostic channel” connecting the state of the patient’s disease and the radiologist or, in the case of agreement analysis, as an “agreement channel” linking two or more radiologists evaluating the same set of images. For both cases, we proposed some measures, derived from Shannon’s mutual information, which can represent an alternative way to express diagnostic accuracy and agreement in radiology. Key points • Diagnostic processes can be modeled with information theory (IT). • IT metrics of diagnostic accuracy are independent from disease prevalence. • IT metrics of inter-reader agreements can overcome Cohen κ pitfalls.

Keywords

• Diagnostic imaging
• Information theory
• Observer variation
• Research design
• Sensitivity and specificity