Accordo assoluto tra valutazioni espresse su scala ordinale

Abstract

Many methods for measuring agreement among raters have been proposed and applied in many domains in the areas of education, psychology, sociology, and medical research. A brief overview of the most used measures of interrater absolute agreements for ordinal rating scales is provided, and a new index is proposed that has several advantages. In particular, the new index allows to evaluate the agreement between raters for each single case (subject or object), and to obtain also a global measure of the interrater agreement for the whole group of cases evaluated. The possibility of having evaluations of the agreement on the single case is particularly useful, for example, in situations where the rating scale is being tested, and it is necessary to identify any changes to it, or to request the raters for a specific comparison on the single case in which the disagreement occurred. The index is not affected by the possible concentration of ratings on a very small number of levels of the ordinal scale.

Keywords

• educational assessment
• interrater agreement
• kappa index
• ordinal rating scales
• statistical dispersion.