Reliable visual analysis of single-case data: A comparison of rating, ranking, and pairwise methods

Abstract

The most common method of single-case data analysis is visual analysis, but interrater reliability among visual raters tends to be poor. A new paradigm of visual analysis is presented and tested with the goal of addressing this persistent limitation. In traditional visual analysis, graphs are viewed and rated one by one. However, in the ranking and pairwise comparison methods introduced here, graphs are compared to each other and sorted from least to most evidence of intervention effectiveness. Four visual raters scored a set of 30 previously published single-case graphs using a traditional rating method as well as the ranking and pairwise methods. As in previous studies of visual analysis, the raters failed to achieve acceptable interrater reliability with the traditional rating approach (α = 0.641). However, interrater reliability increased to satisfactory levels when graphs were scored with ranking (α = 0.847) and pairwise comparison (α = 0.860). Visual analysis scores based on the pairwise method were also used to evaluate the performance of three single-case effect size statistics.

Keywords

• single-case
• single-subject
• visual analysis
• interrater reliability
• effect size
• elo’s system