Assessing Graph Comprehension on Paper and Computer with MBA Students: A Crossover Experimental Study

Abstract

Several decades of research suggesting differences in test performance across paper-based and computer-based assessments have been largely ameliorated through attention to test presentation equivalence, though no studies to date have focused on graph comprehension items. Test items requiring graph comprehension are increasingly common but may be especially prone to format effects. A crossover experimental design was used to compare paper-based (PBT) with computer-based test (CBT) formats in a sample of 28 MBA students answering 6 items requiring comprehension of Venn diagrams, scatterplots, and divided bar charts. Data analysis using a 2x2x2x2 ANOVA revealed that participants’ better performance on CBT, η2 = .07, was not statistically significant, p = .23. While DIF analysis suggested no difference by gender in total score or individual items, interaction analysis between gender and format showed the overall format effect was due mainly to males performing better on CBT, p = .02, d = 0.91. Females performed about the same in both formats. For scatterplot questions, participants also performed better on CBT, p < .005, φ = 0.41. Finally, participants were more likely to answer an easier question requiring attention to fewer graph components correctly after answering a more challenging question that required attention to all graph components, p = .02, φ = 0.5. Interaction analysis also revealed a large carryover effect from the research design (η2 = .48, p = .000), which we interpreted as a learning effect.

Keywords

• assessment format
• graph comprehension
• gender differences
• item order effect
• crossover research design
• mba