The Impact of the “all-of-the-above” Option and Student Ability on Multiple Choice Tests

Abstract

Despite the prevalence of multiple choice items in educational testing, there is a dearth of empirical evidence for multiple choice item writing rules. The purpose of this study was to expand the base of empirical evidence by examining the use of the “all-of-the-above” option in a multiple choice examination in order to assess how different student ability groups would respond to this particular alternative. Ten experimentally manipulated items were generated with “all-of-the-above” as one of the options and were incorporated into three different test formats. Test formats were randomly distributed to university students in the study. The test scores in these test formats were compared as well as the experimentally manipulated items. Results showed that when “all-of-the-above” is used as the correct answer, the item is more difficult for all students, despite the literature assumption that it provides a cueing effect to students. Research findings corroborate literature assumptions that high ability students score significantly higher than other ability students in this type of option.

Keywords

• Scholarship of Teaching and Learning
• SoTL
• Multiple-choice items
• Multiple-choice questions
• MCQ
• All-of-the-above option
• AOTA
• None-of-the-above option
• NOTA