Improving Peak Shift Estimation to Rank Exams by Difficulty

Abstract

Candidates and other stakeholders often wish to know the difficulty level of exams, particularly compared with other exams in the same domain. Peak Shift Estimation (PSE) is a method for estimating the comparative difficulty of exams. However, this method requires users to correctly identify the most difficult exam in advance, which is not always possible. Therefore, we aimed to develop an improved version of PSE to overcome this limitation. Our new algorithm involves two key improvements. First, our method overcomes the issue described above using each exam as the most difficult in turn, and comparing the results. The second improvement addresses a potential problem which the first improvement, caused by the large number of exams of moderate difficulty. Instead of using all exams, our method randomly selects a subset of exams to remove this potential problem. We validated our new algorithm using data for the number of successful applicants at different universities from high schools in the Tokyo metropolitan area. The Improved PSE method correctly estimated the comparative difficulty of university entrance exams without prior knowledge of the most difficult exam. Few previous studies have considered selectively omitted exam data. The current findings should encourage other researchers to use these data in future studies.

Keywords

• Examination
• IRT
• peak shift estimation
• ranking
• selectively omitted exam data