Investigating rater–student interaction, gender bias, and major bias in the assessment of research seminar presentation

Abstract

The study aims to investigate the severity of raters toward examinees' performances and to analyze biases subject to genders and academic majors based on the rater–examinee interactions. Data were collected across a 13-week research seminar course in English. Six raters were selected to rate 33 examinees using 18-item criteria. The many-facet Rasch measurement was utilized to analyze rater–examinee interactions and bias factors. The results confirmed that the instrument is reliable and valid. This study depicted the interaction between raters and other facets using a variable map, where the raters exhibited different levels of severity/leniency in scoring students' performances in oral presentations at the research seminar course. The result based on the Rasch analysis also confirmed that gender and academic majors contaminated rater assessment. Bias interaction between raters and student gender was detected, and Rater 6 displayed bias based on gender due to the tendency to give higher scores to female than male participants with a target contrast of 2.05 logits. Bias interaction between rater academic majors and student academic majors was also identified among raters with linguistics and psychology majors.

Keywords

• Gender bias
• Major bias
• Rater–Student interaction
• Many-facet Rasch measurement
• Assessment