Heliyon (Nov 2024)
Construct validity of primary trait writing rubrics based on assessment use argument (AUA) validation framework
Abstract
In assessing performance-based language assessment, the use of a valid and reliable scoring rubric is crucial to minimize measurement errors that become threats in the rating process. The validation process of rubrics which previously was based on qualitative data is not satisfying since empirical evidence is not present. Thus, drawing on the Assessment Use Argument (AUA) Validation Framework, this study aims to search for evidence to prove a claim which is Primary Trait Writing (PTW) rubrics for students self assessment activities are relevant to the construct being measured. Based on that, two warrants and one rebuttal were derived to prove the claim. The participants consisted of 149 secondary school students in a state in Malaysia and three facets identified in the study were the examinee (149 students), the rater (149 students and 3 teachers), and the trait (Content, Format, Cohesive Device, and Sentence Fluency). Many Facet Rasch Model was employed to search for evidence in approving the warrants and rejecting the rebuttal. Based on the statistical results, evidence has shown that PTW rubrics successfully discriminated between students' writing ability, and fulfilled the six basic conditions of rating scale effectiveness to a certain extent where the Cohesive Device trait became the primary concern. In addition, the fit statistics for all traits demonstrated internal consistency and the high-reliability index portrayed the criteria were well differentiated in terms of difficulty level. Thus, all evidence had shown that PTW rubrics obtained construct validity where the warrants were supported, and the rebuttal was rejected which led to the acceptance of the claim. The implication of this study highlights the importance of validating assessment rubrics to ensure their internal validity and the use of MRFM in providing comprehensive analyses of the strengths and weaknesses of the developed rubrics. The use of a primary trait type scoring rubric as a tool in students’ self-assessment activities must be highlighted as more current studies are focusing on holistic and analytic scoring. Therefore, it is suggested for future research to expand the use of the Primary Trait rubrics in other type of essays and to be used in peer-assessment activities. Therefore, it is suggested for future research to expand the use of the Primary Trait rubrics in other types of essay and to be used in peer-assessment activities.
