Evaluating the role of ChatGPT in enhancing EFL writing assessments in classroom settings: A preliminary investigation

Abstract

Abstract Using generalizability (G-) theory and qualitative feedback analysis, this study evaluated the role of ChatGPT in enhancing English-as-a-foreign-language (EFL) writing assessments in classroom settings. The primary objectives were to assess the reliability of the holistic scores assigned to EFL essays by ChatGPT versions 3.5 and 4 compared to college English teachers and to evaluate the relevance of the qualitative feedback provided by these versions of ChatGPT. The study analyzed 30 College English Test Band 4 (CET-4) essays written by non-English majors at a university in Beijing, China. ChatGPT versions 3.5 and 4, along with four college English teachers, served as raters. They scored the essays holistically following the CET-4 scoring rubric and also provided qualitative feedback on the language, content, and organization of these essays. The G-theory analysis revealed that the scoring reliability of ChatGPT3.5 was consistently lower than that of the teacher raters; however, ChatGPT4 demonstrated consistently higher reliability coefficients than the teachers. The qualitative feedback analysis indicated that both ChatGPT3.5 and 4 consistently provided more relevant feedback on the EFL essays than the teacher raters. Furthermore, ChatGPT versions 3.5 and 4 were equally relevant across the language, content, and organization aspects of the essays, whereas the teacher raters generally focused more on language but provided less relevant feedback on content and organization. Consequently, ChatGPT versions 3.5 and 4 could be useful AI tools for enhancing EFL writing assessments in classroom settings. The implications of adopting ChatGPT for classroom writing assessments are discussed.