Smart grading: A generative AI-based tool for knowledge-grounded answer evaluation in educational assessments

Abstract

Evaluating text-based answers obtained in educational settings or behavioral studies is time-consuming and resource-intensive. Applying novel artificial intelligence tools such as ChatGPT might support the process. Still, currently available implementations do not allow for automated and case-specific evaluations of large numbers of student answers. To counter this limitation, we developed a flexible software and user-friendly web application that enables researchers and educators to use cutting-edge artificial intelligence technologies by providing an interface that combines large language models with options to specify questions of interest, sample solutions, and evaluation instructions for automated answer scoring. We validated the method in an empirical study and found the software with expert ratings to have high reliability. Hence, the present software constitutes a valuable tool to facilitate and enhance text-based answer evaluation. • Generative AI-enhanced software for customizable, case-specific, and automized grading of large amounts of text-based answers. • Open-source software and web application for direct implementation and adaptation.

Keywords

• Artificial intelligence
• Test evaluation
• Educational assessment
• Automated grading
• GPT
• Large language model