Expert-Authored and Machine-Generated Short-Answer Questions for Assessing Students’ Learning Performance

Abstract

Human-guided machine learning can improve computing intelligence, and it can accurately assist humans in various tasks. In education research, artificial intelligence (AI) is applicable in many situations, such as predicting students’ learning paths and strategies. In this study, we explore the benefits of repetitive practice of short-answer questions could enhance students’ long-term memory for subsequent improvements in learning performance. However, frequent authoring questions and grading requires teachers’ professionalism, experience, and considerable efforts. Therefore, this study using modern AI technologies, specifically natural language processing, to provide Automatic question generation (AQG) solution, a combined semantics-based and syntax-based question generation system: Hybrid automatic question generation (Hybrid-AQG) was proposed in this study. We assessed its functionality and student learning performance by asking 91 students to complete short-answer questions and then applied a process similar to the Turing test to evaluate the question and grading quality. The results demonstrated that modern AI technologies can generate highly realistic short-answer questions because: (1) Compared with the control group, the experimental group exhibited significantly better learning performance, implying that students acquired long-term memory of course knowledge through repetitive practice with machine-generated questioning. (2) The experimental group could better distinguish machine-generated and expert-authored questions. Nevertheless, both groups in distinguishing questions presented like guessing. (3) Machine grading was deficient in some respects; but the way students answer questions can be adapted for machine understanding through repetitive practice.

Keywords

• automatic question generation
• learning performance
• artificial intelligence
• practice testing
• turing test