To Evaluate the Efficiency of ChatGPT in Medical Education: An Analysis of MCQ-Based Learning and Assessment

Abstract

Objective: This study aimed to evaluate the potential of ChatGPT to help students for their assess ments via MCQ at different level of cognition by using different subjects of Internal medicine. Methods: This cross-sectional study was conducted in the Department of Internal Medicine in col laboration with post graduate medical education department from June 2023 to August 2023. An MCQ bank was established from three books of MCQ’s on subject of Internal Medicine. Total 1428 MCQ’s were followed for scrutiny and 307 MCQ’s were selected for the assigned task. The selected MCQ’s were manually entered one by one in a fresh Chat GPT session. The response was noted against the replies given in respective MCQ’s book and marked as correct, not correct or partially correct. MCQ’s were categorized as per chapters in Internal medicine and as per cognition level of MCQ’s i.e. C1, C2 and C3. Data was analyzed on SPSS version 21.00. Results: Chat GPT replied with 199 correct replies while 98 were wrong and 10 were partially correct. Chat GPT scored 64% overall in all categories. At level of cognition, it solved C2 MCQ’s by 80 % but scored 69% and 54% in C1 and C3 categories respectively. Chat GPT replied with 80% accuracy for C2 level MCQ’s while results were low for C3 category at around 54%. C1 also had low percentage of correct answers standing close to 69.8%. Almost all subjects showed healthy responses around the mean except for endocrinology and hematology where responses are below 60% and 40% respectively. Conclusion: This study findings suggest that ChatGPT is a useful tool for students and medical educationist with its current framework but a subtle approach should be inclined towards its role in future.

Keywords

• medical education