Accuracy of ChatGPT3.5 in answering clinical questions on guidelines for severe acute pancreatitis

Abstract

Abstract Background Guidelines must be interpreted comprehensively and correctly to standardize the clinical process. However, this process is challenging and requires interpreters to have a medical background and qualifications. In this study, the accuracy of ChatGPT3.5 in answering clinical questions related to the 2019 guidelines for severe acute pancreatitis was evaluated. Methods and results An observational study was conducted using the 2019 guidelines for severe acute pancreatitis. The study compared the accuracy of ChatGPT3.5 in English versus Chinese and found that it was more accurate in English (71%) than in Chinese (59%) (P value: 0.203). Additionally, the study assessed the accuracy of ChatGPT3.5 in answering short-answer questions versus true/false questions and found that it was more accurate in answering short-answer questions (76%) than in answering true/false questions (60%) (P value: 0.405). Conclusions For clinicians managing severe acute pancreatitis, ChatGPT3.5 may have potential value. However, it should not be relied upon excessively for clinical decision making.

Keywords

• ChatGPT
• Severe acute pancreatitis
• Guidelines