Assessing the Performance of ChatGPT in Medical Toxicology Through Simulated Case Scenarios

Abstract

Objective: The insufficient number of medical toxicologists and poison information centers worldwide limits the accessibility of adequate medical recommendations for the management of poisoned patients. This study aimed to assess the effectiveness of Chat Generative Pretrained Transformers (GPTs) medical recommendations in medical toxicology and evaluate its accuracy as a valuable resource when accessing medical toxicologists or poison information centers is limited. Materials and Methods: A toxicologist created 10 different toxicology-simulated case scenarios based on the possible presentations of poisoned patients in an emergency department setting. The categories of general approach and stabilization, diagnostic activities, and medical treatments and follow-up were used to measure case assessment and ChatGPT’s medical recommendation capacity. Results: ChatGPT-4o achieved an average success rate of 90.88% across the simulated case scenarios. ChatGPT-4o received a passing grade in 9 cases (90%) and received “improvable” in only 1 case (10%). ChatGPT-4o’s average success rate in all categories and across all cases increased from 90.88% to 97.22% with the secondary test. Conclusion: Our study indicates that it is possible to improve the success rate of ChatGPT in providing medical toxicology recommendations. The ability to query current medical toxicology information through ChatGPT-4o demonstrates the potential of ChatGPT to serve as a next-generation poison information center.

Keywords

• artificial intelligence (ai)
• chatgpt-4o
• clinical decision support systems
• generative pretrained transformer
• poison control center
• toxicology