Google Gemini’s Performance in Endodontics: A Study on Answer Precision and Reliability

Abstract

(1) Background: Large language models (LLMs) are revolutionising various scientific fields by providing advanced support tools. However, the effectiveness of these applications depends on extensive, up-to-date databases to ensure certainty and predictive power. Transparency about information sources in Medicine remains a significant issue. (2) Methods: To evaluate Google Gemini’s accuracy and reproducibility in endodontic diagnosis and treatment, 60 questions were designed based on the European Society of Endodontology Position Statements. Thirty questions were randomly selected and answered using Gemini during April 2023. Two endodontic experts independently scored the answers using a 3-point Likert scale. Discrepancies were resolved by a third expert. The relative frequency and absolute percentage of responses were detailed. Accuracy was assessed using the Wald binomial method, and repeatability was assessed using percentage agreement, Brennan and Prediger’s coefficient, Conger’s generalised kappa, Fleiss’ kappa, Gwet’s AC, and Krippendorff’s alpha, all with 95% confidence intervals. Statistical analysis was performed using STATA software. (3) Results: A total of 900 answers were generated. The percentage of correct answers varied from 0% to 100% per question. Overall accuracy was 37.11% with a 95% confidence interval of 34.02–40.32%; (4) Conclusions: Gemini is not currently designed for medical use and therefore needs to be used with caution when considered for this purpose.

Keywords

• dentistry
• endodontics
• Google Gemini
• artificial intelligence
• chatbot
• large language models