Acute Achilles Tendon Ruptures: How Well Can Artificial Intelligence Chatbots Answer Patient Inquiries?

Abstract

Category: Sports; Trauma Introduction/Purpose: Artificial intelligence (AI) chatbots have recently gained popularity as a source of information that can be easily accessed by patients given their human-like responses to prompts and questions. Within orthopaedics, the treatment of acute Achilles tendon ruptures is not uniform due to varying surgical repair techniques, postoperative protocols, and nonoperative treatment options dependent on surgeon preference and patient factors. Given that patients are increasingly turning toward AI for questions about medical diagnoses and treatment options, our study looked to compare the adequacy of AI chatbot responses to frequently asked questions regarding acute Achilles tendon ruptures. Methods: Three popular AI platforms (ChatGPT, Google Gemini, and Microsoft Bing AI) were prompted for a concise response to ten commonly asked questions regarding Achilles tendon rupture management (Table 1). Four board-certified subspecialty-trained orthopaedic surgeons (two in foot and ankle, two in sports medicine) were asked to assess the value of the AI response using a four-point scale (1 – satisfactory; 2 – satisfactory requiring minimal clarification; 3 – satisfactory requiring substantial clarification; 4 – unsatisfactory). A Kruskal-Wallis test was used to compare the responses between the three AI platforms using the scores assigned by the surgeons. Results: All three AI chatbots provided comparable answers to 7 of 10 questions (70%). Of all the responses (30 total), only two (6.7%) had a mean rating of 3 or higher. Significant differences were noted between the AI systems for questions 4 [H(2) = 7.258, p = .027], 7 [H(2) = 6.308, p = .043], and 10 [H(2) = 6.796, p = .033]. Post hoc analyses revealed Bing AI had significantly worse scores as compared to ChatGPT for all three of these questions. Conclusion: AI chatbots can appropriately answer concise prompts about the diagnosis and management of acute Achilles tendon ruptures often sought out by patients prior to or after evaluation by an orthopaedic surgeon. The responses provided by the three AI chatbots analyzed in our study were uniform and satisfactory, with only one of the platforms scoring worse on three of the ten questions. As AI chatbots advance, they will become a valuable tool for patient education in orthopaedics. Future studies will be needed to assess performance as new AI chatbots develop and large language models continue to evolve. Table 1: List of 10 selected frequently asked questions regarding acute Achilles tendon ruptures