Comparing Patient’s Confidence in Clinical Capabilities in Urology: Large Language Models Versus Urologists

Abstract

Background and objective: Data on interaction of patients with artificial intelligence (AI) are limited, primarily derived from small-scale studies, cross-sectional surveys, and qualitative reviews. Most patients have not yet encountered AI in their clinical experience. This study explored patients’ confidence in AI, specifically large language models, after a direct interaction with a chatbot in a clinical setting. Through hands-on experience, the study sought to reduce potential biases due to an anticipated lack of AI experience in a real-world urological patient sample. Methods: A total of 300 patients scheduled for counseling were enrolled from February to July 2024. Participants voluntarily conversed about their medical questions with a GPT-4 powered chatbot, followed by a survey assessing their confidence in clinical capabilities of AI compared with their counseling urologists. Clinical capabilities included history taking, diagnostics, treatment recommendation, anxiety reduction, and time allocation. Key findings and limitations: Of the 292 patients who completed the study, AI was significantly preferred to physicians for consultation time allocation (p < 0.001). However, urologists were overwhelmingly favored for all other capabilities, especially treatment recommendations and anxiety reduction. Notably, age did not influence patients’ confidence in AI. Limitations include a potential social desirability bias. Conclusions and clinical implications: Our study demonstrates that urological patients prefer AI as a powerful complement to—rather than a replacement for—human expertise in clinical care. Patients appreciated the additional consultation time provided by AI. Interestingly, age was not associated with confidence in AI, suggesting that large language models are user-friendly tools for patients of all age groups. Patient summary: In this report, we explored how patients feel about using an artificial intelligence (AI)-powered chatbot in a medical setting. Patients interacted with the AI for medical questions and compared its skills with those of doctors through a survey. They appreciated the AI for providing more time during consultations but preferred doctors for other tasks, for example, diagnostics, recommendation of treatments, and reduction of anxieties.

Keywords

• Clinical trial
• Generative artificial intelligence
• Implementation science
• Large language models
• Patient interaction