Assessment of the Reliability and Clinical Applicability of ChatGPT’s Responses to Patients’ Common Queries About Rosacea

Abstract

Sihan Yan,1,2,* Dan Du,1,2,* Xu Liu,1,2 Yingying Dai,1,2 Min-Kyu Kim,1,2 Xinyu Zhou,3 Lian Wang,1,2 Lu Zhang,1,2 Xian Jiang1,2 1Department of Dermatology, West China Hospital, Sichuan University, Chengdu, People’s Republic of China; 2Laboratory of Dermatology, Clinical Institute of Inflammation and Immunology, Frontiers Science Center for Disease-Related Molecular Network, West China Hospital, Sichuan University, Chengdu, People’s Republic of China; 3Department of Dermatology, Nanbu County People’s Hospital, Nanbu County, Nanchong, Sichuan, People’s Republic of China*These authors contributed equally to this workCorrespondence: Xian Jiang; Lu Zhang, Email [email protected]; [email protected]: Artificial intelligence chatbot, particularly ChatGPT (Chat Generative Pre-trained Transformer), is capable of analyzing human input and generating human-like responses, which shows its potential application in healthcare. People with rosacea often have questions about alleviating symptoms and daily skin-care, which is suitable for ChatGPT to response. This study aims to assess the reliability and clinical applicability of ChatGPT 3.5 in responding to patients’ common queries about rosacea and to evaluate the extent of ChatGPT’s coverage in dermatology resources.Methods: Based on a qualitative analysis of the literature on the queries from rosacea patients, we have extracted 20 questions of patients’ greatest concerns, covering four main categories: treatment, triggers and diet, skincare, and special manifestations of rosacea. Each question was inputted into ChatGPT separately for three rounds of question-and-answer conversations. The generated answers will be evaluated by three experienced dermatologists with postgraduate degrees and over five years of clinical experience in dermatology, to assess their reliability and applicability for clinical practice.Results: The analysis results indicate that the reviewers unanimously agreed that ChatGPT achieved a high reliability of 92.22% to 97.78% in responding to patients’ common queries about rosacea. Additionally, almost all answers were applicable for supporting rosacea patient education, with a clinical applicability ranging from 98.61% to 100.00%. The consistency of the expert ratings was excellent (all significance levels were less than 0.05), with a consistency coefficient of 0.404 for content reliability and 0.456 for clinical practicality, indicating significant consistency in the results and a high level of agreement among the expert ratings.Conclusion: ChatGPT 3.5 exhibits excellent reliability and clinical applicability in responding to patients’ common queries about rosacea. This artificial intelligence tool is applicable for supporting rosacea patient education.Keywords: artificial intelligence, ChatGPT, rosacea, patient education

Keywords

• artificial intelligence
• chatgpt
• rosacea
• patient education