Evaluation of ChatGPT-4 in detecting referable diabetic retinopathy using single fundus images

Abstract

Purpose: Evaluate ChatGPT-4′s ability to identify referable diabetic retinopathy (DR) from single fundus images. Design: A cross-sectional study comparing ChatGPT-4′s versus retina specialists’ identification of more than mild DR (mtmDR) and vision-threatening DR (VTDR). Methods: Images in equal proportions of normal, mild, moderate, and severe nonproliferative DR (NPDR), proliferative DR (PDR), and blurry images with and without suspected PDR were presented to a panel of blinded retina specialists who identified images as readable or unreadable, and potentially as mtmDR or VTDR. These images were also submitted to ChatGPT-4 three times with a standardized prompt regarding mtmDR and VTDR. Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were calculated for ChatGPT-4′s responses regarding mtmDR and VTDR as compared to the retina specialists majority determination. Results: Retina specialists read 158/180 prompts (87.7 %) with excellent interrater reliability while ChatGPT-4 read 132/180 (73.33 %) of the image prompts. For mtmDR, ChatGPT-4 demonstrated a sensitivity of 96.2 %, specificity of 19.1 %, PPV of 69.1 %, and NPV of 72.7 %. Overall, 90.9 % of prompts read by ChatGPT-4 were labeled as mtmDR. For VTDR, ChatGPT-4 demonstrated a 63.0 % sensitivity, 62.5 % specificity, 71.9 % PPV, and 52.6 % NPV compared to retina specialists. ChatGPT-4 labeled 51.5 % of read images as VTDR. Overall referability was 66.6 % for retina specialists and 93.3 % for ChatGPT-4. Conclusion: While ChatGPT-4 demonstrates promise in identifying moderate-to-severe DR, its limited specificity and tendency to overcall disease reduce its current utility as a screening tool.

Keywords

• Artificial intelligence
• Diabetic retinopathy
• Large language model
• Telemedicine