npj Digital Medicine (Jan 2025)
Multimodal machine learning enables AI chatbot to diagnose ophthalmic diseases and provide high-quality medical responses
- Ruiqi Ma,
- Qian Cheng,
- Jing Yao,
- Zhiyu Peng,
- Mingxu Yan,
- Jie Lu,
- Jingjing Liao,
- Lejin Tian,
- Wenjun Shu,
- Yunqiu Zhang,
- Jinghan Wang,
- Pengfei Jiang,
- Weiyi Xia,
- Xiaofeng Li,
- Lu Gan,
- Yue Zhao,
- Jiang Zhu,
- Bing Qin,
- Qin Jiang,
- Xiawei Wang,
- Xintong Lin,
- Haifeng Chen,
- Weifang Zhu,
- Dehui Xiang,
- Baoqing Nie,
- Jingtao Wang,
- Jie Guo,
- Kang Xue,
- Hongguang Cui,
- Jinwei Cheng,
- Xiangjia Zhu,
- Jiaxu Hong,
- Fei Shi,
- Rui Zhang,
- Xinjian Chen,
- Chen Zhao
Affiliations
- Ruiqi Ma
- Eye Institute and Department of Ophthalmology, Eye & ENT Hospital, Fudan University
- Qian Cheng
- MIPAV Lab, School of Electronics and Information Engineering, Soochow University
- Jing Yao
- Eye Institute and Department of Ophthalmology, Eye & ENT Hospital, Fudan University
- Zhiyu Peng
- Eye Institute and Department of Ophthalmology, Eye & ENT Hospital, Fudan University
- Mingxu Yan
- Eye Institute and Department of Ophthalmology, Eye & ENT Hospital, Fudan University
- Jie Lu
- Eye Institute and Department of Ophthalmology, Eye & ENT Hospital, Fudan University
- Jingjing Liao
- MIPAV Lab, School of Electronics and Information Engineering, Soochow University
- Lejin Tian
- State Key Laboratory of Genetic Engineering, Department of Computational Biology, School of Life Sciences, Fudan University
- Wenjun Shu
- Eye Institute and Department of Ophthalmology, Eye & ENT Hospital, Fudan University
- Yunqiu Zhang
- Department of Epidemiology, School of Public Health, and The Key Laboratory of Public Health Safety of Ministry of Education, Fudan University
- Jinghan Wang
- Eye Institute and Department of Ophthalmology, Eye & ENT Hospital, Fudan University
- Pengfei Jiang
- Shanghai Jiao Tong University Instrument Analysis Center
- Weiyi Xia
- Eye Institute and Department of Ophthalmology, Eye & ENT Hospital, Fudan University
- Xiaofeng Li
- Eye Institute and Department of Ophthalmology, Eye & ENT Hospital, Fudan University
- Lu Gan
- Eye Institute and Department of Ophthalmology, Eye & ENT Hospital, Fudan University
- Yue Zhao
- The Affiliated Eye Hospital, Nanjing Medical University
- Jiang Zhu
- Department of Ophthalmology, Suqian First Hospital
- Bing Qin
- Department of Ophthalmology, Suqian First Hospital
- Qin Jiang
- The Affiliated Eye Hospital, Nanjing Medical University
- Xiawei Wang
- Department of Ophthalmology, The First Affiliated Hospital, Zhejiang University School of Medicine
- Xintong Lin
- Eye Institute and Department of Ophthalmology, Eye & ENT Hospital, Fudan University
- Haifeng Chen
- Eye Institute and Department of Ophthalmology, Eye & ENT Hospital, Fudan University
- Weifang Zhu
- MIPAV Lab, School of Electronics and Information Engineering, Soochow University
- Dehui Xiang
- MIPAV Lab, School of Electronics and Information Engineering, Soochow University
- Baoqing Nie
- MIPAV Lab, School of Electronics and Information Engineering, Soochow University
- Jingtao Wang
- MIPAV Lab, School of Electronics and Information Engineering, Soochow University
- Jie Guo
- Eye Institute and Department of Ophthalmology, Eye & ENT Hospital, Fudan University
- Kang Xue
- Eye Institute and Department of Ophthalmology, Eye & ENT Hospital, Fudan University
- Hongguang Cui
- Department of Ophthalmology, The First Affiliated Hospital, Zhejiang University School of Medicine
- Jinwei Cheng
- Eye Institute and Department of Ophthalmology, Eye & ENT Hospital, Fudan University
- Xiangjia Zhu
- Eye Institute and Department of Ophthalmology, Eye & ENT Hospital, Fudan University
- Jiaxu Hong
- Eye Institute and Department of Ophthalmology, Eye & ENT Hospital, Fudan University
- Fei Shi
- MIPAV Lab, School of Electronics and Information Engineering, Soochow University
- Rui Zhang
- Eye Institute and Department of Ophthalmology, Eye & ENT Hospital, Fudan University
- Xinjian Chen
- MIPAV Lab, School of Electronics and Information Engineering, Soochow University
- Chen Zhao
- Eye Institute and Department of Ophthalmology, Eye & ENT Hospital, Fudan University
- DOI
- https://doi.org/10.1038/s41746-025-01461-0
- Journal volume & issue
-
Vol. 8,
no. 1
pp. 1 – 18
Abstract
Abstract Chatbot-based multimodal AI holds promise for collecting medical histories and diagnosing ophthalmic diseases using textual and imaging data. This study developed and evaluated the ChatGPT-powered Intelligent Ophthalmic Multimodal Interactive Diagnostic System (IOMIDS) to enable patient self-diagnosis and self-triage. IOMIDS included a text model and three multimodal models (text + slit-lamp, text + smartphone, text + slit-lamp + smartphone). The performance was evaluated through a two-stage cross-sectional study across three medical centers involving 10 subspecialties and 50 diseases. Using 15640 data entries, IOMIDS actively collected and analyzed medical history alongside slit-lamp and/or smartphone images. The text + smartphone model showed the highest diagnostic accuracy (internal: 79.6%, external: 81.1%), while other multimodal models underperformed or matched the text model (internal: 69.6%, external: 72.5%). Moreover, triage accuracy was consistent across models. Multimodal approaches enhanced response quality and reduced misinformation. This proof-of-concept study highlights the potential of chatbot-based multimodal AI for self-diagnosis and self-triage. (The clinical trial was registered on June 26, 2023, on ClinicalTrials.gov under the registration number NCT05930444.).
