Artificial Intelligence-Human Hybrid Workflow Enhances Teleophthalmology for the Detection of Diabetic Retinopathy

Eliot R. Dow, MD, PhD; Nergis C. Khan, BS; Karen M. Chen, BS; Kapil Mishra, MD; Chandrashan Perera, MD; Ramsudha Narala, MD; Marina Basina, MD; Jimmy Dang, BSN; Michael Kim, MD; Marcie Levine, MD; Anuradha Phadke, MD; Marilyn Tan, MD; Kirsti Weng, MD; Diana V. Do, MD; Darius M. Moshfeghi, MD; Vinit B. Mahajan, MD, PhD; Prithvi Mruthyunjaya, MD, MHS; Theodore Leng, MD, MS; David Myung, MD, PhD

Ophthalmology Science (Dec 2023)

Artificial Intelligence-Human Hybrid Workflow Enhances Teleophthalmology for the Detection of Diabetic Retinopathy

Eliot R. Dow, MD, PhD,
Nergis C. Khan, BS,
Karen M. Chen, BS,
Kapil Mishra, MD,
Chandrashan Perera, MD,
Ramsudha Narala, MD,
Marina Basina, MD,
Jimmy Dang, BSN,
Michael Kim, MD,
Marcie Levine, MD,
Anuradha Phadke, MD,
Marilyn Tan, MD,
Kirsti Weng, MD,
Diana V. Do, MD,
Darius M. Moshfeghi, MD,
Vinit B. Mahajan, MD, PhD,
Prithvi Mruthyunjaya, MD, MHS,
Theodore Leng, MD, MS,
David Myung, MD, PhD

Affiliations

Eliot R. Dow, MD, PhD: Byers Eye Institute at Stanford, Stanford University School of Medicine, Palo Alto, California; Veterans Affairs Palo Alto Health Care System, Palo Alto, California
Nergis C. Khan, BS: Byers Eye Institute at Stanford, Stanford University School of Medicine, Palo Alto, California
Karen M. Chen, BS: Byers Eye Institute at Stanford, Stanford University School of Medicine, Palo Alto, California
Kapil Mishra, MD: Byers Eye Institute at Stanford, Stanford University School of Medicine, Palo Alto, California
Chandrashan Perera, MD: Byers Eye Institute at Stanford, Stanford University School of Medicine, Palo Alto, California
Ramsudha Narala, MD: Byers Eye Institute at Stanford, Stanford University School of Medicine, Palo Alto, California
Marina Basina, MD: Stanford Healthcare, Stanford University, Palo Alto, California
Jimmy Dang, BSN: Stanford Healthcare, Stanford University, Palo Alto, California
Michael Kim, MD: Stanford Healthcare, Stanford University, Palo Alto, California
Marcie Levine, MD: Stanford Healthcare, Stanford University, Palo Alto, California
Anuradha Phadke, MD: Stanford Healthcare, Stanford University, Palo Alto, California
Marilyn Tan, MD: Stanford Healthcare, Stanford University, Palo Alto, California
Kirsti Weng, MD: Stanford Healthcare, Stanford University, Palo Alto, California
Diana V. Do, MD: Byers Eye Institute at Stanford, Stanford University School of Medicine, Palo Alto, California
Darius M. Moshfeghi, MD: Byers Eye Institute at Stanford, Stanford University School of Medicine, Palo Alto, California
Vinit B. Mahajan, MD, PhD: Byers Eye Institute at Stanford, Stanford University School of Medicine, Palo Alto, California; Veterans Affairs Palo Alto Health Care System, Palo Alto, California
Prithvi Mruthyunjaya, MD, MHS: Byers Eye Institute at Stanford, Stanford University School of Medicine, Palo Alto, California
Theodore Leng, MD, MS: Byers Eye Institute at Stanford, Stanford University School of Medicine, Palo Alto, California
David Myung, MD, PhD: Byers Eye Institute at Stanford, Stanford University School of Medicine, Palo Alto, California; Correspondence: David Myung, MD, PhD, Byers Eye Institute at Stanford, Stanford University School of Medicine, Palo Alto, CA 94303.

Journal volume & issue: Vol. 3, no. 4
p. 100330

Abstract

Read online

Objective: Detection of diabetic retinopathy (DR) outside of specialized eye care settings is an important means of access to vision-preserving health maintenance. Remote interpretation of fundus photographs acquired in a primary care or other nonophthalmic setting in a store-and-forward manner is a predominant paradigm of teleophthalmology screening programs. Artificial intelligence (AI)-based image interpretation offers an alternative means of DR detection. IDx-DR (Digital Diagnostics Inc) is a Food and Drug Administration-authorized autonomous testing device for DR. We evaluated the diagnostic performance of IDx-DR compared with human-based teleophthalmology over 2 and a half years. Additionally, we evaluated an AI-human hybrid workflow that combines AI-system evaluation with human expert-based assessment for referable cases. Design: Prospective cohort study and retrospective analysis. Participants: Diabetic patients ≥ 18 years old without a prior DR diagnosis or DR examination in the past year presenting for routine DR screening in a primary care clinic. Methods: Macula-centered and optic nerve-centered fundus photographs were evaluated by an AI algorithm followed by consensus-based overreading by retina specialists at the Stanford Ophthalmic Reading Center. Detection of more-than-mild diabetic retinopathy (MTMDR) was compared with in-person examination by a retina specialist. Main Outcome Measures: Sensitivity, specificity, accuracy, positive predictive value, and gradability achieved by the AI algorithm and retina specialists. Results: The AI algorithm had higher sensitivity (95.5% sensitivity; 95% confidence interval [CI], 86.7%–100%) but lower specificity (60.3% specificity; 95% CI, 47.7%–72.9%) for detection of MTMDR compared with remote image interpretation by retina specialists (69.5% sensitivity; 95% CI, 50.7%–88.3%; 96.9% specificity; 95% CI, 93.5%–100%). Gradability of encounters was also lower for the AI algorithm (62.5%) compared with retina specialists (93.1%). A 2-step AI-human hybrid workflow in which the AI algorithm initially rendered an assessment followed by overread by a retina specialist of MTMDR-positive encounters resulted in a sensitivity of 95.5% (95% CI, 86.7%–100%) and a specificity of 98.2% (95% CI, 94.6%–100%). Similarly, a 2-step overread by retina specialists of AI-ungradable encounters improved gradability from 63.5% to 95.6% of encounters. Conclusions: Implementation of an AI-human hybrid teleophthalmology workflow may both decrease reliance on human specialist effort and improve diagnostic accuracy. Financial Disclosure(s): Proprietary or commercial disclosure may be found in the Footnotes and Disclosures at the end of this article.

Published in Ophthalmology Science

ISSN: 2666-9145 (Online)
Publisher: Elsevier
Country of publisher: United States
LCC subjects: Medicine: Ophthalmology
Website: https://www.journals.elsevier.com/ophthalmology-science/

About the journal

Abstract

Keywords