Diagnostic stewardship for blood cultures in the emergency department: A multicenter validation and prospective evaluation of a machine learning prediction tool
Michiel Schinkel,
Anneroos W. Boerman,
Frank C. Bennis,
Tanca C. Minderhoud,
Mei Lie,
Hessel Peters-Sengers,
Frits Holleman,
Rogier P. Schade,
Robert de Jonge,
W. Joost Wiersinga,
Prabath W.B. Nanayakkara
Affiliations
Michiel Schinkel
Section General Internal Medicine, Department of Internal Medicine, Amsterdam Public Health Research Institute, Amsterdam UMC, location VU University Medical Center, De Boelelaan 1118, 1081 HZ Amsterdam, the Netherlands; Center for Experimental and Molecular Medicine (CEMM), Amsterdam UMC, location Academic Medical Center, Meibergdreef 9, 1105 AZ Amsterdam, the Netherlands
Anneroos W. Boerman
Section General Internal Medicine, Department of Internal Medicine, Amsterdam Public Health Research Institute, Amsterdam UMC, location VU University Medical Center, De Boelelaan 1118, 1081 HZ Amsterdam, the Netherlands; Department of Clinical Chemistry, Amsterdam UMC, Vrije Universiteit Amsterdam, AGEM Research Institute, De Boelelaan 1118, 1081 HZ Amsterdam, the Netherlands
Frank C. Bennis
Department of Computer Science, Quantitative Data Analytics Group, Department of Computer Science, Faculty of Science, VU University, De Boelelaan 1105, 1081HV Amsterdam, the Netherlands
Tanca C. Minderhoud
Section General Internal Medicine, Department of Internal Medicine, Amsterdam Public Health Research Institute, Amsterdam UMC, location VU University Medical Center, De Boelelaan 1118, 1081 HZ Amsterdam, the Netherlands
Mei Lie
Department of EVA Service Center, Amsterdam UMC, location VU University Medical Center, De Boelelaan 1118, 1081 HZ Amsterdam, the Netherlands; Department of EVA Service Center, Amsterdam UMC, location Academic Medical Center, Meibergdreef 9, 1105 AZ Amsterdam, the Netherlands
Hessel Peters-Sengers
Center for Experimental and Molecular Medicine (CEMM), Amsterdam UMC, location Academic Medical Center, Meibergdreef 9, 1105 AZ Amsterdam, the Netherlands
Frits Holleman
Section General and Acute Internal Medicine, Department of Internal Medicine, Amsterdam UMC, location Academic Medical Center, Meibergdreef 9, 1105 AZ Amsterdam, the Netherlands
Rogier P. Schade
Department of Medical Microbiology and Infection Prevention, Amsterdam UMC, location Academic Medical Center, Meibergdreef 9, 1105 AZ Amsterdam, the Netherlands
Robert de Jonge
Department of Clinical Chemistry, Amsterdam UMC, Vrije Universiteit Amsterdam, AGEM Research Institute, De Boelelaan 1118, 1081 HZ Amsterdam, the Netherlands
W. Joost Wiersinga
Center for Experimental and Molecular Medicine (CEMM), Amsterdam UMC, location Academic Medical Center, Meibergdreef 9, 1105 AZ Amsterdam, the Netherlands; Section Infectious Diseases, Department of Internal Medicine, Amsterdam UMC, location Academic Medical Center, Meibergdreef 9, 1105 AZ Amsterdam, the Netherlands
Prabath W.B. Nanayakkara
Section General Internal Medicine, Department of Internal Medicine, Amsterdam Public Health Research Institute, Amsterdam UMC, location VU University Medical Center, De Boelelaan 1118, 1081 HZ Amsterdam, the Netherlands; Corresponding author.
Summary: Background: Overuse of blood cultures (BCs) in emergency departments (EDs) leads to low yields and high numbers of contaminated cultures, accompanied by increased diagnostics, antibiotic usage, prolonged hospitalization, and mortality. We aimed to simplify and validate a recently developed machine learning model to help safely withhold BC testing in low-risk patients. Methods: We extracted data from the electronic health records (EHR) for 44.123 unique ED visits with BC sampling in the Amsterdam UMC (locations VUMC and AMC; the Netherlands), Zaans Medical Center (ZMC; the Netherlands), and Beth Israel Deaconess Medical Center (BIDMC; United States) in periods between 2011 and 2021. We trained a machine learning model on the VUMC data to predict blood culture outcomes and validated it in the AMC, ZMC, and BIDMC with subsequent real-time prospective evaluation in the VUMC. Findings: The model had an Area Under the Receiver Operating Characteristics curve (AUROC) of 0.81 (95%-CI = 0.78–0.83) in the VUMC test set. The most important predictors were temperature, creatinine, and C-reactive protein. The AUROCs in the validation cohorts were 0.80 (AMC; 0.78–0.82), 0.76 (ZMC; 0.74–0.78), and 0.75 (BIDMC; 0.74–0.76). During real-time prospective evaluation in the EHR of the VUMC, it reached an AUROC of 0.76 (0.71–0.81) among 590 patients with BC draws in the ED. The prospective evaluation showed that the model can be used to safely withhold blood culture analyses in at least 30% of patients in the ED. Interpretation: We developed a machine learning model to predict blood culture outcomes in the ED, which retained its performance during external validation and real-time prospective evaluation. Our model can identify patients at low risk of having a positive blood culture. Using the model in practice can significantly reduce the number of blood culture analyses and thus avoid the hidden costs of false-positive culture results. Funding: This research project was funded by the Amsterdam Public Health – Quality of Care program and the Dutch “Doen of Laten” project (project number: 839205002).
