External Validation and Comparison of Statistical and Machine Learning–Based Models in Predicting Outcomes Following Out‐of‐Hospital Cardiac Arrest: A Multicenter Retrospective Analysis

Abstract

Background The aim of this study was to validate and compare the performance of statistical (Utstein‐Based Return of Spontaneous Circulation and Shockable Rhythm–Witness–Age–pH) and machine learning–based (Prehospital Return of Spontaneous Circulation and Swedish Cardiac Arrest Risk Score) models in predicting the outcomes following out‐of‐hospital cardiac arrest and to assess the impact of the COVID‐19 pandemic on the models' performance. Methods and Results This retrospective analysis included adult patients with out‐of‐hospital cardiac arrest treated at 3 academic hospitals between 2015 and 2023. The primary outcome was neurological outcomes at hospital discharge. Patients were divided into pre‐ (2015–2019) and post‐2020 (2020–2023) subgroups to examine the effect of the COVID‐19 pandemic on out‐of‐hospital cardiac arrest outcome prediction. The models' performance was evaluated using the area under the receiver operating characteristic curve and compared by the DeLong test. The analysis included 2161 patients, 1241 (57.4%) of whom were resuscitated after 2020. The cohort had a median age of 69.2 years, and 1399 patients (64.7%) were men. Overall, 69 patients (3.2%) had neurologically intact survival. The area under the receiver operating characteristic curves for predicting neurological outcomes were 0.85 (95% CI, 0.83–0.87) for the Utstein‐Based Return of Spontaneous Circulation score, 0.82 (95% CI, 0.81–0.84) for the Shockable Rhythm–Witness–Age–pH score, 0.79 (95% CI, 0.78–0.81) for the Prehospital Return of Spontaneous Circulation score, and 0.79 (95% CI, 0.77–0.81) for the Swedish Cardiac Arrest Risk Score model. The Utstein‐Based Return of Spontaneous Circulation score significantly outperformed both the Prehospital Return of Spontaneous Circulation score (P<0.001) and the Swedish Cardiac Arrest Risk Score model (P=0.007). Subgroup analysis indicated no significant difference in predictive performance for patients resuscitated before versus after 2020. Conclusions In this external validation, both statistical and machine learning–based models demonstrated excellent and fair performance, respectively, in predicting neurological outcomes despite different model architectures. The predictive performance of all evaluated clinical scoring systems was not significantly influenced by the COVID‐19 pandemic.

Keywords

• cardiopulmonary resuscitation
• COVID‐19
• machine learning
• out‐of‐hospital cardiac arrest
• prediction model
• prognostication