Critical Care Explorations (Jan 2020)

The Effect of Outcome Selection on the Performance of Prediction Models in Patients at Risk for Sepsis

  • Stephanie P. Taylor, MD, MS,
  • Shih-Hsiung Chou, PhD,
  • Andrew D. McWilliams, MD, MPH,
  • Mark Russo, MD,
  • Alan C. Heffner, MD,
  • Stephanie Murphy, DO,
  • Susan L. Evans, MD, FACS, FCCM,
  • Whitney Rossman, MS,
  • Marc Kowalkowski, PhD,
  • on behalf of Acute Care Outcomes Research Network (ACORN) Investigators,
  • Ryan Brown,
  • Larry Burke,
  • Shih-Hsiung Chou,
  • Kyle Cunningham,
  • Susan L. Evans,
  • Scott Furney,
  • Michael Gibbs,
  • Alan Heffner,
  • Timothy Hetherington,
  • Daniel Howard,
  • Marc Kowalkowski,
  • Scott Lindblom,
  • Andrea McCall,
  • Lewis McCurdy,
  • Andrew McWilliams,
  • Stephanie Murphy,
  • Alfred Papali,
  • Christopher Polk,
  • Whitney Rossman,
  • Michael Runyon,
  • Mark Russo,
  • Melanie Spencer,
  • Brice Taylor,
  • Stephanie Taylor

DOI
https://doi.org/10.1097/CCE.0000000000000078
Journal volume & issue
Vol. 2, no. 1
p. e0078

Abstract

Read online

Importance:. Risk prediction models for patients with suspected sepsis have been derived on and applied to various outcomes, including readily available outcomes such as hospital mortality and ICU admission as well as longer-term mortality outcomes that may be more important to patients. It is unknown how selecting different outcomes influences model performance in patients at risk for sepsis. Objectives:. Evaluate the impact of outcome selection on risk model performance and weighting of individual predictor variables. Design, Setting, and Participants:. We retrospectively analyzed adults hospitalized with suspected infection from January 2014 to September 2017 at 12 hospitals. Main Outcomes and Measures:. We used routinely collected clinical data to derive logistic regression models for four outcomes: hospital mortality, composite ICU length of stay greater than 72 hours or hospital mortality, 30-day mortality, and 90-day mortality. We compared the performance of the models using area under the receiver operating characteristic curve and calibration plots. Results:. Among 52,184 admissions, 2,030 (4%) experienced hospital mortality, 6,659 (13%) experienced the composite of hospital mortality or ICU length of stay greater than 72 hours, 3,417 (7%) experienced 30-day mortality, and 5,655 (11%) experienced 90-day mortality. Area under the receiver operating characteristic curves decreased when hospital-based models were applied to predict 30-day (hospital mortality = 0.88–0.85; –0.03, composite ICU length of stay greater than 72 hours or hospital mortality = 0.90–0.81; –0.09) and 90-day mortality (hospital mortality = 0.88–0.81; –0.07, composite ICU length of stay greater than 72 hours or hospital mortality = 0.90–0.76; –0.14; all p < 0.01). Models were well calibrated for derived (root-mean-square error = 5–15) but not alternate outcomes (root-mean-square error = 8–35). Conclusions and Relevance:. Risk models trained to predict readily available hospital-based outcomes in suspected sepsis show poorer discrimination and calibration when applied to 30- and 90-day mortality. Interpretation and application of risk models for patients at risk of sepsis should consider these findings.