Demonstration of a Fair Level of Agreement Between Escalation Scores Reported by Hospital Managers and Analysis of Stress-Related Hospital Metrics

Abstract

Background: The National Health System in Wales has developed a novel national electronic dashboard which reports a daily “escalation score,” reflecting management’s opinion of the pressure each hospital is facing, primarily due to unscheduled care. The aim of this study was to examine the possibility of replacing human scores with a quantitative model, based on the relationship between reported escalation scores and selected hospital metrics. Methods: Generalized linear mixed models were used to model the association between hospital metrics and escalation scores between October one year and October the next year utilizing hospital bed occupancy rate, ambulance hours lost waiting outside emergency departments, number of “boarded out” patients in the hospital, and the daily ratio of admissions to discharges in the hospital. These models were tested against a subsequent period (December unto May the following year), using three models: “general,” “hospital-specific,” and “group-specific.” The model generated by the initial time frame was tested against data from the subsequent time frame using weighted κ. Results: Across 16 hospitals, using 3343 escalation scores, the rates of agreement and weighted κ were: general model (48.8%; 0.16), hospital-specific model (45.0%; 0.25), and group-specific model (43.1%; 0.25). A 17th small hospital was excluded due to missing data. Conclusions: This is novel research as no similar studies were identified, although the topic is important as it addresses a major current health-care challenge. Automated scores can be derived which have the advantage of being derived objectively, avoiding human inter- and intraindividual variation. Prospective testing is recommended to assess potential service planning benefit.