The dangers of ignoring underlying trends in before-and-after studies – A cautionary tale using hip fracture mortality data

Abstract

Background: Before-and-after studies are a valuable study design in situations where randomization is not feasible. These studies measure an outcome both before and after an intervention and compare the outcome rates in both time periods to determine the effectiveness of the intervention. Before-and-after studies do not involve a contemporaneous control group and must, therefore, take into account any underlying secular trends to separate the effect of the intervention from any pre-existing trend. Methods: To illustrate the importance of accounting for underlying trends, we performed a before-and-after study assessing 30-day mortality in hip fracture patients without any actual intervention, and instead designated an arbitrarily chosen time point as our ‘intervention’. We then analysed the data first disregarding and then incorporating the pre-existing underlying trend. We did this to show that even intervention of nothing may be spuriously interpreted to have an effect if the before-and-after study design is incorrectly analysed. Our study involved a secondary analysis of routinely collected data on 30-day mortality following hip fracture in our institution. Results: We found a secular trend in our data showing improving 30-day mortality in hip fracture patients in our institution. We then demonstrated that disregarding this underlying trend showed that our intervention of nothing ‘resulted’ in a significant 54% decrease in mortality, from 6.7% in the ‘before’ period to 3.1% in the ‘after’ period ( p < 0.0008). Though the 30-day mortality rate decreased during the ‘after’ period, the decrease was not significantly different from the underlying trend in the ‘before’ period, projected onto the ‘after’ period. When we accounted for the underlying trend in our analysis, the impact of the intervention (nothing) on 30-day mortality was no longer apparent (incidence rate ratio 0.75, 95% confidence interval 0.32–1.78; p = 0.5). Conclusion: Our study highlights the importance of appropriate measurement and consideration of underlying trends when analysing data from before-and-after studies and illustrates what can happen should researchers neglect this important step.