How often does an individual trial agree with its corresponding meta-analysis? A meta-epidemiologic study.

Abstract

OBJECTIVE: A meta-analysis may provide more conclusive results than a single trial. The major cost of meta-analysis is the time of waiting before the meta-analysis becomes available and resources spent on consequent trials that may not be necessary. The objective of this study is to address how often the result of a single trial, in particular the first trial, differs from that of its corresponding meta-analysis so as to reduce unnecessary waiting time and subsequent trials. STUDY DESIGN AND SETTINGS: A meta-epidemiologic study was conducted by collecting meta-analyses from the Cochrane Database of Systematic Reviews and five major medical journals. Effect size of a single trial was compared with that of its corresponding meta-analysis. The single trial includes the first trial, last trial and any trial randomly selected from the meta-analysis. RESULTS: 647 meta-analyses are included and the median number of trials in a meta-analysis is 5. 233 (36.0%) meta-analyses have the first trial with a statistically significant result. When the first trial is statistically significant, 84.1% (95% CI: 79.4%, 88.8%) of the corresponding meta-analyses is both in the same direction and statistically significant. When the first trial is statistically insignificant, 57.9% (95% CI: 53.2%, 62.8%) of the meta-analysis is also statistically insignificant regardless of direction. The median number of years is 6.5 years from the first to the 5th trial. CONCLUSION: The conclusion of the first trial that the treatment is effective or harmful is mostly likely correct. A statistically significant trial agrees more often with its corresponding meta-analysis than a large trial. These findings imply that particularly in some urgent, life-saving or other critical circumstances for which no other effective methods are available, cautious recommendation based on the significant result of the first trial seems justifiable and could start use of an effective intervention by 5-8 years earlier.