Tests for publication bias are unreliable in case of heteroscedasticity

Abstract

Regression based methods for the detection of publication bias in meta-analysis have been extensively evaluated in literature. When dealing with continuous outcomes, specific hidden factors (e.g., heteroscedasticity) may interfere with the test statistics. In this paper we investigate the influence of residual heteroscedasticity on the performance of four tests for publication bias: the Egger test, the Begg-Mazumdar test and two tests based on weighted regression. In the presence of heteroscedasticity, the Egger test and the weighted regression tests highly inflate the Type I error rate, while the Begg-Mazumdar test deflates the Type I error rate. Although all three tests already have low statistical power, heteroscedasticity typically reduces it further. Our results in combination with earlier discussions on publication bias tests lead us to conclude that application of these tests on continuous treatment effects is not warranted.

Keywords

• Heteroscedastic mixed effects model
• Aggregated data meta-analysis
• Mean difference treatment effect sizes