The Development of a New Generic Risk-of-Bias Measure for Systematic Reviews of Surveys

Abstract

It is important to evaluate risk of bias of the primary studies included in systematic reviews and meta-analyses. Since tools pertinent to surveys are scarce, the goal of the current research was to develop a measure to address this need. In Study 1, an initial list of 10 relevant topics was compiled from previous measures. In Study 2, the list was refined into an eight-item risk-of-bias measure via discussion and a pilot study. In Study 3, experienced researchers used the measure to asses 70 studies, demonstrating high interrater agreement (weighted Kappa = .82). Inexperienced raters also utilized the measure to code 26 different studies included in a prior meta-analysis, which resulted in adequate interrater agreement (weighted Kappa = .64) and excellent convergent validity (r = .66). Thus, the new measure, designed to be accessible and flexible, can increase standardization of risk-of-bias evaluations and contribute to the interpretation of systematic reviews and meta-analytic findings.

Keywords

• risk of bias
• systematic review
• meta-analysis
• survey quality
• correlational studies