Reduction of Measurement Error due to Survey Length: Evaluation of the Split Questionnaire Design Approach

Abstract

Long survey instruments can be taxing to respondents, which may result in greater measurement error. There is little empirical evidence on the relationship between length and measurement error, possibly leading to longer surveys than desirable. At least equally important is the need for methods to reduce survey length while meeting the survey’s objectives. This study tests the ability to reduce measurement error related to survey length through split questionnaire design, in which the survey is modularized and respondents are randomly assigned to receive subsets of the survey modules. The omitted questions are then multiply imputed for all respondents. The imputation variance, however, may overwhelm any benefits to survey estimates from the reduction of survey length. We use an experimental design to further evaluate the effect of survey length on measurement error and to examine the degree to which a split questionnaire design can yield estimates with less measurement error. We found strong evidence for greater measurement error when the questions were asked late in the survey. We also found that a split questionnaire design retained lower measurement error without compromising total error from the additional imputation variance. This is the first study with an experimental design used to evaluate split questionnaire design, demonstrating substantial benefits in reduction of measurement error. Future experimental designs are needed to empirically evaluate the approach’s ability to reduce nonresponse bias.

Keywords

• Split Questionnaire Design
• Matrix Sampling
• Survey Length
• Measurement Error
• Multiple Imputation