Random responding from participants is a threat to the validity of social science research results

Abstract

Read online

Research in the social sciences often relies upon the motivation and goodwill of research participants (e.g., teachers, students, minimally-compensated volunteers) to do their best on low stakes assessments of the effects of interventions. Research participants who are unmotivated to perform well can engage in random responding on outcome measures, which can cause substantial mis-estimation of results, biasing results toward the null hypothesis. Data from a recent educational intervention study served as a clear example of this problem: participants identified as random responders showed substantially lower scores than other participants on tests during the study, and failed to show growth in scores from pre- to posttest, while those not engaging in random responding showed much higher scores and significant growth over time. This served to mask the hypothesized group differences across instructional method when random responders were retained in the sample (anticipated group differences were significant when these random responders were removed). We remind researchers to screen their data for random responding (and other response biases) in their critical outcome measures in order to improve the odds of detecting effects of their interventions.

Keywords

• best practices
• random responding
• response set
• type II error