Dissecting the Onion: Identifying and Remediating Issues Surrounding Data Integrity in Online Survey Research

Abstract

In this non-empirical article, which is intended as a decision-making resource for researchers, we identify issues surrounding data integrity that commonly arise in online survey research and we propose remediation strategies based on challenges we encountered during a particular pilot study as well as our collective experience with conducting online survey research. Using the metaphor of an onion, we peel off the layers of this complex problem, synthesize the various available strategies used across disciplines, and propose some novel ones based on our perspective as psychologists. Corresponding to this multi-layered problem, we propose multi-layered solutions to prevent illegitimate responding—by both humans and non-humans (robots or “bots” for short)—from compromising the quality of data collected via online survey research. The first layer entails strategic item selection and protective programming in survey development. The second layer involves astute advertising and recruitment tactics to minimize illegitimate responses during survey dissemination. The third layer includes algorithms and other mechanisms to identify suspicious responses for possible exclusion during data verification. When we peel off the layers and reach the core problem of illegitimate responses to online surveys—financial incentives—we will propose ways of navigating respondent reimbursement to mitigate their inadvertent harmful impacts on the research process. By proposing these solutions, we aim to protect the integrity of scientific inquiry in psychology, especially given how often this method is used in the discipline.

Keywords

• online surveys
• data integrity
• survey development
• respondent incentives
• bots