Equal Quality for Online and Lab Data: A Direct Comparison from Two Dual-Task Paradigms

Abstract

Conducting behavioral experiments online has become more prevalent recently. Still, there is reluctance to embrace the possibilities this technology has to offer. So far, only simple tasks have been replicated in an online setting. In order to investigate whether collecting online also leads to high quality data in demanding tasks, we directly compared data collected in the lab with data collected online from a demanding dual-task paradigm and a psychological refractory period paradigm. In Experiment 1, we recruited from local pools, online and offline; in Experiment 2, we collected lab data from our local pool and online data from a remote commercial participant platform. We found that all relevant effects were replicated in the lab and online settings; effect sizes were similar. Additionally, most response time distributions were even statistically equivalent when comparing online and lab data. Thus, online effect sizes and variances can be comparable to lab-based data. Online studies are time-efficient and recruiting an online sample instead or on top of a laboratory sample should be considered for basic behavioral research. This can serve an important role in the generalizability and replicability of findings in the cognitive and behavioral sciences.

Keywords

• online experiments
• web-based experiments
• dual-task