The Understanding on How to Determine Sample Size in Psychological Science: Data and Simulation

Abstract

The lack of knowledge on how to determine sample sizes in experiments is arguably one of the main reasons underlying the replication crisis in psychological science. A survey distributed among Indonesian students and researchers concerning 1) familiarity and understanding of statistical concepts related to sample size determination, 2) current sample size determination practices in experiments, and 3) ideal sample sizes for experiments. Subsequently, we simulated expected statistical power given the sample sizes reported in the survey. Results demonstrated that 1) while a majority of participants were somewhat familiar with statistical concepts related to sample size determination, they did not always endorse the correct and/or complete definition of each concept. Furthermore, 2) participants relied on practical considerations in determining sample sizes. Consequently, 3) the reported sample sizes did not have sufficient power to detect small to medium effect sizes, which are commonly present in psychological science.

Keywords

• effect size
• replication crisis
• sample size
• statistical power