Simulation-Based Inference: Random Sampling vs. Random Assignment? What Instructors Should Know

Abstract

“Simulation-based inference” is often considered a pedagogical strategy for helping students develop inferential reasoning, for example, giving them a visual and concrete reference for deciding whether the observed statistic is unlikely to happen by chance alone when the null hypothesis is true. In this article, we highlight for teachers some implications of different simulation strategies when analyzing two variables. In particular, does it matter whether the simulation models random sampling or random assignment? We present examples from comparing two means and simple linear regression, highlighting the impact on the standard deviation of the null distribution. We also highlight some possible extensions that simulation-based inference easily allows. Supplementary materials for this article are available online.

Keywords

• Permutation test
• Randomization distribution
• Standard error
• Theory-based inference