Impact of Differential Item Functioning on Subsequent Statistical Conclusions Based on Observed Test Score Data

Abstract

This simulation study investigated the impact of differential item functioning (DIF) on the Type I error rate and effect size of the independent samples t-test on the observed total test scores. Five studies were conducted: studies one to three investigated the impact of unidirectional DIF (i.e., DIF amplification) on the Type I error rate and effect size of the independent sample t-test, studies four and five investigated the DIF cancellation effects on the Type I error rate, and effect size of the independent sample t-test. The Type I error rate and effect size were defined in terms of latent population means rather than observed sample means. The results showed that the amplification and cancellation effects among uniform DIF items did transfer to test level. These findings highlight the importance of screening DIF before conducting any further statistical analysis.