Effect Sizes for Estimating Differential Item Functioning Influence at the Test Level

Abstract

Differential item functioning (DIF) is a critical step in providing evidence to support a scoring inference in building a validity argument for a psychological or educational assessment. Effect sizes can assist in understanding the accumulation of DIF at the test score level. The current simulation study investigated the performance of several proposed effect size measures under a variety of conditions. Conditions under study included varied sample sizes, DIF effect sizes, the proportion of items with DIF, and the type of DIF (additive vs. non-additive). DIF effect sizes under study included sDTF%, uDTF%, τ^w2, d, R¯Δ2, IDIF2*, and S−DIF−V. The results of this study suggest that across study conditions, τ^w2, IDIF2*, and d were consistently the most accurate measures of the DIF effects. The effect sizes were also estimated in an empirical example. Recommendations and implications for practice are discussed.

Keywords

• differential item functioning
• effect sizes
• measurement invariance
• validity