Why Welch’s test is Type I error robust

Abstract

The comparison of two means is one of the most commonly applied statistical procedures in psychology. The independent samples t-test corrected for unequal variances is commonly known as Welch’s test, and is widely considered to be a robust alternative to the independent samples t-test. The properties of Welch’s test that make it Type I error robust are examined. The degrees of freedom used in Welch’s test are a random variable, the distributions of which are examined using simulation. It is shown how the distribution for the degrees of freedom is dependent on the sample sizes and the variances of the samples. The impact of sample variances on the degrees of freedom, the resultant critical value and the test statistic is considered, and hence gives an insight into why Welch’s test is Type I error robust under normality.

Keywords

• Independent samples t-test
• Welch’s test
• Welch’s approximation
• Behrens-Fisher problem
• Equality of means