The permutation testing approach: a review

Abstract

In recent years permutation testing methods have increased both in number of applications and in solving complex multivariate problems. A large number of testing problems may also be usefully and effectively solved by traditional parametric or rank-based nonparametric methods, although in relatively mild conditions their permutation counterparts are generally asymptotically as good as the best ones. Permutation tests are essentially of an exact nonparametric nature in a conditional context, where conditioning is on the pooled observed data as a set of sufficient statistics in the null hypothesis. Instead, the reference null distribution of most parametric tests is only known asymptotically. Thus, for most sample sizes of practical interest, the possible lack of efficiency of permutation solutions may be compensated by the lack of approximation of parametric counterparts. There are many complex multivariate problems (quite common in biostatistics, clinical trials, engineering, the environment, epidemiology, experimental data, industrial statistics, pharmacology, psychology, social sciences, etc.) which are difficult to solve outside the conditional framework and outside the nonparametric combination (NPC) method for dependent permutation tests. In this paper we review this method along with a number of applications in different experimental and observational situations (e.g. multi-sided alternatives, zero-inflated data and testing for a stochastic ordering) and we present properties specific to this methodology, such as: for a given number of subjects, when the number of variables diverges and the noncentrality of the combined test diverges accordingly, then the power of combination-based permutation tests converges to one.