Reducing our dependence on null hypothesis testing: A key to enhance the reproducibility and credibility of our science

Abstract

Problemification: Over-reliance on null hypothesis significance testing (NHST) is one of the most important causes of the emerging crisis over the credibility and reproducibility of our science. Implications: Most studies in the behavioural and social sciences have low levels of statistical power. Because ‘significant’ results are often required, but often difficult to produce, the temptation to engage in questionable research practices that will produce these results is immense. Purpose: Methodologists have been trying for decades to convince researchers, reviewers and editors that significance tests are neither informative nor useful. A recent set of articles published in top journals and endorsed by hundreds of scientists around the world seem to provide a fresh impetus for overturning the practice of using NHST as the primary, and sometimes sole basis for evaluating research results. Recommendations: Authors, reviewers and journal editors are asked to change long-engrained habits and realise that ‘statistically significant’ says more about the design of one’s study than about the importance of one’s results. They are urged to embrace the ATOM principle in evaluating research results, that is, accept that there will always be uncertainty, and be thoughtful, open and modest in evaluating what the data mean.

Keywords

• significance testing
• confidence intervals
• questionable research practices
• null hypothesis