An assessment of the autism neuroimaging literature for the prospects of re-executability [version 2; peer review: 2 approved]

Abstract

Background: The degree of reproducibility of the neuroimaging literature in psychiatric application areas has been called into question and the issues that relate to this reproducibility are extremely complex. Some of these complexities have to do with the underlying biology of the disorders that we study and others arise due to the technology we apply to the analysis of the data we collect. Ultimately, the observations we make get communicated to the rest of the community through publications in the scientific literature. Methods: We sought to perform a ‘re-executability survey’ to evaluate the recent neuroimaging literature with an eye toward seeing if the technical aspects of our publication practices are helping or hindering the overall quest for a more reproducible understanding of brain development and aging. The topic areas examined include availability of the data, the precision of the imaging method description and the reporting of the statistical analytic approach, and the availability of the complete results. We applied the survey to 50 publications in the autism neuroimaging literature that were published between September 16, 2017 to October 1, 2018. Results: The results of the survey indicate that for the literature examined, data that is not already part of a public repository is rarely available, software tools are usually named but versions and operating system are not, it is expected that reasonably skilled analysts could approximately perform the analyses described, and the complete results of the studies are rarely available. Conclusions: We have identified that there is ample room for improvement in research publication practices. We hope exposing these issues in the retrospective literature can provide guidance and motivation for improving this aspect of our reporting practices in the future.