The empirical replicability of task-based fMRI as a function of sample size
Han Bossier,
Sanne P. Roels,
Ruth Seurinck,
Tobias Banaschewski,
Gareth J. Barker,
Arun L.W. Bokde,
Erin Burke Quinlan,
Sylvane Desrivières,
Herta Flor,
Antoine Grigis,
Hugh Garavan,
Penny Gowland,
Andreas Heinz,
Bernd Ittermann,
Jean-Luc Martinot,
Eric Artiges,
Frauke Nees,
Dimitri Papadopoulos Orfanos,
Luise Poustka,
Juliane H. Fröhner Dipl-Psych,
Michael N. Smolka,
Henrik Walter,
Robert Whelan,
Gunter Schumann,
Beatrijs Moerkerke
Affiliations
Han Bossier
Department of Data Analysis, Ghent University, Ghent, Belgium; Corresponding author.
Sanne P. Roels
Department of Data Analysis, Ghent University, Ghent, Belgium
Ruth Seurinck
Department of Data Analysis, Ghent University, Ghent, Belgium
Tobias Banaschewski
Department of Child and Adolescent Psychiatry and Psychotherapy, Central Institute of Mental Health, Medical Faculty Mannheim, Heidelberg University, Square J5, 68159, Mannheim, Germany
Gareth J. Barker
Department of Neuroimaging, Institute of Psychiatry, Psychology & Neuroscience, King’s College London, United Kingdom
Arun L.W. Bokde
Discipline of Psychiatry, School of Medicine and Trinity College Institute of Neuroscience, Trinity College Dublin, Ireland
Erin Burke Quinlan
Medical Research Council - Social, Genetic and Developmental Psychiatry Centre, Institute of Psychiatry, Psychology & Neuroscience, King’s College London, United Kingdom
Sylvane Desrivières
Medical Research Council - Social, Genetic and Developmental Psychiatry Centre, Institute of Psychiatry, Psychology & Neuroscience, King’s College London, United Kingdom
Herta Flor
Department of Cognitive and Clinical Neuroscience, Central Institute of Mental Health, Medical Faculty Mannheim, Heidelberg University, Square J5, Mannheim, Germany; Department of Psychology, School of Social Sciences, University of Mannheim, 68131, Mannheim, Germany
Antoine Grigis
NeuroSpin, CEA, Université Paris-Saclay, F-91191, Gif-sur-Yvette, France
Hugh Garavan
Departments of Psychiatry and Psychology, University of Vermont, 05405, Burlington, VT, USA
Penny Gowland
Sir Peter Mansfield Imaging Centre School of Physics and Astronomy, University of Nottingham, University Park, Nottingham, United Kingdom
Andreas Heinz
Charité – Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin, Humboldt-Universität zu Berlin, Berlin Institute of Health, Department of Psychiatry and Psychotherapy, Campus Charité Mitte, Charitéplatz 1, Berlin, Germany
Bernd Ittermann
Physikalisch-Technische Bundesanstalt (PTB), Braunschweig and Berlin, Germany
Jean-Luc Martinot
Institut National de la Santé et de la Recherche Médicale, INSERM Unit 1000 “Neuroimaging & Psychiatry”, University Paris Sud, University Paris Descartes, Sorbonne Paris Cité; and Maison de Solenn, Paris, France
Eric Artiges
Institut National de la Santé et de la Recherche Médicale, INSERM Unit 1000 “Neuroimaging & Psychiatry”, University Paris Sud, University Paris Descartes, Sorbonne Paris Cité; and Psychiatry Department 91G16, Orsay Hospital, France
Frauke Nees
Department of Child and Adolescent Psychiatry and Psychotherapy, Central Institute of Mental Health, Medical Faculty Mannheim, Heidelberg University, Square J5, 68159, Mannheim, Germany; Department of Cognitive and Clinical Neuroscience, Central Institute of Mental Health, Medical Faculty Mannheim, Heidelberg University, Square J5, Mannheim, Germany
Dimitri Papadopoulos Orfanos
NeuroSpin, CEA, Université Paris-Saclay, F-91191, Gif-sur-Yvette, France
Luise Poustka
Department of Child and Adolescent Psychiatry and Psychotherapy, University Medical Centre Göttingen, von-Siebold-Str. 5, 37075, Göttingen, Germany
Juliane H. Fröhner Dipl-Psych
Department of Psychiatry and Neuroimaging Center, Technische Universität Dresden, Dresden, Germany
Michael N. Smolka
Department of Psychiatry and Neuroimaging Center, Technische Universität Dresden, Dresden, Germany
Henrik Walter
Charité – Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin, Humboldt-Universität zu Berlin, Berlin Institute of Health, Department of Psychiatry and Psychotherapy, Campus Charité Mitte, Charitéplatz 1, Berlin, Germany
Robert Whelan
School of Psychology and Global Brain Health Institute, Trinity College Dublin, Ireland
Gunter Schumann
Medical Research Council - Social, Genetic and Developmental Psychiatry Centre, Institute of Psychiatry, Psychology & Neuroscience, King’s College London, United Kingdom
Beatrijs Moerkerke
Department of Data Analysis, Ghent University, Ghent, Belgium
Replicating results (i.e. obtaining consistent results using a new independent dataset) is an essential part of good science. As replicability has consequences for theories derived from empirical studies, it is of utmost importance to better understand the underlying mechanisms influencing it. A popular tool for non-invasive neuroimaging studies is functional magnetic resonance imaging (fMRI). While the effect of underpowered studies is well documented, the empirical assessment of the interplay between sample size and replicability of results for task-based fMRI studies remains limited. In this work, we extend existing work on this assessment in two ways. Firstly, we use a large database of 1400 subjects performing four types of tasks from the IMAGEN project to subsample a series of independent samples of increasing size. Secondly, replicability is evaluated using a multi-dimensional framework consisting of 3 different measures: (un)conditional test-retest reliability, coherence and stability. We demonstrate not only a positive effect of sample size, but also a trade-off between spatial resolution and replicability. When replicability is assessed voxelwise or when observing small areas of activation, a larger sample size than typically used in fMRI is required to replicate results. On the other hand, when focussing on clusters of voxels, we observe a higher replicability. In addition, we observe variability in the size of clusters of activation between experimental paradigms or contrasts of parameter estimates within these.
