eLife (Nov 2021)
Consensus-based guidance for conducting and reporting multi-analyst studies
- Balazs Aczel,
- Barnabas Szaszi,
- Gustav Nilsonne,
- Olmo R van den Akker,
- Casper J Albers,
- Marcel ALM van Assen,
- Jojanneke A Bastiaansen,
- Daniel Benjamin,
- Udo Boehm,
- Rotem Botvinik-Nezer,
- Laura F Bringmann,
- Niko A Busch,
- Emmanuel Caruyer,
- Andrea M Cataldo,
- Nelson Cowan,
- Andrew Delios,
- Noah NN van Dongen,
- Chris Donkin,
- Johnny B van Doorn,
- Anna Dreber,
- Gilles Dutilh,
- Gary F Egan,
- Morton Ann Gernsbacher,
- Rink Hoekstra,
- Sabine Hoffmann,
- Felix Holzmeister,
- Juergen Huber,
- Magnus Johannesson,
- Kai J Jonas,
- Alexander T Kindel,
- Michael Kirchler,
- Yoram K Kunkels,
- D Stephen Lindsay,
- Jean-Francois Mangin,
- Dora Matzke,
- Marcus R Munafò,
- Ben R Newell,
- Brian A Nosek,
- Russell A Poldrack,
- Don van Ravenzwaaij,
- Jörg Rieskamp,
- Matthew J Salganik,
- Alexandra Sarafoglou,
- Tom Schonberg,
- Martin Schweinsberg,
- David Shanks,
- Raphael Silberzahn,
- Daniel J Simons,
- Barbara A Spellman,
- Samuel St-Jean,
- Jeffrey J Starns,
- Eric Luis Uhlmann,
- Jelte Wicherts,
- Eric-Jan Wagenmakers
Affiliations
- Balazs Aczel
- ORCiD
- ELTE Eotvos Lorand University, Budapest, Hungary
- Barnabas Szaszi
- ORCiD
- ELTE Eotvos Lorand University, Budapest, Hungary
- Gustav Nilsonne
- ORCiD
- Karolinska Institutet, Stockholm, Sweden; Stockholm University, Stockholm, Sweden
- Olmo R van den Akker
- Tilburg University, Tilburg, Netherlands
- Casper J Albers
- University of Groningen, Groningen, Netherlands
- Marcel ALM van Assen
- Tilburg University, Tilburg, Netherlands; Utrecht University, Utrecht, Netherlands
- Jojanneke A Bastiaansen
- ORCiD
- University Medical Center Groningen, University of Groningen, Groningen, Netherlands; Friesland Mental Health Care Services, Leeuwarden, Netherlands
- Daniel Benjamin
- ORCiD
- University of California Los Angeles, Los Angeles, United States; National Bureau of Economic Research, Cambridge, United States
- Udo Boehm
- ORCiD
- University of Amsterdam, Amsterdam, Netherlands
- Rotem Botvinik-Nezer
- ORCiD
- Dartmouth College, Hanover, United States
- Laura F Bringmann
- ORCiD
- University of Groningen, Groningen, Netherlands
- Niko A Busch
- ORCiD
- University of Münster, Muenster, Germany
- Emmanuel Caruyer
- ORCiD
- University of Rennes, CNRS, Inria and Inserm, Rennes, France
- Andrea M Cataldo
- ORCiD
- McLean Hospital, Belmont, United States; Harvard Medical School, Boston, United States
- Nelson Cowan
- ORCiD
- University of Missouri, Columbia, United States
- Andrew Delios
- ORCiD
- National University of Singapore, Singapore, Singapore
- Noah NN van Dongen
- ORCiD
- University of Amsterdam, Amsterdam, Netherlands
- Chris Donkin
- University of New South Wales, Sydney, Australia
- Johnny B van Doorn
- ORCiD
- University of Amsterdam, Amsterdam, Netherlands
- Anna Dreber
- ORCiD
- Stockholm School of Economics, Stockholm, Sweden; University of Innsbruck, Innsbruck, Austria
- Gilles Dutilh
- University Hospital Basel, Basel, Switzerland
- Gary F Egan
- ORCiD
- Monash University, Melbourne, Australia
- Morton Ann Gernsbacher
- ORCiD
- University of Wisconsin-Madison, Madison, United States
- Rink Hoekstra
- ORCiD
- University of Groningen, Groningen, Netherlands
- Sabine Hoffmann
- ORCiD
- Ludwig-Maximilians-University, Munich, Germany
- Felix Holzmeister
- ORCiD
- University of Innsbruck, Innsbruck, Austria
- Juergen Huber
- ORCiD
- University of Innsbruck, Innsbruck, Austria
- Magnus Johannesson
- ORCiD
- Stockholm School of Economics, Stockholm, Sweden
- Kai J Jonas
- Maastricht University, Maastricht, Netherlands
- Alexander T Kindel
- Princeton University, Princeton, United States
- Michael Kirchler
- ORCiD
- University of Innsbruck, Innsbruck, Austria
- Yoram K Kunkels
- University Medical Center Groningen, University of Groningen, Groningen, Netherlands
- D Stephen Lindsay
- University of Victoria, Victoria, Canada
- Jean-Francois Mangin
- ORCiD
- Université Paris-Saclay, Paris, France; Neurospin, CEA, Paris, France
- Dora Matzke
- Amsterdam University, Amsterdam, Netherlands
- Marcus R Munafò
- University of Bristol, Bristol, United Kingdom
- Ben R Newell
- ORCiD
- University of New South Wales, Sydney, Australia
- Brian A Nosek
- Center for Open Science, Charlottesville, United States; University of Virginia, Charlottesville, United States
- Russell A Poldrack
- ORCiD
- Stanford University, Stanford, United States
- Don van Ravenzwaaij
- ORCiD
- University of Groningen, Groningen, Netherlands
- Jörg Rieskamp
- ORCiD
- University of Basel, Basel, Switzerland
- Matthew J Salganik
- Princeton University, Princeton, United States
- Alexandra Sarafoglou
- Amsterdam University, Amsterdam, Netherlands
- Tom Schonberg
- ORCiD
- Tel Aviv University, Tel Aviv, Israel
- Martin Schweinsberg
- ORCiD
- ESMT Berlin, Berlin, Germany
- David Shanks
- ORCiD
- University College London, London, United Kingdom
- Raphael Silberzahn
- University of Sussex, Brighton, United Kingdom
- Daniel J Simons
- University of Illinois, Urbana-Champaign, United States
- Barbara A Spellman
- University of Virginia, Charlottesville, United States
- Samuel St-Jean
- ORCiD
- University of Alberta, Edmonton, Canada; Lund University, Lund, United States
- Jeffrey J Starns
- University of Massachusetts Amherst, Amherst, United States
- Eric Luis Uhlmann
- INSEAD, Singapore, Singapore
- Jelte Wicherts
- ORCiD
- Tilburg University, Tilburg, Netherlands
- Eric-Jan Wagenmakers
- Amsterdam University, Amsterdam, Netherlands
- DOI
- https://doi.org/10.7554/eLife.72185
- Journal volume & issue
-
Vol. 10
Abstract
Any large dataset can be analyzed in a number of ways, and it is possible that the use of different analysis strategies will lead to different results and conclusions. One way to assess whether the results obtained depend on the analysis strategy chosen is to employ multiple analysts and leave each of them free to follow their own approach. Here, we present consensus-based guidance for conducting and reporting such multi-analyst studies, and we discuss how broader adoption of the multi-analyst approach has the potential to strengthen the robustness of results and conclusions obtained from analyses of datasets in basic and applied research.
Keywords
- multi-analyst
- metascience
- statistical practice
- science forum
- expert consensus
- analytical variability
