Royal Society Open Science (Jul 2024)
Subjective evidence evaluation survey for many-analysts studies
- Alexandra Sarafoglou,
- Suzanne Hoogeveen,
- Don van den Bergh,
- Balazs Aczel,
- Casper J. Albers,
- Tim Althoff,
- Rotem Botvinik-Nezer,
- Niko A. Busch,
- Andrea M. Cataldo,
- Berna Devezer,
- Noah N. N. van Dongen,
- Anna Dreber,
- Eiko I. Fried,
- Rink Hoekstra,
- Sabine Hoffman,
- Felix Holzmeister,
- Jürgen Huber,
- Nick Huntington-Klein,
- John Ioannidis,
- Magnus Johannesson,
- Michael Kirchler,
- Eric Loken,
- Jan-Francois Mangin,
- Dora Matzke,
- Albert J. Menkveld,
- Gustav Nilsonne,
- Don van Ravenzwaaij,
- Martin Schweinsberg,
- Hannah Schulz-Kuempel,
- David R. Shanks,
- Daniel J. Simons,
- Barbara A. Spellman,
- Andrea H. Stoevenbelt,
- Barnabas Szaszi,
- Darinka Trübutschek,
- Francis Tuerlinckx,
- Eric L. Uhlmann,
- Wolf Vanpaemel,
- Jelte Wicherts,
- Eric-Jan Wagenmakers
Affiliations
- Alexandra Sarafoglou
- Department of Psychology, University of Amsterdam, Amsterdam, The Netherlands
- Suzanne Hoogeveen
- Utrecht University, Utrecht, The Netherlands
- Don van den Bergh
- Department of Psychology, University of Amsterdam, Amsterdam, The Netherlands
- Balazs Aczel
- Institute of Psychology, ELTE Eötvös Lorénd University, Budapest, Hungary
- Casper J. Albers
- Heymans Institute for Psychological Research, University of Groningen, Groningen, The Netherlands
- Tim Althoff
- Allen School of Computer Science and Engineering, University of Washington, Seattle, WA, USA
- Rotem Botvinik-Nezer
- Hebrew University of Jerusalem, Jerusalem, Israel
- Niko A. Busch
- Institute for Psychology, University of Münster, Münster, Germany
- Andrea M. Cataldo
- Center for Depression, Anxiety and Stress Research, McLean Hospital, Belmont, MA, USA
- Berna Devezer
- Department of Business, University of Idaho, Moscow, ID, USA
- Noah N. N. van Dongen
- Department of Psychology, University of Amsterdam, Amsterdam, The Netherlands
- Anna Dreber
- Stockholm School of Economics, Stockholm, Sweden
- Eiko I. Fried
- Department of Psychology, Leiden University, Leiden, The Netherlands
- Rink Hoekstra
- Nieuwenhuis Institute for Educational Research, University of Groningen, Groningen, The Netherlands
- Sabine Hoffman
- Department of Statistics, Ludwig-Maximilians-Universität München, Munchen, Bayern, Germany
- Felix Holzmeister
- University of Innsbruck, Innsbruck, Tirol, Austria
- Jürgen Huber
- University of Innsbruck, Innsbruck, Tirol, Austria
- Nick Huntington-Klein
- Seattle University, Seattle, WA, USA
- John Ioannidis
- Meta-Research Innovation Center at Stanford (METRICS) and Departments of Medicine, of Epidemiology and of Population Health, of Biomedical Data Science, and of Statistics, Stanford University, Stanford, CA, USA
- Magnus Johannesson
- Stockholm School of Economics, Stockholm, Sweden
- Michael Kirchler
- University of Innsbruck, Innsbruck, Tirol, Austria
- Eric Loken
- University of Conneticut, Storrs, CT, USA
- Jan-Francois Mangin
- University Paris-Saclay, Gif-sur-Yvette, France
- Dora Matzke
- Department of Psychology, University of Amsterdam, Amsterdam, The Netherlands
- Albert J. Menkveld
- Vrije Universiteit Amsterdam, Amsterdam, Noord-Holland, The Netherlands
- Gustav Nilsonne
- Karolinska Institutet, Solna, Sweden
- Don van Ravenzwaaij
- Heymans Institute for Psychological Research, University of Groningen, Groningen, The Netherlands
- Martin Schweinsberg
- ESMT Berlin, Berlin, Germany
- Hannah Schulz-Kuempel
- Department of Statistics and The Institute for Medical Information Processing, Biometry, and Epidemiology, LMU Munich, Munchen, Bayern, Germany
- David R. Shanks
- Division of Psychology and Language Sciences, University College London, 26 Bedford Way, London WC1H 0AP, UK
- Daniel J. Simons
- University of Illinois—Urbana-Champaign, Urbana, IL, USA
- Barbara A. Spellman
- School of Law, University of Virginia, 580 Massie Road, Charlottesville, VA, USA
- Andrea H. Stoevenbelt
- Nieuwenhuis Institute for Educational Research, University of Groningen, Groningen, The Netherlands
- Barnabas Szaszi
- Institute of Psychology, ELTE Eötvös Lorénd University, Budapest, Hungary
- Darinka Trübutschek
- Max Planck Institute for Empirical Aesthetics, Frankfurt am Main, Germany
- Francis Tuerlinckx
- University of Leuven, Leuven, Belgium
- Eric L. Uhlmann
- INSEAD, Fontainebleau, Île-de-France, France
- Wolf Vanpaemel
- University of Leuven, Leuven, Belgium
- Jelte Wicherts
- Department of Methodology and Statistics, Tilburg University, Tilburg, The Netherlands
- Eric-Jan Wagenmakers
- Department of Psychology, University of Amsterdam, Amsterdam, The Netherlands
- DOI
- https://doi.org/10.1098/rsos.240125
- Journal volume & issue
-
Vol. 11,
no. 7
Abstract
Many-analysts studies explore how well an empirical claim withstands plausible alternative analyses of the same dataset by multiple, independent analysis teams. Conclusions from these studies typically rely on a single outcome metric (e.g. effect size) provided by each analysis team. Although informative about the range of plausible effects in a dataset, a single effect size from each team does not provide a complete, nuanced understanding of how analysis choices are related to the outcome. We used the Delphi consensus technique with input from 37 experts to develop an 18-item subjective evidence evaluation survey (SEES) to evaluate how each analysis team views the methodological appropriateness of the research design and the strength of evidence for the hypothesis. We illustrate the usefulness of the SEES in providing richer evidence assessment with pilot data from a previous many-analysts study.
Keywords
