Royal Society Open Science (Mar 2025)
The reliability of replications: a study in computational reproductions
- Nate Breznau,
- Eike Mark Rinke,
- Alexander Wuttke,
- Muna Adem,
- Jule Adriaans,
- Esra Akdeniz,
- Amalia Alvarez-Benjumea,
- Henrik K. Andersen,
- Daniel Auer,
- Flavio Azevedo,
- Oke Bahnsen,
- Ling Bai,
- Dave Balzer,
- Paul C. Bauer,
- Gerrit Bauer,
- Markus Baumann,
- Sharon Baute,
- Verena Benoit,
- Julian Bernauer,
- Carl Berning,
- Anna Berthold,
- Felix S. Bethke,
- Thomas Biegert,
- Katharina Blinzler,
- Johannes N. Blumenberg,
- Licia Bobzien,
- Andrea Bohman,
- Thijs Bol,
- Amie Bostic,
- Zuzanna Brzozowska,
- Katharina Burgdorf,
- Kaspar Burger,
- Kathrin Busch,
- Juan-Carlos Castillo,
- Nathan Chan,
- Pablo Christmann,
- Roxanne Connelly,
- Christian S. Czymara,
- Elena Damian,
- Eline A. de Rooij,
- Alejandro Ecker,
- Achim Edelmann,
- Christina Eder,
- Maureen A. Eger,
- Simon Ellerbrock,
- Anna Forke,
- Andrea Forster,
- Danilo Freire,
- Chris Gaasendam,
- Konstantin Gavras,
- Vernon Gayle,
- Theresa Gessler,
- Timo Gnambs,
- Amélie Godefroidt,
- Max Grömping,
- Martin Groß,
- Stefan Gruber,
- Tobias Gummer,
- Andreas Hadjar,
- Verena Halbherr,
- Jan Paul Heisig,
- Sebastian Hellmeier,
- Stefanie Heyne,
- Magdalena Hirsch,
- Mikael Hjerm,
- Oshrat Hochman,
- Jan H. Höffler,
- Andreas Hövermann,
- Sophia Hunger,
- Christian Hunkler,
- Nora Huth-Stöckle,
- Zsófia S. Ignácz,
- Sabine Israel,
- Laura Jacobs,
- Jannes Jacobsen,
- Bastian Jaeger,
- Sebastian Jungkunz,
- Nils Jungmann,
- Jennifer Kanjana,
- Mathias Kauff,
- Salman Khan,
- Sayak Khatua,
- Manuel Kleinert,
- Julia Klinger,
- Jan-Philipp Kolb,
- Marta Kołczyńska,
- John Kuk,
- Katharina Kunißen,
- Dafina Kurti Sinatra,
- Alexander Langenkamp,
- Robin C. Lee,
- Philipp M. Lersch,
- David Liu,
- Lea-Maria Löbel,
- Philipp Lutscher,
- Matthias Mader,
- Joan E. Madia,
- Natalia Malancu,
- Luis Maldonado,
- Helge Marahrens,
- Nicole Martin,
- Paul Martinez,
- Jochen Mayerl,
- Oscar J. Mayorga,
- Robert McDonnell,
- Patricia McManus,
- Kyle McWagner,
- Cecil Meeusen,
- Daniel Meierrieks,
- Jonathan Mellon,
- Friedolin Merhout,
- Samuel Merk,
- Daniel Meyer,
- Leticia Micheli,
- Jonathan Mijs,
- Cristóbal Moya,
- Marcel Neunhoeffer,
- Daniel Nüst,
- Olav Nygård,
- Fabian Ochsenfeld,
- Gunnar Otte,
- Anna Pechenkina,
- Mark Pickup,
- Christopher Prosser,
- Louis Raes,
- Kevin Ralston,
- Miguel Ramos,
- Frank Reichert,
- Arne Roets,
- Jonathan Rogers,
- Guido Ropers,
- Robin Samuel,
- Gregor Sand,
- Constanza Sanhueza Petrarca,
- Ariela Schachter,
- Merlin Schaeffer,
- David Schieferdecker,
- Elmar Schlueter,
- Katja Schmidt,
- Regine Schmidt,
- Alexander Schmidt-Catran,
- Claudia Schmiedeberg,
- Jürgen Schneider,
- Martijn Schoonvelde,
- Julia Schulte-Cloos,
- Sandy Schumann,
- Reinhard Schunck,
- Julian Seuring,
- Henning Silber,
- Willem Sleegers,
- Nico Sonntag,
- Alexander Staudt,
- Nadia Steiber,
- Nils D. Steiner,
- Sebastian Sternberg,
- Dieter Stiers,
- Dragana Stojmenovska,
- Nora Storz,
- Erich Striessnig,
- Anne-Kathrin Stroppe,
- Jordan W. Suchow,
- Janna Teltemann,
- Andrey Tibajev,
- Brian Tung,
- Giacomo Vagni,
- Jasper Van Assche,
- Meta van der Linden,
- Jolanda van der Noll,
- Arno Van Hootegem,
- Stefan Vogtenhuber,
- Bogdan Voicu,
- Fieke Wagemans,
- Nadja Wehl,
- Hannah Werner,
- Brenton M. Wiernik,
- Fabian Winter,
- Christof Wolf,
- Cary Wu,
- Yuki Yamada,
- Björn Zakula,
- Nan Zhang,
- Conrad Ziller,
- Stefan Zins,
- Tomasz Żółtak,
- Hung H.V. Nguyen
Affiliations
- Nate Breznau
- Organization and Program Planning, German Institute for Adult Education—Leibniz Center of Lifelong Learning, Bonn 53175, Germany
- Eike Mark Rinke
- School of Politics and International Studies, University of Leeds, Leeds LS2 9JT, UK
- Alexander Wuttke
- Geschwister Scholl Institute, LMU Munich, Munich 81541, Germany
- Muna Adem
- Department of Sociology, University of Maryland, College Park 47405, USA
- Jule Adriaans
- Faculty of Sociology, University of Bielefeld, Bielefeld 33615, Germany
- Esra Akdeniz
- School of Medicine, Marmara University, Istanbul 34722, Turkey
- Amalia Alvarez-Benjumea
- The Institute of Public Goods and Policies (IPP), Centro de Ciencias Humanas y Sociales—Consejo Superior de Investigaciones Científicas, Madrid 28003, Spain
- Henrik K. Andersen
- Institute of Sociology, Chemnitz University of Technology, Chemnitz 09126, Germany
- Daniel Auer
- Social and Political Science, Collegio Carlo Alberto, Turin 10122, Italy
- Flavio Azevedo
- Department of Interdisciplinary Social Science, Utrecht University, Utrecht 3584CH, Netherlands
- Oke Bahnsen
- School of Social Sciences, University of Mannheim, Mannheim 68159, Germany
- Ling Bai
- Dave Balzer
- Institute of Sociology, Johannes Gutenberg University Mainz, Mainz 55128, Germany
- Paul C. Bauer
- Institute of Political Science, University of Freiburg, Freiburg im Breisgau 79085, Germany
- Gerrit Bauer
- Department of Sociology, LMU Munich, Munich 80801, Germany
- Markus Baumann
- Office of the Executive Board, GESIS—Leibniz Institute for the Social Sciences, Mannheim 68159, Germany
- Sharon Baute
- Department of Politics and Public Administration, University of Konstanz, Konstanz 78457, Germany
- Verena Benoit
- Faculty of Social Sciences, Economics, and Business Administration, University of Bamberg, Bamberg 96052, Germany
- Julian Bernauer
- Mannheim Centre for European Social Research (MZES), University of Mannheim, Mannheim 68131, Germany
- Carl Berning
- Institute for Political Science, Johannes Gutenberg University Mainz, Mainz 55099, Germany
- Anna Berthold
- Faculty of Social Sciences, Economics, and Business Administration, University of Bamberg, Bamberg 96052, Germany
- Felix S. Bethke
- Research Department IV: Intrastate Conflict, Peace Research Institute Frankfurt (PRIF), Frankfurt 60329, Germany
- Thomas Biegert
- Department of Social Policy, London School of Economics and Political Science, London WC2A 2AE, UK
- Katharina Blinzler
- Survey Data Curation, GESIS—Leibniz Institute for the Social Sciences, Mannheim 50667, Germany
- Johannes N. Blumenberg
- Knowledge Exchange and Outreach, GESIS—Leibniz-Institute for the Social Sciences, Mannheim 67067, Germany
- Licia Bobzien
- Faculty of Economics and Social Sciences, University of Potsdam, Potsdam 10117, Germany
- Andrea Bohman
- Department of Sociology, Umeå University, Umeå 901 87, Sweden
- Thijs Bol
- Department of Sociology, University of Amsterdam, Amsterdam 1012 WP, Netherlands
- Amie Bostic
- Department of Sociology, The University of Texas Rio Grande Valley, Edinburg TX 78539, USA
- Zuzanna Brzozowska
- Vienna Institute of Demography, Austrian Academy of Sciences, Vienna, Austria & Austrian National Public Health Institute (Gesundheit Österreich GmbH, GÖG), Vienna, Austria
- Katharina Burgdorf
- School of Social Sciences, University of Bremen, Bremen 28359, Germany
- Kaspar Burger
- Department of Education, University of Potsdam, Potsdam, 14469, Germany
- Kathrin Busch
- Independent Researcher
- Juan-Carlos Castillo
- Department of Sociology Universidad de Chile, Millennium Nucleus on Digital Inequalities and Opportunities (NUDOS NCS2022_046), and Center for Social Conflict and Cohesion Studies (COES), University of Chile, Santiago 7800284, Chile
- Nathan Chan
- Department of Political Science and International Relations, Loyola Marymount University, Los Angeles CA 90045, USA
- Pablo Christmann
- Data and Research on Society, GESIS—Leibniz-Institute for the Social Sciences, Mannheim 68159, Germany
- Roxanne Connelly
- School of Social and Political Science, University of Edinburgh, Edinburgh EH8 9LD, Scotland
- Christian S. Czymara
- Institute of Sociology, Goethe-University Frankfurt, Frankfurt 60323, Germany
- Elena Damian
- Epidemiology and Public Health, Sciensano, Brussels 1050, Belgium
- Eline A. de Rooij
- Political Science, Simon Fraser University, Burnaby V5A 1S6, Canada
- Alejandro Ecker
- Institute of Political Science, Heidelberg University, Heidelberg 69115, Germany
- Achim Edelmann
- Médialab Sciences Po, Paris 75007, France
- Christina Eder
- Survey Data Curation, GESIS—Leibniz Institute for the Social Sciences, Mannheim 50667, Germany
- Maureen A. Eger
- Department of Sociology, Umeå University, Umeå 901 87, Sweden
- Simon Ellerbrock
- Mannheim Centre for European Social Research (MZES), University of Mannheim, Mannheim 68131, Germany
- Anna Forke
- Andrea Forster
- Department of Sociology, Utrecht University, Utrecht 3584 CS, Netherlands
- Danilo Freire
- Department of Quantitative Theory and Methods, Emory University, Atlanta 30306, USA
- Chris Gaasendam
- Department of Work and Social Economy, Government of Flanders-Belgium,
- Konstantin Gavras
- School of Social Sciences, University of Mannheim, Mannheim 68159, Germany
- Vernon Gayle
- School of Social and Political Science, University of Edinburgh, Edinburgh EH8 9LD, Scotland
- Theresa Gessler
- Kulturwissenschaftliche Fakultät, Europa Universität Viadrina, Frankfurt 15230, Germany
- Timo Gnambs
- Educational Measurement, Leibniz Institute for Educational Trajectories, Bamberg 96047, Germany
- Amélie Godefroidt
- Centre for Research on Peace and Development, KU Leuven, Leuven 3000, Belgium
- Max Grömping
- School of Government and International Relations, Griffith University, Nathan, Queensland 4111, Australia
- Martin Groß
- Department of Sociology, University of Tübingen, Tübingen 72074, Germany
- Stefan Gruber
- Research Data Center and Communication, SHARE BERLIN Institute, Berlin 10115, Germany
- Tobias Gummer
- Data and Research on Society, GESIS—Leibniz Institute for the Social Sciences, Mannheim 68159, Germany
- Andreas Hadjar
- Division Sociology, Social Policy and Social Work, University of Fribourg, Fribourg CH-1700, Switzerland
- Verena Halbherr
- Association for Doctoral Studies Baden-Wuerttemberg 70174, Germany
- Jan Paul Heisig
- Research Group ‘Health and Social Inequality’, WZB Berlin Social Science Center, Berlin 10785, Germany
- Sebastian Hellmeier
- Transformations of Democracy Unit, WZB Berlin Social Science Center, Berlin 10785, Germany
- Stefanie Heyne
- Mannheim Centre for European Social Research (MZES), University of Mannheim, Mannheim 68131, Germany
- Magdalena Hirsch
- Research Unit Migration, Integration, Transnationalization, WZB Berlin Social Science Center, Berlin 10785, Germany
- Mikael Hjerm
- Department of Sociology, Umeå University, Umeå 901 87, Sweden
- Oshrat Hochman
- Data and Research on Society, GESIS—Leibniz Institue for the Social Sciences, Mannheim 68159, Germany
- Jan H. Höffler
- Facultad de Emprendimiento, Negocios y Economía, Universidad Espíritu Santo, ReplicationWiki, and EQ-Lab,
- Andreas Hövermann
- Wirtschafts- und Sozialwissenschaftliches Institut (WSI), Hans-Böckler-Foundation, Düsseldorf 40474, Germany
- Sophia Hunger
- SOCIUM—Research Center on Inequality and Social Policy, University of Bremen, Bremen 10785, Germany
- Christian Hunkler
- Berlin Institute for Integration and Migration Research (BIM), Humboldt Universität zu Berlin, Berlin 10099, Germany
- Nora Huth-Stöckle
- School of Human and Social Sciences, University of Wuppertal, Wuppertal 42119, Germany
- Zsófia S. Ignácz
- Institute of Sociology, Goethe-University Frankfurt, Frankfurt 60323, Germany
- Sabine Israel
- Independent Researcher
- Laura Jacobs
- Department of Political Science, University of Antwerp, Antwerpen 2000, Belgium
- Jannes Jacobsen
- Cluster ‘Data-Methods-Monitoring’, German Center for Integration and Migration Research (DeZIM),
- Bastian Jaeger
- Department of Social Psychology, Tilburg University, Tilburg 5037AB, Netherlands
- Sebastian Jungkunz
- Institute of Political Science and Sociology, University of Bonn, Bonn 53111, Germany
- Nils Jungmann
- Survey Data Curation, GESIS—Leibniz Institute for the Social Sciences, Mannheim 50667, Germany
- Jennifer Kanjana
- Mathias Kauff
- Department of Psychology, Medical School Hamburg, Hamburg 20457, Germany
- Salman Khan
- Economics, University of Illinois, Chicago, Chicago, IL, USA
- Sayak Khatua
- School of Public Policy, Oregon State University, Corvallis 97330, USA
- Manuel Kleinert
- Institute of Sociology, Justus Liebig University Giessen, Giessen 35394, Germany
- Julia Klinger
- Independent Researcher
- Jan-Philipp Kolb
- Statistisches Bundesamt, Statistisches Bundesamt Wiesbaden, Wiesbaden 67549, Germany
- Marta Kołczyńska
- Department of Research on Social and Institutional Transformations, Institute of Political Studies of the Polish Academy of Sciences, Warsaw 00-625, Poland
- John Kuk
- Department of Political Science, Michigan State University, East Lansing 48823, USA
- Katharina Kunißen
- Institute of Sociology, Johannes Gutenberg University Mainz, Mainz 55128, Germany
- Dafina Kurti Sinatra
- Center for Evaluation, Independent Researcher (Formerly Uni Cologne),
- Alexander Langenkamp
- Institute of Sociology, Goethe-University Frankfurt, Frankfurt 60323, Germany
- Robin C. Lee
- Department of Sociology, Princeton University, Princeton, USA
- Philipp M. Lersch
- Socio-Economic Panel, German Institute for Economic Research, Berlin 10117, Germany
- David Liu
- Lea-Maria Löbel
- Socio-Economic Panel, German Institute for Economic Research, Berlin 10117, Germany
- Philipp Lutscher
- Department of Political Science, University of Oslo, Oslo 0851, Norway
- Matthias Mader
- Department of Philosophy, Politics and Economics, Witen/Herdecke University, Witten 58488, Germany
- Joan E. Madia
- Department of Primary Care and Health Sciences, University of Oxford, Oxford OX11JD, England
- Natalia Malancu
- Swiss Forum for Migration and Population Studies, University of Neuchatel, Neuchâtel 1205, Switzerland
- Luis Maldonado
- Instituto de Sociologia, Pontifical Catholic University of Chile, Santiago 7820436, Chile
- Helge Marahrens
- Massive Data Institute, Georgetown University, Washington D.C. 20057, USA
- Nicole Martin
- Department of Politics, University of Manchester, Manchester M19JS, UK
- Paul Martinez
- Department of Institutional Research, Western Governors University, Millcreek 84107, USA
- Jochen Mayerl
- Institute of Sociology, Chemnitz University of Technology, Chemnitz 09126, Germany
- Oscar J. Mayorga
- Director of Data for Freedom, Equity Research Cooperative, 19107
- Robert McDonnell
- Patricia McManus
- Department of Sociology, Indiana University Bloomington, Bloomington, IN 47405, USA
- Kyle McWagner
- Department of Political Science, University of Wisconsin-Milwaukee, Milwaukee, WI 53211, USA
- Cecil Meeusen
- Department of Sociology, Center for Sociological Research, KU Leuven, 3000
- Daniel Meierrieks
- Research Unit Migration, Integration, Transnationalization, WZB Berlin Social Science Center, Berlin 10785, Germany
- Jonathan Mellon
- Department of Politics, Westpoint Department of Systems Engineering, M19 2JS
- Friedolin Merhout
- Department of Sociology and Centre for Social Data Science, University of Copenhagen, 1353
- Samuel Merk
- Department of School Development, Karlsruhe University of Education, Karlsruhe 76133, Germany
- Daniel Meyer
- Competence Centre for Regional Development, Federal Institute for Research on Building, Urban Affairs and Spatial Development (BBSR), 03048
- Leticia Micheli
- Department of Social, Economic & Organisational Psychology, Leiden University, Leiden 2333AK, The Netherlands
- Jonathan Mijs
- Department of Sociology, Boston University, Boston, MA 02215, USA
- Cristóbal Moya
- Socio-Economic Panel, Institute for Economic Research, Berlin 10117, Germany
- Marcel Neunhoeffer
- School of Social Sciences, LMU Munich, Munich, Germany
- Daniel Nüst
- Department of Geosciences, Technische Universität Dresden, Dresden 01069, Germany
- Olav Nygård
- Division of Migration, Ethnicity and Society (REMESO), Linköping University, 60174
- Fabian Ochsenfeld
- Administrative Headquarters, Max Planck Society, Munich 80539, Germany
- Gunnar Otte
- Institute of Sociology, Johannes Gutenberg University Mainz, Mainz 55128, Germany
- Anna Pechenkina
- Department of Political Science, Utah State University, Logan, UT 84321, USA
- Mark Pickup
- Political Science, Simon Fraser University, Canada
- Christopher Prosser
- Louis Raes
- Department of Economics, Tilburg University, Tilburg 5037 AB, The Netherlands
- Kevin Ralston
- School of Social and Political Science, University of Edinburgh, Edinburgh EH8 9LD, Scotland
- Miguel Ramos
- Department of Social Policy, Sociology and Criminology, University of Birmingham, Birmingham B15 2TT, UK
- Frank Reichert
- Arne Roets
- Department of Developmental, Personality, and Social Psychology, Ghent University, Sint-Pietersnieuwstraat, B-9000, Belgium
- Jonathan Rogers
- School of Law, Empirical Research Group, University of California, Los Angeles, Los Angeles, CA, USA
- Guido Ropers
- Independent Researcher
- Robin Samuel
- Department of Social Sciences, University of Luxembourg, 4366, Luxembourg
- Gregor Sand
- SHARE Operations, SHARE Berlin Institute, Berlin 10115, Germany
- Constanza Sanhueza Petrarca
- School of Politics and International Relations, Australian National University, Canberra 2132, Australia
- Ariela Schachter
- Department of Sociology, Washington University in St. Louis, St. Louis MO 63130, USA
- Merlin Schaeffer
- Department of Sociology and Centre for Social Data Science, University of Copenhagen, 1353
- David Schieferdecker
- Institute for Media and Communication Studies, Freie Universität Berlin, Berlin 14195, Germany
- Elmar Schlueter
- Institute of Sociology, Justus Liebig University Giessen, Giessen 35394, Germany
- Katja Schmidt
- Department of Social Sciences, Humboldt University Berlin, Berlin, 10117, Germany
- Regine Schmidt
- Faculty of Social Sciences, Economics, and Business Administration, University of Bamberg, Bamberg 96052, Germany
- Alexander Schmidt-Catran
- Institute of Sociology, Goethe-University Frankfurt, Frankfurt 60323, Germany
- Claudia Schmiedeberg
- Department of Sociology, LMU Munich, Munich 80801, Germany
- Jürgen Schneider
- Teacher and Teaching Quality, Leibniz Institute for Research and Information in Education, Frankfurt 60323, Germany
- Martijn Schoonvelde
- Chair group European Politics and Society, University of Groningen, Groningen 9712 EK, Netherlands
- Julia Schulte-Cloos
- Department of Political Science, University of Marburg, Marburg 35037, Germany
- Sandy Schumann
- Department of Security and Crime Science, University College London, London WC1E 6BT, UK
- Reinhard Schunck
- School of Human and Social Sciences, University of Wuppertal, Wuppertal 42119, Germany
- Julian Seuring
- Department Migration, Leibniz Institute for Educational Trajectories, Bamberg 96047, Germany
- Henning Silber
- Institute for Social Research, University of Michigan, Ann Arbor MI 48109, USA
- Willem Sleegers
- Department of Social Psychology, Tilburg University, Tilburg 5037AB, Netherlands
- Nico Sonntag
- Institute of Sociology, Johannes Gutenberg University Mainz, Mainz 55128, Germany
- Alexander Staudt
- Nadia Steiber
- Department of Sociology, University of Vienna, Vienna 1090, Austria
- Nils D. Steiner
- Institute for Political Science, Johannes Gutenberg University Mainz, Mainz 55099, Germany
- Sebastian Sternberg
- Independent Researcher
- Dieter Stiers
- Centre for Political Science Research, KU Leuven, Leuven 3000, Belgium
- Dragana Stojmenovska
- Department of Sociology, University of Amsterdam, Amsterdam 1012 WP, Netherlands
- Nora Storz
- Expert Council on Intergration and Migration, Berlin 10178, Germany
- Erich Striessnig
- Department of Sociology, University of Vienna, Vienna 1090, Austria
- Anne-Kathrin Stroppe
- Survey Data Curation, GESIS—Leibniz Institute for the Social Sciences, Mannheim 50667, Germany
- Jordan W. Suchow
- School of Business, Stevens Institute of Technology, Hoboken 07030, USA
- Janna Teltemann
- Institute for Social Sciences, University of Hildesheim, Hildesheim 31141, Germany
- Andrey Tibajev
- Department of Women's and Children's Health, Uppsala University, Uppsala SE-751 05, Sweden
- Brian Tung
- Department of Sociology, Washington University in St. Louis, St. Louis 63130, USA
- Giacomo Vagni
- Social Research Institute (UCL), University College London, London WC1E 6BT, USA
- Jasper Van Assche
- Center for Social and Cultural Psychology (CESCUP), Université Libre de Bruxelles, Brussels, BE-1050, Belgium
- Meta van der Linden
- Department of Interdisciplinary Social Science, Utrecht University, Utrecht 3584CH, Netherlands
- Jolanda van der Noll
- Department of Psychology, University of Hagen, Hagen 58097, Germany
- Arno Van Hootegem
- Department of Sociology and Human Geography, University of Oslo, Oslo 851, Norway
- Stefan Vogtenhuber
- Education and Employment, Institute for Advanced Studies, Vienna 1080, Austria
- Bogdan Voicu
- Research Institute for Quality of Life, Romanian Academy, Bucharest, 010071, Romania
- Fieke Wagemans
- Beleidsvisies, Burgervisies en Gedragingen (Policy Perspectives, Citizen Perspectives, and Behaviors), Netherlands Institute for Social Research, The Hague 2594, Netherlands
- Nadja Wehl
- Research Cluster ‘The Politics of Inequality’, University of Konstanz, Konstanz 78464, Germany
- Hannah Werner
- Department of Political Science, University of Zurich, Zürich 8050, Switzerland
- Brenton M. Wiernik
- Independent Researcher
- Fabian Winter
- Mechanisms of Normative Change, Max-Planck-Institute for Research on Collective Goods, Bonn 53113, Germany
- Christof Wolf
- GESIS Leibniz-Institute for the Social Science & University of Mannheim, Mannheim 68159, Germany
- Cary Wu
- Department of Sociology, York University, Toronto M3J 1P3, Canada
- Yuki Yamada
- Faculty of Social Sciences, Economics, and Business Administration, University of Bamberg, Bamberg 96052, Germany
- Björn Zakula
- Nan Zhang
- Faculty of Arts and Science, Kyushu University, Fukuoka 819-0395, Japan
- Conrad Ziller
- Department of Political Science, University of Duisburg-Essen, Duisburg-Essen 47057, Germany
- Stefan Zins
- Institute for Employment Research, Federal Employment Agency, Nuremberg 90478, Germany
- Tomasz Żółtak
- Department of Computational Social Sciences, Institute of Philosophy and Sociology of the Polish Academy of Sciences, Warsaw 00-330, Poland
- Hung H.V. Nguyen
- Political Science, University of Bremen, Bremen 28359, Germany
- DOI
- https://doi.org/10.1098/rsos.241038
- Journal volume & issue
-
Vol. 12,
no. 3
Abstract
This study investigates researcher variability in computational reproduction, an activity for which it is least expected. Eighty-five independent teams attempted numerical replication of results from an original study of policy preferences and immigration. Reproduction teams were randomly grouped into a ‘transparent group’ receiving original study and code or ‘opaque group’ receiving only a method and results description and no code. The transparent group mostly verified original results (95.7% same sign and p-value cutoff), while the opaque group had less success (89.3%). Second-decimal place exact numerical reproductions were less common (76.9 and 48.1%). Qualitative investigation of the workflows revealed many causes of error, including mistakes and procedural variations. When curating mistakes, we still find that only the transparent group was reliably successful. Our findings imply a need for transparency, but also more. Institutional checks and less subjective difficulty for researchers ‘doing reproduction’ would help, implying a need for better training. We also urge increased awareness of complexity in the research process and in ‘push button’ replications.
Keywords
