This work is licensed under the terms of the Creative Commons Attribution 4.0 (CC-BY 4.0) license. You may reuse, remix, and share all parts of this work for any purpose. given that you provide appropriate credit, provide a link to the license, and indicate if changes were made.
The year 2011 ushered in a challenging era for the behavioral and social sciences (Wagenmakers, 2012). High-profile fraud cases, wide scale failures to reproduce some of psychology’s keystone studies, and reports of many researchers admitting to having engaged in research misconduct (John, Loewenstein & Prelec, 2012) all painted a grim picture of science.
In response to these concerns, a movement has arisen to reform science (Munafo et al., 2017; Spellman, Gilbert & Corker, 2018). This movement has led to initiatives that intend to improve research methods, decrease misconduct, and make science more transparent. Studies that conduct direct replications of previous findings and articles reporting null results have become easier to publish. Preregistration (in which authors register their study plans in advance of conducting a study) and registered reports (in which a research proposal is peer-reviewed and accepted for publication before a study is conducted) have grown exponentially in popularity. And more parts of science are now “open,” such as preprints, open-access publications, open peer review, and openly available data and code.
These initiatives generate clear and direct benefits for both science and scientists. At the same time, the scientific reform movement is still in its infancy and the long-term effects of many interventions have not been evaluated. The situation is made even more difficult by the fact that academic science is a complex system, and the downstream consequences of modifying complex systems are notoriously difficult to anticipate.
Several scholars have expressed concerns about these issues. In a 2021 article on scientific reform methodology, Devezer, Navarro, Vandekerckhove and Buzbas argue that the reform policies have “little evidentiary backing” and are “based on methods which are suggested with no framework for assessing their validity or evaluating their efficacy’’ (p. 1). Others (Ioannidis, 2014; Tiokhin et al., 2021) have expressed related sentiments, particularly regarding the potential for unintended consequences of otherwise well-intentioned reforms
To have a better chance of generating interventions that improve scientific practice, to mitigate the effects of unintended consequences, and to maintain its credibility as a movement open to self-criticism, the scientific reform movement can benefit by continually interrogating its practices and proposed initiatives.
by Arielle Bennett, Daniel Garside, Cassandra Gould van Praag, Thomas J. Hostler, Ismael Kherroubi Garcia, Esther Plomp, Antonio Schettino, Samantha Teplitzky, and 1 more
