Skip to main content
SearchLoginLogin or Signup

Consequences of the Scientific Reform Movement

Is the scientific reform movement headed in the right direction?

Published onMar 11, 2022
Consequences of the Scientific Reform Movement
·

In brief

Call for proposals

Call for contributions

APCs

None

Word limit

200

4000 (excl. references & abstract)

Due

May 18, 2022
1700h UTC+01:00

September 30, 2022
1700h UTC+01:00

Submit to

[email protected]

submit.jtrialerror.com

Special Issue: Call for Contributions 

The year 2011 ushered in a challenging era for the behavioral and social sciences (Wagenmakers, 2012). High-profile fraud cases, wide scale failures to reproduce some of psychology’s keystone studies, and reports of many researchers admitting to having engaged in research misconduct (John, Loewenstein & Prelec, 2012) all painted a grim picture of science.  

In response to these concerns, a movement has arisen to reform science (Munafo et al., 2017; Spellman, Gilbert & Corker, 2018). This movement has led to initiatives that intend to improve research methods, decrease misconduct, and make science more transparent. Studies that conduct direct replications of previous findings and articles reporting null results have become easier to publish. Preregistration (in which authors register their study plans in advance of conducting a study) and registered reports (in which a research proposal is peer-reviewed and accepted for publication before a study is conducted) have grown exponentially in popularity. And more parts of science are now “open,” such as preprints, open-access publications, open peer review, and openly available data and code.  

These initiatives generate clear and direct benefits for both science and scientists. At the same time, the scientific reform movement is still in its infancy and the long-term effects of many interventions have not been evaluated. The situation is made even more difficult by the fact that academic science is a complex system, and the downstream consequences of modifying complex systems are notoriously difficult to anticipate. 

Several scholars have expressed concerns about these issues. In a 2021 article on scientific reform methodology, Devezer, Navarro, Vandekerckhove and Buzbas argue that the reform policies have “little evidentiary backing” and are “based on methods which are suggested with no framework for assessing their validity or evaluating their efficacy’’ (p. 1). Others (Ioannidis, 2014; Tiokhin et al., 2021) have expressed related sentiments, particularly regarding the potential for unintended consequences of otherwise well-intentioned reforms

To have a better chance of generating interventions that improve scientific practice, to mitigate the effects of unintended consequences, and to maintain its credibility as a movement open to self-criticism, the scientific reform movement can benefit by continually interrogating its practices and proposed initiatives. 

In this call, we aim to inspire scholars to take heed and critically reflect on the direction in which the scientific reform movement is heading. Is scientific thought and practice being steered in a good direction? What are the known unknowns and unknown unknowns that challenge the field? Are there lessons that can be learned from other fields that have struggled with similar issues?

With the goal of stimulating a critical discussion in the field of scientific reform, we invite submissions that focus on “second-order” effects or “second-generation” challenges: issues that arise during or as a result of the resolution of a primary issue. We encourage a diversity of contributions, but all contributions should discuss a second-generation challenge for the scientific-reform movement. Examples of potential contributions include:

  • Historical and philosophical articles

  • Opinion pieces

  • Articles communicating insights from other fields

  • New data

  • Empirical syntheses

  • Formal theoretical models   

Potential topics include:

  • Unintended consequences

  • Tipping points in complex systems

  • Second-order responses to changing incentive structures

  • Insights from innovation sciences and cultural evolution about cumulative knowledge accumulation

  • Excessive use of scientific resources (on e.g., open access in high-profile journals)

  • Gaming open science

  • Improved methodological rigor being circumvented novel Questionable Research Practices 

  • Mindless check-box scientific practices

  • Issues with measuring scientific progress

  • The effects of reforms on scientific creativity 

Practical Information

Special Issue Editor-in-Chief

Sarahanne M. Field ([email protected])

Guest Editors

Leonid Tiokhin ([email protected])

Noah van Dongen ([email protected])

Proposal

Please submit a proposal for consideration prior to beginning work on your contribution. The proposal should contain a brief outline (200 words max) of your potential contribution to this special issue. We will respond to proposals once the due date has passed.

Proposals are due by 1700h UTC+01:00 (Amsterdam), May 18th, 2022.

Final contributions are limited to 4000 words, excluding references and abstract. Please ensure that you submit a proposal early enough to have time to complete your full contribution by the deadline. 

Full contributions are due by 1700h UTC+01:00 (Amsterdam), September 30, 2022.

Other general guidelines for submission to JOTE, and the submission portal itself can be found at: https://submit.jtrialerror.com/index.php/jote/about/submissions

Note that JOTE is a diamond open access journal. You will not be charged article processing fees.

Proposals and inquiries should be sent to Sarahanne M. Field at [email protected]

References

Devezer, B., Navarro, D. J., Vandekerckhove, J., & Ozge Buzbas, E. (2020). The case for formal methodology in scientific reform. Royal Society Open Science, 8(3), 200805.

Ioannidis JPA (2014) How to Make More Published Research True. PLOS Medicine 11(10): e1001747.

John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23(5), 524-532.

Munafò, M. R., Nosek, B. A., Bishop, D. V., Button, K. S., Chambers, C. D., Percie du Sert, N., ... & Ioannidis, J. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1(1), 1-9.

Spellman, B. A., Gilbert, E. A., & Corker, K. S. (2018). Open Science. Stevens' Handbook of Experimental Psychology and Cognitive Neuroscience, 5, 1-47.

Tiokhin, L., Panchanathan, K., Lakens, D., Vazire, S., Morgan, T., & Zollman, K. (2021). Honest signaling in academic publishing. PloS one, 16(2), e0246675.

Wagenmakers, E. J. (2012). A year of horrors. De Psychonoom, 27, 12-13.

Comments
0
comment
No comments here
Why not start the discussion?