Keywords: editorial, metascience, science reform

In most industries, error is a problem. Take, for instance, the aviation industry. An error in the calculation of a plane’s gross weight could be catastrophic, as how much a plane weighs directly influences how it needs to be configured for take-off and landing. Countless fatal aviation accidents underscore the importance of error-free weight calculations. The crash of Air Midwest flight 5481 in 2003, involving 22 human fatalities (National Transportation Safety Board, 2004) is just one example. Errors in communication between a pilot and an air-traffic controller can also lead to serious problems for safe aviation. Poor communication is listed as one of the primary causes of the accident with the highest number of human fatalities in the world to date – the Tenerife Airport disaster of 1977, in which a KLM aircraft struck a taxiing Pan Am jumbo, killing 583 people (McCreary et al., 1998).

Although the stakes and scope tend to be radically different in comparison with aviation, error can also be a problem in science. The nature of the error is important for understanding how detrimental its effect may be. While errors such as making mistakes in reporting statistical tests or mislabeling graph axes are undesirable and sloppy, it is unreported error that can truly undermine research quality. Unreported errors, such as inappropriate handling of missing data, errors in data processing, and undisclosed bias can lead to, and likely has led to, an unreliable literature body in all scientific disciplines. Consider well known examples such as the 1998 Lancet study on vaccines and autism, or the 2006 Potti Affair in cancer research (note, both of these articles have been retracted from the scientific literature). These errors and biases ultimately cause a flawed understanding of our universe and its inhabitants both for us and for future generations.

Consider the number of article retractions that occur every year. In 2023 alone, more than 10,000 articles were retracted from the scientific record — a record number, far exceeding previous years, and representing a trebling in numbers in the past decade (compare for instance, van Noorden’s 2023 report that in 2022, just over 4,000 articles were retracted). Some see these numbers as evidence of a self-correcting science. Another perspective, however, is that these articles represent only a fraction of the literature that should be retracted from the scientific record – a fraction of the ‘error’ in the literature that still gets cited and still undermines the foundations of science.

Reframing Error as Constructive

But what if error could be informative? Certainly, the commercial aviation industry has constantly and greatly evolved through seeing error as being informative. As a result of the Air Midwest flight 5481 disaster, the American Federal Aviation Administration revised erroneous estimated weight values (which had not been reviewed since 1936; National Transportation Safety Board, 2004). As a consequence of the Tenerife disaster, new requirements for a standard aviation phraseology, as well as air traffic instruction and response, were implemented across the world, among other changes (see Helmreich et al., 2017). In the case of aviation, learning from error in this context comes at a great cost of human lives. Luckily that is not typically true in the research sphere.

Provided that mistakes are reported and used as learning opportunities, we at the Journal of Trial & Error (JOTE) argue that error should be embraced as a normal part of the scientific process. We argue that far from error hurting the scientific process, it can in fact enrich, and even move it forward. Error in science – when thoroughly and transparently documented and contextualized – can help us improve how we conduct scientific research (through finding out and reflecting on how the scientific process shouldn’t work) and can even help us discover boundary conditions for effects and phenomena. While most journals only tend to publish error-free accounts of research, JOTE provides an outlet for publishing when things do go wrong, or do not go as expected. Our objective is to make these errors informative and prompt reflection for the scientific community, as well as to normalize discussion and disclosure of error. We approach the scientific enterprise pragmatically and critically, recognizing that the research process is rarely smooth, linear, and mistake-free.

In the same vein, JOTE also seeks to stimulate a fundamental aspect of cumulative science that has been de-emphasized (and even somewhat eliminated) in many other academic venues. Namely, the possibility of genuine scientific debate through a variety of publication formats, as if the scientific literature should function as a ‘book of conversations’ (Davis-Stober et al., 2024). In this effect, JOTE supports a plethora of different ways of doing and discussing science, be it from an empirical, conceptual, or meta-perspective standpoint. Furthermore, the journal seeks to go beyond the usual lens of published work by inviting reports on internal scientific processes, such as rejected grant applications or corrigenda and errata. JOTE regularly invites metascientists and humanities scholars to contribute their reflections (in the form of an independent article) on each recently published paper in JOTE. These reflections are also forwarded to the wider audience of scientists to add contributions. All these approaches are intended to fulfill the original idea of science on which its cumulative endeavor is based: an open, curious, and critical conversation, incorporating a diversity of rational perspectives (Field et al., 2024; Salmieri, 2024).

Changes at JOTE

Changes to the JOTE Team

Maura Burke served as JOTE’s second editor-in-chief from 2020. In 2024, her developing career in academia led her to new challenges outside of the journal. In her place, JOTE is proud to present the new editor-in-chief, Dr. Sarahanne M. Field. In what follows, Dr. Field shares her plans for the journal going forward. Speaking of her involvement so far, she says:

In May 2023 I was proud to join the journal as the editor of the then-new metascience section; proud to be part of an organization which was trying to shift how we see error in science. I was proud in explicitly helping JOTE onto the metascience map. This year, I am honored to step into the role of the journal’s Editor-in-Chief, to continue to bring it further in its aims alongside my colleagues. In stepping into this role, I have the opportunity to introduce three changes to how we operate at JOTE, all of which, in my opinion, allow the journal to better provide value not only to the metascience community, but to the broader scientific sphere.

Registered Report Submission Format for JOTE

First, as of March 25, 2024, JOTE began a one-year trial of the registered report (RR) submission type. Doing an RR involves carefully documenting the plans for a research study and getting it peer-reviewed before the study is conducted. Once the plan receives positive peer review (typically after at least one review round), the study plan receives what is known as in-principle acceptance (IPA). A decision of IPA means that as long as the study adheres to the plan (or clearly describes deviations), the completed study will be published (Field et al., 2020).

Introducing the RR submission format strengthens our commitment to normalizing the discussion of error in science, in that it provides a way to publish well-planned studies regardless of the outcomes. It doesn’t matter if the hypotheses aren’t supported by the empirical findings – if you conduct a high-quality planned study and transparently report on what happened, it will be published. It should be emphasized that the RR format is no panacea. It does not fix all the problems we face in research, indeed, if people want to conduct poor-quality research, they can (although we will not be publishing it)! Nevertheless, it can be a powerful tool in the hands of researchers who are motivated to conduct more reliable and valid studies.

Motivated researchers benefit from not having to worry about whether they find support for their hypotheses or not, can avoid the undue influences of hindsight and confirmation bias, and can test whether their planned study might be good enough to be part of the scientific record before they do the work of conducting the study (Field et al., 2020). The literature body benefits as well, from articles that tell the whole story of research rather than just the nice, perfect parts. Members of the public, in turn receive trustworthy information about themselves and the world in which they exist, so long as the media accurately represent study results – a contingency that unfortunately cannot be relied upon. Finally, the benefits of the RR format can be conferred to other researchers. Researchers can avoid conducting studies that will end up being uninformative and wasting time and money (resources that are scarce and hard-come-by in academia). To do so, the mismatch between what is researched and what is published should be kept to a minimum (the overarching goal of JOTE, and a likely outcome of more and more RRs in the scientific record; see Devine et al., 2020, and Field et al., 2020, respectively).

Streamlined Review

The second change we will be making to submission at JOTE is to introduce the streamlined review submission type, which is also being implemented in Collabra: Psychology (the first scientific publishing outlet to do so, to our knowledge). Authors can now request a streamlined review process for articles they submit to JOTE. The streamlined review process allows for articles that have undergone peer review at another journal, but which have been rejected by that journal for reasons unrelated to rigor or methodological soundness or which have been withdrawn by the author for improper journal conduct. Usually, such articles will have been rejected for lack of novelty or impact, or for reporting inconclusive results.

JOTE’s approach to research dissemination attempts to break the mold of traditional publishing in many ways. Part of this involves publishing articles that do not fulfill traditional criteria, such as being groundbreaking or following the expected trajectory of supporting the stated hypotheses exactly as expected. To the editors of JOTE, an article has earned a place in the research record if it is scientifically sound, provides scientific value (including informative failures to replicate, or failures to support study hypotheses) and, crucially, communicates these qualities effectively and truthfully to the research community.

When authors submit an article for streamlined review, they must submit all documentation of the previous review process, including editorial decisions, the reviews, and responses to reviews. These may be submitted along with the manuscript itself. We also require a cover letter describing what has occurred so far in terms of the previous peer review process and what is known about why the article was rejected. Additionally, authors should provide the name of the previous journal, and whether or not the editor(s) and reviewers have given permission for the review documents to be made openly available at JOTE alongside the article, if it is published. The editor-in-chief will assess streamlined review submissions on a case-by-case basis, and the emphasis will be on reusing the existing reviews, rather than engaging with the reason for why the article did not proceed to publication at the previous outlet. More details about the process can be found on JOTE’s submission guidelines page.

JOTE Introduces Fully Open Peer Review

The third change that we are implementing concerns open peer review. It is a smaller-scale shift in our operation, but nonetheless an important one. Until now, the default peer review option in our system was double-blind, masking both the identity of the authors and reviewers. Double-blind peer review undoubtedly has its benefits. For instance, complete anonymity is thought to reduce bias (though in practice, depending on how small or specialized a scientific discipline is, complete anonymity is often an unachievable goal). However, a lack of transparency leads to a lack of accountability for both authors and peer reviewers. We are committed to transparency in the peer review process and aim to allow reviewers to get recognition for their work. While our system has yet to be configured to formally require that peer-reviewers identify themselves, this change is in progress and will be realized.

To implement open peer review, we will require that both authors’ and reviewers’ identities are known during the peer review process. This aligns well with our existing practices of allowing readers to comment on articles via PubPub, encouraging the use of a preprint server for the unpublished manuscript, and requiring that authors either share their empirical data or give us permission to share it on their behalf (in line with the GDPR and other ethical guidelines). We are also working on making all peer reviews and response letters open, accompanying published articles. In our view, transparency and accountability are crucial parts of normalizing error and the human element that underpins the scientific enterprise.

A Closing Comment from the New Editor-in-Chief

We close this editorial with a final remark from our new editor-in-chief, Dr. Sarahanne M. Field:

I follow an inspiring and successful predecessor, Maura Burke, who has served as the EiC of JOTE since 2020. Maura’s initial work at JOTE is an incredibly hard act to follow, and the scientific community at large owes her and the rest of our colleagues at JOTE a debt for the work they have already done, paving the way for normalizing and embracing error, as the first journal of its kind in our industry. I thank my colleagues at JOTE for giving me the opportunity to take up this role, and for trusting me with the heavy responsibility it entails. I hope that I can serve both the journal and the wider research community well at the helm of this unique and avant-garde publishing outlet in the coming years.

For full transparency, and in the spirit of scientific integrity, I wish to disclose that I receive a small, taxed financial stipend for my work in the role of editor-in-chief for this journal.

References

Devine, S., Bautista-Perpinya, M., Delrue, V., Gaillard, S. D. M., Jorna, T. F. K., van der Meer, R. M., Millett, L., Pozzebon, C., & Visser, J. (2020). Science fails. Let’s publish. Journal of Trial and Error, 1(1), 1-5. https://doi.org/10.36850/ed1

Field, S. M., Kiers, H., & Derksen, M. (2024). Exploring the constellation of communities shaping science reform and its future progress. Preprint under review.

Field, S. M., Wagenmakers, E. J., Kiers, H. A., Hoekstra, R., Ernst, A. F., & van Ravenzwaaij, D. (2020). The effect of preregistration on trust in empirical research findings: Results of a registered report. Royal Society Open Science, 7(4), Article 181351. https://doi.org/10.1098/rsos.181351

Helmreich, R. L., Merritt, A. C., & Wilhelm, J. A. (2017). The evolution of crew resource management training in commercial aviation. In R. K. Dismukes (Ed.), Human error in aviation (pp. 275-288). Routledge. https://doi.org/10.4324/9781315092898-15

McCreary, J., Pollard, M., Stevenson, K., & Wilson, M. B. (1998). Human factors: Tenerife revisited. Journal of Air Transportation World Wide, 3(1), 23-32.

National Transportation Safety Board. (2004). Loss of pitch control during takeoff Air Midwest Flight 5481 (NTSB/AAR-04/01). Federal government of the United States of America. https://www.ntsb.gov/investigations/AccidentReports/Reports/AAR0401.pdf

Potti, A., Dressman, H. K., Bild, A., Riedel, R. F., Chan, G., Sayer, R., Cragun, J., Cottrill, H., Kelley, M. J., Petersen, R., Harpole, D., Marks, J., Berchuck, A., Ginsburg, G. S., Febbo, P., Lancaster, J. M., Nevins, J. R., & Nevins, J. R. (2006). Genomic signatures to guide the use of chemotherapeutics. Nature Medicine, 12(11), 1294-1300. https://doi.org/10.1038/nm1491 (Retraction published 2011, Nature Medicine, 17(1), 135)

Salmieri, G. (2024). Free speech as a right and a way of life. In T. Smith (Ed.), The first amendment: Essays on the imperative of intellectual freedom (pp. 193-239). Ayn Rand Institute Press.

van Noorden, R. (2023). More than 10,000 research papers were retracted in 2023 — a new record. Nature, 624, 479-481. https://doi.org/10.1038/d41586-023-03974-8

Wakefield, A. J., Murch, S. H., Anthony, A., Linnell, J., Casson, D. M., Malik, M., Berelowitz, M., Dhillon, A. P., Thomson, M. A., Harvey, P., Valentine, A., Davies, S. E., & Walker-Smith, J. A. (1998). Ileal-lymphoid-nodular hyperplasia, non-specific colitis, and pervasive developmental disorder in children. The Lancet, 351(9103), 637-641. https://doi.org/10.1016/s0140-6736(97)11096-0 (Retraction published 2004, The Lancet, 363(9411), 750; 2010, The Lancet, 375(9713), 445)