Replication Crisis

The scientific Replication and Reproducibility Under the Microscope

Read

Published on the 22 April 2020

For years, a growing alarm has been building up about results that cannot be reproduced because of scientific method.

The replication crisis, pinpointed around 2010, focus on the difficulty or even impossibility to reproduce many scientific studies. This issue, though well-known (>50% awareness), has spread from different factors. Causes are both linked to the pressure to publish, inadequate training of scientists and the lack of clear guidelines. But initiatives coming from all the actors of the scientific community start to emerge.

1. Replication crisis: what is it?

2. What are the causes?

3. How to address it?

1. Replication crisis: what is it?

The replication crisis is defined as a methodological crisis in which it has been found that many scientific studies are difficult or impossible to replicate or reproduce.

It is now a well-known issue: in a survey published by Nature in 2019, out of 1576 surveyed researchers, 819 (i.e. 52%) acknowledged there is a significant crisis¹.

"70% of interviewed scientists failed to reproduce someone else’s experiment."

Most scientists have experience failure to reproduce results: the same Nature survey presents the rate of reproducibility failure in different fields. For biology, more than 70% of interviewed scientists failed to reproduce someone else’s experiment, and 60% failed to reproduce their own experiment.

For example, the story of three biologists that have found out discrepancies that prevented them to reproduce their own study²: https://www.nature.com/news/a-long-journey-to-reproducible-results-1.22478

The reproducibility of experimental results plays an essential part in the validation of science advances, whether for inter-laboratories share of knowledge or for the transfer of discoveries in production. As protocols.io says “Scientists are constantly re-discovering knowledge that others have not had the time to publish and improving existing methods without the ability to share the improvements.”

2. What are the causes?

Many factors impact the ability to perform rigorous and reproducible research. The Committee on Core Rigor and Reproducibility (CCoRRe) from the Association of Biomolecular Resource Facilities (ABRF), has listed 9 main categories of factor³‘s.

The top three are:

Lack of training or expertise,
Time pressure,
Inadequate standardization of protocols, guidelines and data analysis, sometimes called “pre-producibility”⁴.

What is required to reproduce? Indeed, the researchers may find difficult to know which parameters to record and share. Due to a lack of training and awareness, it is common to see inaccurate descriptions about reagents and methods used to perform the experiments.

This may also come from the high turn-over in the Academic field (PhD finishing their research and leaving, short time contracts…), meaning that the knowledge of the experiment conditions are not always passed when the person that has carried it out leaves the lab.

“The experiment conditions are not always passed when the person that has carried it out leaves the lab…”

Guillermo Marques, from the University of Minnesota, during an excellent talk⁵ at ABRF 2020, explained the difficulty of many researchers to know which imaging conditions to report in their scientific papers. They surveyed 185 biomedical articles that contained imaging, and only 17% of them reported the image acquisition conditions correctly.

Some dynamics might also distort practices: the pressure, whether of time or number of publications, may also be a factor that leads researchers to involuntarily focus on the results rather than to precise experiments conditions.

3. How to address it?

Even though reproducibility is an increasing concern for researchers, it is not as easy to address the issue when faced with it at their home institutions.

But as Nature’s surveys states, “Journals, funders and research institutions that advance policies to address the issue would find cooperation.”

Indeed, the support of formal institutions such as NIH, through uniform guidelines are helpful to guide researchers in the implementation process or correction of their methods⁶.

Major editors are also well-aware of that issue, and most of them have started working on guidelines and methods to help scientists provide accurate acquisition data. Cell Press introduced in 2016 the STAR methods for example⁷.

In Core facilities, with the support of funders, the core staff, who is already working hard to train users, could continue its education role on what to report and why.

Screenshot from Nature's video "Is there a reproducibility crisis in science?"

The Committee on Core Rigor and Reproducibility (CCoRRe) from the Association of Biomolecular Resource Facilities (ABRF) in the USA, or the recent initiatives from the German BioImaging Network⁸ are great example to follow.

There are many other leads to correct this crisis, both as individual and collective effort. All those steps are slowly leading to an inevitable change in culture, that value reproducibility and includes rigor as a critical part of any research activity, rather than a painful and time-consuming operation.

Read the full portfolio in nature special, “Challenges in irreproducible research”: https://www.nature.com/collections/prbfkwmwvz

"How are you or your institution involved in the fight for reproducibility?"

You can send us a quick message through the chat box to share. Relevant initiatives will be added to the text.

Sources:

(1) M. Baker, « 1,500 scientists lift the lid on reproducibility», Nature 553, 452-454 (26 May 2016) doi:10.1038/533452a.
Download survey’s data : https://figshare.com/articles/Nature_Reproducibility_survey/3394951/1

(2) G. Lithgow, M. Driscoll and P. Phillips, « A long journey to reproducible results », Nature 548, 387–388 (24 August 2017) doi:10.1038/548387a.

(3) K. Knudtson, R. Carnahan, and al., « Survey on Scientific Shared Resource Rigor and Reproducibility », Journal of Biomolecul Techniques, 30(3): 36–44, Septembre 2019, doi: 10.7171/jbt.19-3003-001.

(4) P. Stark, « Before reproducibility must come pre-producibility», Nature 557, 613 (2018) doi: 10.1038/d41586-018-05256-0.

(5) G. Marques, T. Pengo, and M. Sanders, « Imaging in Biomedical Research: An Essential Tool with No Instructions », February 2020.

(6) https://grants.nih.gov/policy/reproducibility/index.htm

(7) https://www.cell.com/star-authors-guide

(8) https://www.gerbi-gmb.de/