It is essential that scientific research findings can be reproduced independently. But, warn Oscar Flórez-Vargas and Michael Bramhall, this is often not possible.
The potential for the development of new treatments for diseases is being damaged by scientific research papers that are providing insufficient, and sometimes misleading, information. The incidence of these reporting weaknesses is increasing.
Large scale releases of data – such as those related to the human genome project – have led to greater interest in replicating research to develop new treatments. But published research findings are often impossible to replicate. Reported findings in animal models or at the laboratory bench are, increasingly, not being translated into successful human trials for drugs or vaccines against disease.
Cancer research has probably the highest failure rate. Recently, two biopharmaceutical companies – Amgen and Bayer HealthCare – attempted to replicate the findings of 53 landmark cancer studies. The results were confirmed in only six cases.
This lack of reproducibility is crippling the translation from animal models to human trials. According to Lisa Hutchinson and Rebecca Kirk, editors of Nature Reviews Clinical Oncology, only 5% of potential drugs that have shown promising anti-cancer activity in animal models move on to human clinical studies.
There are several possible reasons for the observed lack of reproducibility. These include over-interpretation, misinterpretation, or even falsification of data. The technical nature of the scientific method is extremely important in this respect, since it provides a firm footing for drawing conclusions.
As biomedical researchers we are aware of how extremely complex this field of science is. Nevertheless, its complexity goes beyond the biological nature itself. It includes a wide variety of protocols, sophisticated instruments, chemical substances and a growing staff of researchers. These and other factors could play an important role in determining why an experiment returns result ‘a’ rather than result ‘b’.
For example, early last year, Sorge et al. reported in Nature Methods that rodents show increased stress levels when handled by men rather than women, which could skew study results. Although the impact of various experimental factors is still unknown, there are many factors that we have clear evidence do affect findings, such as the sex and age of animal models.
Females tend to have more pro-inflammatory immune responses than male counterparts and therefore must be recognised as different. Clarity is essential when using experimental animals: sex of experimental animals should be both reported and factored into the experimental design. Bacteria that live in the gut have a huge impact on health and outcome to infection or injury and are affected by several factors, including environment, diet and parentage. Mice reared in one lab may respond differently to those brought in from another lab, irrespective of whether they have the same genetic background.
The Reproducibility Initiative – published in PLOS One in December 2014 -attempted to reproduce an experiment in which natural hormones from cows were shown to be efficient killers of the parasite Leishmania major. However, the team could not reproduce the original data without drastically increasing the dosage of hormones administered.
The authors of the original paper had not precisely described a specific modification that they had made to the structure of the hormone and incorrectly assumed that other scientists would make the same modification without being prompted. There is an assumption that experts in the field will understand which details are essential for an experiment to be performed reproducibly and thus what data is needed for proper interpretation.
However, increasingly people assess work outside of their field and so research has become more cross-disciplinary. Transparency about how experiments were conducted is therefore critical for impartial evaluation and reproducibility. All essential experimental details need to be stated in the methods section of a scientific paper; but in reality this is rarely done.
We realised the extent of the problem when we tried to integrate and analyse biomedical data from several sources. To integrate data from different experiments, we needed to know that an identical experimental method had been used. When we assessed the literature around trypanosomiasis – a widespread parasitic tropical disease – we found that only 65.5% of the essential information required to reproduce a Trypanosoma experiment was reported, meaning we could not reliably integrate and analyse this data.
A similar trend was observed in other parasite experiments such as Leishmania, Toxoplasma and Plasmodium. Our current paper, published in the journal Inflammatory Bowel Diseases, assessed animal models of inflammatory bowel disease and found that only one of the 58 articles we investigated reported all essential criteria on our checklist. On average, around 20% of the information needed to repeat the experiment was missing.
To know what makes an experimental detail essential for reproducing a particular finding we have taken inspiration from Picasso’s Bull. The artist deconstructs the process of drawing a bull in a series of increasingly simplistic lithographs, capturing the bare essence of what makes a bull. We followed the same approach with biomedical experimentation, stripping back complex experimental methods and protocols to reveal the bare essence required for a particular experiment.
Just as the bull missing its horns is no longer recognisable as a bull, the absence of an essential experimental component means we cannot correctly interpret the experiment. The essential parameters are selected based on scientific evidence about which parameters could influence the findings.
The BioSharing catalogue is a resource of standards for describing many experimental tests, for example Minimum Information About a Microarray Experiment. However, there is an urgent need for designing field-specific reporting guidelines that will help scientists to provide a full account of what they did in the laboratory and allow their readers to understand the context in which their data was produced.
This would represent an important step towards improving reproducibility and, hence, could speed up the progression of further research and translation into human clinical trials.
