Almost 90% of biomedical research is ‘wasteful and inefficient’, according to Yale‘s Dr Michael Bracken, speaking at a National Institutes of Health lecture.
The government agency, which spends over $30bn of taxpayer funding on research each year, said in a report “As much as 87.5% of biomedical research may be wasteful and inefficient,” quoting Bracken.
“Waste is more than just a waste of money and resources,” said Bracken, the Susan Dwight Bliss professor of epidemiology at Yale University School of Public Health. “It can actually be harmful to people’s health.”
The NIH said: “Researchers who publish novel findings are more likely to have produced exaggerated results or drawn incorrect conclusions. Many times, these early or ‘first’ studies are based on limited evidence from small studies. When the experiments are reproduced in larger studies, Bracken said the predictive value of the finding may decrease up to 90% of the time. While replicating studies is necessary, there is sometimes waste in unnecessarily repeating studies. Repeating studies when there is already evidence suggesting a treatment is effective, for example, ‘means patients are being submitted to placebo when they can receive active therapy,’ he noted.
Once several large-scale studies show a treatment is effective, he said, ‘you need an incredible amount of evidence to switch an estimated treatment effect from being protective to being even neutral, let alone a risk.’
To reduce wasteful duplication, Bracken suggested researchers conduct systematic reviews for their research protocols. This is a type of literature review that involves systematically collecting and analysing all of the relevant research papers. The goal is to summarise all the best currently available evidence. He advised scientists to conduct the reviews concurrently as research accumulates. Then, they would see if new studies are necessary in real-time. In human studies, he said, there are presently only some 408 systematic reviews published for every 10,000 publications.
Preclinical research, particularly in animal studies, is especially inefficient. Bracken said, for example, one of the cornerstones in clinical research – blinding investigators – is widely ignored in animal studies. Existing systematic reviews of animal studies show that very few studies are blinded or randomised. He believes animal study methodology is 40 years behind human clinical study design. ‘There is never any justification for the use of animals or humans in poorly designed studies,’ he said.
Bracken said there are strategies researchers can use to reduce the element of chance when designing studies. They can, for example, design larger studies based on more stringent statistical tests of the hypothesis. He says that the traditional 0.05 ‘p-value’ has lost its ability to discriminate important research findings, especially when numerous comparisons are being made, and can produce results that are irreproducible and in some examples actually harmful to the public’s health.
P-values are used to determine the strength of an association. The smaller the p-value, the stronger the evidence, while the larger the p-value, the weaker the evidence. However, a p-value less than 5% ‘does not by any means indicate that the association is not a chance finding.’
Researchers should also be aware that many sources of bias are being underestimated in drawing conclusions from biomedical research in both pre-clinical and clinical research. Bracken believes scientists can substantially reduce bias if they are aware of it. ‘Reducing bias is usually not an issue of cost. It’s an issue of understanding where bias may come from,’ he said.
Bracken applauded NIH for increasing its efforts to train young scientists on how to design and conduct experiments based on best practices. Bracken believes many scientists are incorporating these kinds of changes into their research, but thinks there is still room for considerable improvement.NIH material