In 2009, a now highly-cited study found an average of around 2% of scientists admitted to having falsified, fabricated, or modified data at least once in their career, causing shockwaves worldwide.
Fifteen years on, a new analysis tried to quantify how much science is fake – but the real number may remain elusive, some observers say.
The analysis, published before peer review on the Open Science Framework last week, found one in seven scientific papers may be at least partly fake.
The author, James Heathers, a long-standing scientific sleuth, arrived at that figure by averaging data from 12 existing studies – collectively containing a sample of around 75 000 studies – that estimate the volume of problematic scientific output.
“I have been reading for years and still continue to read this 2% figure, which is ubiquitous,” said Heathers, an affiliated researcher in psychology at Linnaeus University in Vaxjo, Sweden.
“The only problem is that it’s 20 years out of date,” he told Retraction Watch, noting that the last dataset that went into the 2009 study was from 2005.
So he tried to come up with a more up-to-date estimate of scholarly literature containing signs of irregularities. “A lot has changed in 20 years,” he said. “It’s been a persistent irritant to me for years now to see this figure cited over and over and over again.”
Past studies predominantly focused on asking researchers directly if they had engaged in dishonest research practices, Heathers said, “which I think is a very bad approach”. But he noted that it was probably the only method available when the research was conducted.
“I think it’s pretty naive to ask people who are faking research whether or not they’ll honestly answer the question that they were dishonest previously,” he pointed out.
Heathers’ study pulls data from 12 different analyses from the social sciences, medicine, biology, and other fields of research. All of those studies have one thing in common: the authors of each used various online tools to estimate the amount of fakery taking place in a set of papers.
“There’s a really persistent commonality to them,” Heathers said. “The rough approximation for where we end up is that one in seven research papers is fake.”
He decided to conduct his study as a meta-analysis because his figures are “far flung”.
“They are a little bit from everywhere; it’s wildly non-systematic as a piece of work.”
Daniele Fanelli, a metascientist at Heriot-Watt University in Edinburgh, Scotland, who authored the 2009 study, is not convinced by the new analysis.
“Meta-science research is sometimes not meta-scientific,” he said, arguing that the study falsely labels studies with some problem as definitely being a fake, and incorrectly lumps together others measuring different phenomena.
“The papers are all different, they’re all over the place in highlighting all sorts of different problems in all sorts of different contexts using all sorts of different methods,” Fanelli said. “That’s not a rigorous way to get an estimate of anything.”
Fanelli added that the study would draw unnecessary negative media attention: “It’s not the kind of attention that science either deserves or from which it will benefit.”
“I don’t think it’s entirely wrong but I think that it can be slightly misleading,” said Gowri Gopalakrishna, an epidemiologist at the Maastricht University in the Netherlands who co-authored a 2021 study that found 8% of researchers in a survey of nearly 7 000 scientists in the Netherlands confessed to falsifying or fabricating data at least once between 2017 and 2020.
Gopalakrishna said fabrication and falsification may be more prevalent in some fields than others, so grouping them together may not be helpful. “If you want to get the attention of the government and try to shake things up, putting them all together and saying, ‘look how big the problem is’, may be useful but I really think it’s important to drill down,” she said.
Heathers acknowledged those limitations but argued that he had to conduct the analysis with the data that exist.
“If we waited for the resources necessary to be able to do really big systematic treatments of a problem like this within a specific area, I think we’d be waiting far too long,” he said. “This is crucially underfunded.”
Heathers said he decided to pursue coming up with a figure for the average percentage of fakery in science because few such estimates are available.
“Even if you do something that’s an incredibly systematic review in a very formal sense, I strongly suspect you’ll get the same estimate that I’ve got,” he said.
OSF article – How much science if fake (Open access)
PLOS One article – How many scientists fabricate and falsify research? (Open access)
See more from MedicalBrief archives:
Medical research at stake as fake scientific papers balloon
Shady clinical trials, faked, flawed studies abound, experts warn