Investigations suggest that, in some fields, at least one-quarter of clinical trials might be problematic or even entirely made up, warn some researchers, who are urging stronger scrutiny.
In October 2020, John Carlisle – an anaesthetist with England’s National Health Service, renowned for his ability to spot dodgy data in medical trials – reported a startling estimate.
Carlisle is also an editor at the journal Anaesthesia, and in 2017, had decided to scour every manuscript he handled that reported a randomised controlled trial (RCT), the gold standard of medical research. Over three years, he scrutinised more than 500 studies.
For more than 150 trials, Carlisle got access to anonymised individual participant data (IPD) – and judged that 44% of them contained at least some flawed data: impossible statistics, incorrect calculations or duplicated figures, reports Nature.
And 26% had problems so widespread the trial was impossible to trust, either because the authors were incompetent, or because they had faked data.
Carlisle called these “zombie trials” because they had the semblance of real research, but scrutiny showed they were actually hollow shells, masquerading as reliable information. He was surprised by their prevalence. “I anticipated maybe one in 10,” he says.
When he couldn’t access a trial’s raw data, however, he could study only the aggregated information in the summary tables. Just 1% of these cases were zombies, and 2% had flawed data, he judged.
This alarmed him, suggesting that, without access to the IPD – which journal editors usually don’t request and reviewers don’t see – even an experienced sleuth cannot spot hidden flaws.
“Journals should assume all submitted papers are potentially flawed and editors should review individual patient data before publishing randomised controlled trials,” Carlisle wrote in his report.
He rejected every zombie trial, but by now, almost three years later, most have been published in other journals – sometimes with different data from those submitted with the manuscript he had seen.
Do Carlisle’s findings in anaesthesiology extend to other fields? For years, various scientists, physicians and data sleuths have argued that fake or unreliable trials are widespread. They’ve scoured RCTs in fields like women’s health, pain research, anaesthesiology, bone health and Covid-19, and found dozens or hundreds with seemingly statistically impossible data.
Some say one-quarter of trials being untrustworthy might be an underestimate. “If you search for all randomised trials on a topic, about a third are fabricated,” said Ian Roberts, an epidemiologist at the London School of Hygiene & Tropical Medicine.
But faked or unreliable RCTs are particularly dangerous. They not only are about medical interventions, but also can be laundered into respectability by being included in meta-analyses and systematic reviews, which thoroughly comb the literature to assess evidence for clinical treatments.
Medical guidelines often cite such assessments, and physicians look to them when deciding how to treat patients.
Ben Mol, who specialises in obstetrics and gynaecology at Monash University in Melbourne, Australia, said 20%–30% of the RCTs included in systematic reviews in women’s health are suspect.
Many research-integrity specialists confirm the problem exists, but some doubt whether it’s as bad as the most alarming examples suggest.
“We have to recognise that, in the field of high-quality evidence, we increasingly have a lot of noise. Some good people are championing that and producing scary statistics. But there are also many who think it’s scaremongering,” said Žarko Alfirević, a specialist in foetal and maternal medicine at Liverpool University, UK.
Laundering fake trials
Roberts first encountered the issue when he co-authored a 2005 systematic review for the Cochrane Collaboration, which suggested high doses of a sugary solution could reduce death after head injury.
But Roberts retracted it after doubts arose about three of the key trials cited in the paper, all authored by the same Brazilian neurosurgeon, Julio Cruz. (Roberts never discovered whether the trials were fake, because Cruz committed suicide before investigations began. Cruz’s articles have not been retracted.)
A more recent example is that of Yoshihiro Sato, a Japanese bone-health researcher. Sato, who died in 2016, fabricated data in dozens of trials of drugs or supplements that might prevent bone fracture.
He has 113 retracted papers, shows a list compiled by Retraction Watch. His work has had a wide impact: researchers found 27 of Sato’s retracted RCTs had been cited by 88 systematic reviews and clinical guidelines, some of which had informed Japan’s recommended osteoporosis treatments.
Some of the findings in about half of these reviews would have changed had Sato’s trials been excluded, said Alison Avenell, a medical researcher at the University of Aberdeen, UK.
She and fellow researchers Andrew Grey, Mark Bolland and Greg Gamble, all at New Zealand’s University of Auckland, have pushed universities to investigate Sato’s work and monitored its influence.
The concerns over zombie trials, however, are beyond individual fakers flying under the radar. In some fields, swathes of RCTs from different research groups might be unreliable, researchers worry.
During the pandemic, for instance, numerous RCTs was conducted into whether Ivermectin could treat Covid-19. But researchers who were not involved have since pointed out data flaws in many of the studies, some of which have been retracted. A 2022 update of a Cochrane review argued that more than 40% of these RCTs were untrustworthy.
In maternal health, Roberts and Mol have flagged studies into whether a drug called tranexamic acid can stem dangerously heavy bleeding after childbirth. Annually, around 14m women experience this condition, and some 70 000 die: it’s the world’s leading cause of maternal death.
In 2016, Roberts reviewed evidence for using tranexamic acid to treat the condition, reporting that many of the 26 RCTs investigating the drug had flaws. Some had identical text, others had data inconsistencies or no records of ethical approval.
Some seemed not to have adequately randomised the assignment of their participants to control and treatment groups.
When he followed up with authors to ask for more details and raw data, he generally got no response or was told records were missing or had been lost via computer theft.
Fortunately, in 2017, a large, high-quality multi-centre trial, which Roberts helped to run, established that the drug was effective. It’s likely, says Roberts, that in these and other such cases, some of the dubious trials were copycat fraud – researchers saw a large trial was ongoing and produced small, substandard copies no one would question.
This isn’t a victimless crime, however. “It results in narrowed confidence intervals such that the results look much more certain than they are. It can also amplify a wrong result, suggesting treatments work when they don’t,” he said.
That might have happened for another question: what if doctors were to inject the drug into all women undergoing a Caesarean, just after they give birth, as a preventative measure? A 2021 review of 36 RCTs investigating this idea, involving more than 10 000 participants, concluded this would reduce heavy blood loss risk by 60%.
Yet this April, an enormous US-led RCT with 11 000 people reported only a slight, not statistically significant benefit.
Mol thinks problems with some of the 36 previous RCTs explain the discrepancy. The 2021 meta-analysis had included one French multi-centre study of more than 4 000 participants, which found a modest 16% reduction in severe blood loss, and another 35 smaller, single-centre studies, mostly in India, Iran, Egypt and China, which collectively estimated a 93% drop.
Many of the smaller RCTs were untrustworthy, said Mol.
The WHO recommends using tranexamic acid to treat blood loss after childbirth, but it doesn’t have a guideline on preventive administration.
From four trials to one
Mol highlights a different example where untrustworthy trials might have influenced clinical practice. In 2018, researchers published a Cochrane review on whether giving steroids to women due to undergo Caesarean-section births helped to reduce their babies’ breathing problems.
Steroids are good for a baby’s lungs but can harm the developing brain, said Mol; benefits generally outweigh harms when babies are born prematurely, but the balance is less clear when steroids are used later in pregnancy.
The authors of the 2018 review, led by Alexandros Sotiriadis, a specialist in maternal–foetal medicine at the Aristotle University of Thessaloniki in Greece, analysed the evidence for administering steroids to women delivering by C-section later in pregnancy.
They ended up with four RCTs: a British study from 2005 with more than 940 participants, and three Egyptian trials conducted between 2015 and 2018 that added another 3 000 people into the evidence base.
The review concluded the steroids “may” reduce rates of breathing problems; it was cited in more than 200 documents and some clinical guidelines.
In January 2021, however, Mol and others, raised concerns about the Egyptian trials. The largest study, with nearly 1 300 participants, was based on the second author’s thesis, he noted – but the trial end dates in the thesis differed from the paper. And the reported ratio of male to female babies was an impossible 40% to 60%.
Mol queried the other papers, too, and wrote to the authors, but did not get satisfactory replies. His team also reported issues with other works by the same authors.
In December 2021, Sotiriadis’s team updated its review, this time adopting a new screening protocol.
Until that year, Cochrane reviews had aimed to include all relevant RCTs; if researchers spotted potential issues, using a “risk of bias” checklist, they would downgrade their confidence in its findings, but not remove it from their analysis.
But in 2021, Cochrane’s research-integrity team introduced new guidance: authors should try to identify “problematic” or “untrustworthy” trials and exclude them from reviews.
Sotiriadis’s group now excluded all but the British research. With only one trial left, there was “insufficient data” to draw firm conclusions about the steroids, the researchers said.
By last May, as Retraction Watch reported, the large Egyptian trial was retracted. The journal’s editors said they had not received its data or a satisfactory response from the authors, that “if the data are unreliable, women and babies are being harmed”.
The other two trials are still under investigation by publisher Taylor & Francis as part of a larger case of papers.
Before the 2018 review, some clinical guidelines had suggested administering steroids later in pregnancy could be beneficial, and the practice had been growing in some countries, like Australia, Mol has reported. The latest updated WHO and regional guidelines, however, recommend against this.
Overall, Mol and his colleagues have alleged problems in more than 800 published medical research papers, at least 500 of which are on RCTs, leading to more than 80 retractions and 50 expressions of concern.
Screening for trustworthiness
“Mol has undoubtedly been a pioneer in detecting and fighting data falsification,” said Sotiriadis – but added it is difficult to prove a paper is falsified. He said he didn’t depend on Mol’s work when his team excluded those trials in its update, and he can’t say whether the trials were corrupt.
Instead, his group followed a screening protocol designed to check for “trustworthiness”, developed by the Cochrane Pregnancy and Childbirth (CPC) group, coordinated by Alfirević.
It provided a detailed list of criteria authors should follow to check an RCT’s trustworthiness – like whether a trial is prospectively registered and free of unusual statistics, such as implausibly narrow or wide distributions of mean values in participant height, weight or other characteristics, and other red flags.
If RCTs fail the checks, reviewers must contact the original authors – and, if the replies are not adequate, exclude the study.
For Sotiriadis, this avoided his having to declare the trials faulty or fraudulent; they had merely failed a test of trustworthiness. His team ultimately excluded the Egyptian trials because they hadn’t been prospectively registered and the authors didn’t explain why.
Alfirević’s team, meanwhile, has found in a study yet to be published that 25% of around 350 RCTs in 18 Cochrane reviews on nutrition and pregnancy would have failed trustworthiness checks, using the CPC’s method.
With these RCTs excluded, the team found one-third of the reviews would require updating because their findings would have changed.
See more from MedicalBrief archives:
The high costs to a medical watchdog of challenging bad science
Scientists sound alarm on badly run medical studies
Famous vitamin C study ‘may have relied on fraudulent data’
Withdrawn: Mexican Ivermectin paper claiming reduced COVID hospitalisation