Some of the world’s leading medical journals are reviewing their processes after they were forced to retract major covid studies based on flawed data, reports The Guardian. None of the peer reviewers who examined a questionable study on the impact of blood pressure medications on COVID-19 saw the raw data behind the findings before it was approved for publication in world-renowned medical journal, the New England Journal of Medicine. The report says the study was based on a massive dataset supposedly gathered from hospitals worldwide by a US company called Surgisphere, but a Guardian investigation has since revealed the database to be seriously flawed. The revelation, combined with concerns highlighted by scientists worldwide about the data, prompted the journal to retract the study. The Lancet, another leading medical journal, also published a study based on the Surgisphere database.
The publication and retraction of the studies in renowned medical journals has reignited concerns in the research community about the rigour of peer review. Peer review is where scientists evaluate the quality of other scientists’ work to identify any issues before it is published in industry journals. This process is designed to prevent weak studies and their findings from being published by journals, which is important because what appears in leading medical journals often changes health and medical guidelines for patients.
The Lancet and New England Journal of Medicine are said to have among the most stringent peer review processes of the scientific journals.
A spokesperson for the New England Journal of Medicine is quoted in the report as saying that the Surgisphere paper was reviewed by four external experts and a statistical reviewer. But none of these experts saw the raw data from Surgisphere, she said.
Both The Lancet and the New England Journal of Medicine subscribe to the voluntary editorial recommendations of the International Committee of Medical Journal Editors. Those guidelines state: “As part of peer review, editors are encouraged to review research protocols, plans for statistical analysis if separate from the protocol, and/or contracts associated with project-specific studies. Editors should encourage authors to make such documents publicly available at the time of or after publication, before accepting such studies for publication.”
The New England Journal of Medicine spokesperson said in The Guardian that following the Surgisphere paper retraction earlier in June, the journal completed an internal review of the editorial process. She said the Surgisphere study “received excellent peer reviews, and the reviewers raised relevant questions about the work”.
“We learned two things from this review that will result in changes to our process,” she said. We have limited experience with reviewing or publishing studies like this one, which used a large database based on electronic medical records. The reviewers and editors asked the authors questions about the data sources and data validity. The editors accepted the authors’ responses, rather than asking for help from reviewers with expertise in this type of data. In the future, our review process of big data research will include reviewers with such specific expertise.”
The report says the journal is now in the process of assessing existing guidelines for the conduct and reporting of research on big data, and is also developing internal policies for reviewing and reporting these articles.
The Lancet did not provide as much detail when asked the same questions by The Guardian about how the Surgisphere hydroxychloroquine study passed peer review, and whether the incident would trigger a review of the process.
The report says it was The Lancet that under the same editor, Richard Horton, published a fraudulent study linking the measles, mumps and rubella vaccine with autism. In 2010, 12 years after the paper was first published, The Lancet retracted the paper. The author of the paper, Dr Andrew Wakefield, was also banned from practising medicine in the UK due to his fraudulent work.
The report says the latest retraction from the journal has left researchers and the public questioning whether the peer review process and editorial policies are rigorous enough.
The New York Times reports that the hasty retractions, on the same day this month, have alarmed scientists worldwide who fear that the rush for research on the coronavirus has overwhelmed the peer review process and opened the door to fraud, threatening the credibility of respected medical journals just when they are needed most.
Peer review is supposed to safeguard the quality of scientific research. When a journal receives a manuscript, the editors ask three or more experts in the field for comments. The reviewers’ written assessments may force revisions in a paper or prompt the journal to reject the work altogether. The system, widely adopted by medical journals in the middle of the 20th century, undergirds scientific discourse around the world.
“The problem with trust is that it’s too easy to lose and too hard to get back,” said Dr Jerome Kassirer, a former editor in chief of the New England Journal of Medicine. “These are big blunders.” If outside scientists detected problems that weren’t identified by the peer reviewers, then the journals failed, he said. Like hundreds of other researchers, Kassirer called on the editors to publish full explanations of what happened.
The Lancet’s Horton and Dr Eric Rubin, editor in chief of the NEJM, said that the studies should never have appeared in their journals but insisted that the review process was still working. “We shouldn’t have published this,” Rubin said of the study appearing in the NEJM. “We should have had reviewers who would recognise the problem.”
In addition, the editors said, there is an urgent need to rapidly publish new findings to improve treatments for desperately ill coronavirus patients. Since the pandemic began, The New York Times quotes Horton as saying that The Lancet is receiving three times the usual number of papers for consideration. And the NEJM has fielded as many as 200 submissions in a day, including essays, according to Rubin.
“I’m an infectious disease doctor, I treat COVID-19 patients,” Rubin said. “I’ve been in the hospital recently treating patients, and we have no idea what to do. I’m the primary driver at the journal of saying, ‘We have to get data out there that people can use.’”
“We are very careful,” he added. “At our editorial meetings, this comes up almost every day. ‘If we publish this, will it hurt people?’ That’s our biggest concern.”
The New York Times reports that the NEJM and The Lancet are among the oldest, most respected and most widely read medical journals in the world. They were established in 1821 and 1823 and are ranked often first and second among general-interest medical journals by their “impact factor,” the frequency with which their studies are cited in other research.
A report in one of these journals can have immediate repercussions both for patients and for research.
The reputation of these journals rests in large part on vigorous peer review. But the process is opaque and fallible: Journals generally do not disclose who reviewed a study, what they found, how long it took or even when a manuscript was submitted. Horton and Rubin declined to provide those details regarding the retracted studies, as well, the report says.
Critics have long worried that the safeguards are cracking, and have called on medical journals to operate with greater transparency.
“We are in the midst of a pandemic, and science is moving really fast, so there are extenuating circumstances here,” said Dr Ivan Oransky, co-founder of Retraction Watch, which tracks discredited research. “But peer review fails more often than anyone admits,” he said. “We should be surprised it catches anything at all, the way it’s set up.”
The report says journals used to take many months, or even a full year, to scrutinise and edit a complicated study, a process that included several weeks for outside experts to peer review the research. Now peer review may be condensed to as little as 48 hours; some studies deemed of vital importance to patients may be published online within 20 days of submission.
“There is always a tension between getting it fast and getting it right,” said Dr Marcia Angell, another former editor in chief of the NEJM. “I always favoured getting it right. But in the current pandemic, that balance may have shifted too far toward getting it fast.”
It’s not just the journal editors who are inundated. Knowledgeable scientists who donate time as peer reviewers are already stretched thin, trying to understand how the coronavirus affects the body, or to find treatments and vaccines. The research is happening at an unprecedented pace.
“I think the academic system is saturated – it’s at capacity,” said Dr Peter Jüni, a professor of epidemiology at the University of Toronto who has reviewed papers for scientific journals. “People are tired; they’re working at the edge of their limits,” he said. “They struggle to get good peer reviewers and try to do as well as they can, but the system is at risk of failing, as you see here.”
Journal editors are caught in a Catch-22 of sorts, said Hassan Murad, of the Mayo Clinic, who works with a federal project to review medical evidence. “You want to push the information out quickly to practitioners,” he said. “It’s a pandemic, it’s an urgent situation. At the same time, you want quality control.”
[link url="https://www.theguardian.com/world/2020/jun/12/covid-19-studies-based-on-flawed-surgisphere-data-force-medical-journals-to-review-processes"]Full report in The Guardian[/link]
[link url="https://www.nytimes.com/2020/06/14/health/virus-journals.html?campaign_id=2&emc=edit_th_20200615&instance_id=19394&nl=todaysheadlines®i_id=42505380&segment_id=30915&user_id=c3d9778832e13a8ee81a16ebe4996cb9"]Full report in The New York Times[/link]