The The demand for COVID-19 treatments has created a demand for meta-analyses by systematic review — a process fraught with risk, as evidenced by questionable studies on Ivermectin, write scientists in Nature Medicine.
Jack Lawrence, Gideon Meyerowitz-Katz, James Heathers, Nicholas Brown and Kyle Sheldrick explain why they feel so strongly about this controversial medication as a treatment for COVID-19.
They write:
The global demand for prophylactic and treatment options for COVID-19 has in turn created a demand for both randomised clinical trials, and the synthesis of those trials into meta-analyses by systematic review. This process has been fraught, and has demonstrated the inherent risks in current approaches and accepted standards of quantitative evidence synthesis when dealing with high volumes of recent, often unpublished trial data of variable quality.
Research into the use of Ivermectin (a drug that has an established safety and efficacy record in many parasitic diseases) for the treatment and/or prophylaxis of COVID-19 has illustrated this problem well. Recently, we described flaws in one randomised control trial of Ivermectin, the results of which represented more than 10% of the overall effect in at least two major meta-analyses.
We described several irregularities in the data that could not be consistent with them being experimentally derived. That study has now been withdrawn by the preprint server5 on which it was hosted. We also raised concerns about unexpected stratification across baseline variables in another randomised controlled trial for Ivermectin, which were highly suggestive of randomisation failure.
We have requested data from the authors but, as of 6 September 2021, have not yet received a response. This second Ivermectin study has now been published, and there is still no response from the authors in a request for data.
The authors of one recently published meta-analysis of Ivermectin for COVID-19 have publicly stated that they will now reanalyse and republish their now-retracted meta-analysis and will no longer include either of the two papers just mentioned. As these two papers were the only studies included in that meta-analysis to demonstrate an independently significant reduction in mortality, the revision will probably show no mortality benefit for Ivermectin.
Several other studies that claim a clinical benefit for Ivermectin are similarly fraught, and contain impossible numbers in their results, unexplainable mismatches between trial registry updates and published patient demographics, purported timelines that are not consistent with the veracity of the data collection, and substantial methodological weaknesses. We expect further studies supporting Ivermectin to be withdrawn over the coming months.
Since the above primary studies were published, many hundreds of thousands of patients have been dosed with Ivermectin, relying on an evidence base that has substantially evaporated under close scrutiny.
Relying on low-quality or questionable studies in the current global climate presents severe and immediate harms. The enormous impact of COVID-19 and the consequent urgent need to demonstrate the clinical efficacy of new therapeutic options provides fertile ground for even poorly evidenced claims of efficacy to be amplified, both in the scientific literature and on social media. This context can lead to the rapid translation of almost any apparently favourable conclusion from a relatively weak trial or set of trials into widespread clinical practice and public policy.
We believe that this situation requires immediate remediation. The most salient change required is a change in perspective on the part of both primary researchers and those who bring together the results of individual studies to draw wider conclusions. Specifically, we propose that clinical research should be seen as a contribution of data toward a larger omnibus question rather than an assemblage of summary statistics. Most, if not all, of the flaws described above would have been immediately detected if meta-analyses were performed on an individual patient data (IPD) basis. In particular, irregularities such as extreme terminal digit bias and the duplication of blocks of patient records would have been both obvious and immediately interrogable from raw data if provided.
We recommend that meta-analysts who study interventions for COVID-19 should request and personally review IPD in all cases, even if IPD synthesis techniques are not used. In a similar vein, all clinical trials published on COVID-19 should immediately follow best-practice guidelines and upload anonymised IPD so that this type of analysis can occur. Any study for which authors are not able or not willing to provide suitably anonymised IPD should be considered at high risk of bias for incomplete reporting and/or excluded entirely from meta-syntheses.
Hurdles to the release of IPD from clinical trials are well described, and generally addressable with careful anonymisation and integration of data sharing plans at the ethical approval stage of trial planning.
We recognise that this is a change to long-accepted practice and is substantially more rigorous than the standards that are typically currently applied, but we believe that what has happened in the case of Ivermectin justifies our proposal: a poorly scrutinised evidence base supported the administration of millions of doses of a potentially ineffective drug globally, and yet when this evidence was subjected to a very basic numerical scrutiny it collapsed in a matter of weeks.
This research has created undue confidence in the use of Ivermectin as a prophylactic or treatment for COVID-19, has usurped other research agendas, and probably resulted in inappropriate treatment or substandard care of patients.
We recognise that by recommending IPD review by default for meta-analysis of potential therapeutic agents in COVID-19 we are calling for change to nearly universally accepted practice over many decades, but the consequent potential for patient harm on a global scale demands nothing less.
Author affiliations: St George’s, University of London, Jack Lawrence;
School of Health and Society, University of Wollongong, New South Wales, Australia, Gideon Meyerowitz-Katz; Cipher Skin, Denver, USA, James Heathers;
Linnaeus University, Växjö, Sweden, Nicholas Brown; University of New South Wales, Australia, Kyle A. Sheldrick.
See more from MedicalBrief archives:
Study supporting Ivermectin for COVID withdrawn over ethical concerns
Ivermectin: The legal pitfalls facing a prescribing practitioner
Australian regulator bans off-label Ivermectin use as prescriptions climb
Ivermectin fails as COVID-19 treatment in limited Colombian trial