Results from the first studies designed to determine how widely the coronavirus has spread in communities have started to trickle in, drawing immense attention. Stat News reports that experts are already raising concerns about the validity of some of the studies and cautioning officials and the general public not to put too much weight on any one finding.
Known as serological surveys, the studies involve testing the blood of people not diagnosed with COVID-19 to determine whether they had previously been infected by the SARS-CoV-2 virus. They are important because they can flesh out the picture of how many people in any given community may have had COVID-19, even if they were unaware they were infected. These findings can help policymakers and experts gauge how vulnerable a community remains to the virus, how deadly the pathogen is, and how frequently asymptomatic or mild cases occur. “That number, whether it turns out that we’ve had 1% of the population infected or 10 or 15% of the population infected, is going to really be crucial to the decisions we make next,” said Michael Mina, an epidemiologist at the Harvard TH Chan School of Public Health.
But, the report says, in the rush to get answers, the market has been flooded with a vast number of tests, most of which haven’t been validated. Some of the serological studies have also been more rigorous than others. Epidemiologists and statisticians have criticized some of the early headline-generating surveys for how they recruited participants and analysed the data. More results are expected soon. Each study will be a snapshot of a community at a given time, and each will have its limitations.
“What is really important for us to be able to do to put these numbers into context is to understand how the studies were done,” said Maria Van Kerkhove, the technical lead on the pandemic for the World Health Organisation, which has released a protocol for conducting serosurveys.
The report says below is an outline how these surveys work and some of the considerations experts use to assess the strength of them.
The snapshots are starting to pile up. What do they show?
These surveys involve testing a sampling of a population for antibodies – a sign that a person was exposed to the virus and mounted an immune response – and extrapolating those findings to suggest what they mean for a community as a whole. Already, researchers have found that in particularly hard-hit places, the virus could have infected double-digit percentages of the population.
Of 200 people tested on the street in the Boston suburb of Chelsea, some 30% had been exposed to the virus. A sampling of a German town where a carnival drove the spread of the virus found 14% were positive for antibodies. And in New York Govenor Andrew Cuomo has announced that roughly one in five people in New York City and nearly 14% state-wide had antibodies. The New York estimate was based on a sampling of 3,000 grocery store shoppers.
But those double-digit findings are the exceptions, not the rule. Most of the results to date have shown just a tiny fraction of people have been exposed to the virus, including one survey released this week that found 5.5% of 760 participants in Geneva were infected. Others have found 2% to 3% of people test positive.
In a way, finding that just a small percentage of people in communities has been infected is disheartening. It is thought – though not yet proven – that people who recover from an initial infection will be protected from another case for some amount of time. The higher the percentage of immune people in an area, the less well the virus spreads. Even a third of people being protected from contracting and transmitting the virus does not mean that community is close to reaching “herd immunity” – when enough people are protected that the virus can’t gain a foothold.
“There’s been an expectation that herd immunity may have been achieved and that the majority of people in society may already have developed antibodies,” said Mike Ryan, the head of the WHO’s emergencies programme. “I think the general evidence is pointing against that and pointing towards a much lower seroprevalence.”
Experts say it is imperative in these early days to review the methodologies of these studies and assess the performance of the antibody tests being used. It’s also important to recognise that many have not yet gone through the rigorous peer review process before results are made public. Some are publicized in press releases, others on preprint servers, which post drafts of scientific papers before publication in journals so that they can be shared rapidly.
“I have to admit, I am struggling with this in this pandemic,” van Kerkhove said of the tsunami of papers being released as preprints. She applauded the rapid sharing in general, but said a lot of the papers really need the critiquing of peer review to sharpen findings and correct errors. “By the time a paper is actually published, what it looks like from the time of submission is drastically different. And that is hard to interpret.”
How do scientists recruit for their studies?
The ideal patient sample in one of these surveys will be a random one that reflects a community’s socioeconomic, geographic, age, and ethnic diversity. One of the issues outside experts raised with a widely criticised study out of Santa Clara County, California, was that the researchers recruited participants through Facebook ads. They say this approach could have drawn people who thought they had been infected and wanted confirmation – and wouldn’t necessarily lead to a sampling that represented the county’s population as a whole. Instead, it could have resulted in infection rates that were misleadingly high.
“It biases your sample very much toward people who want to be tested, who might suspect they’ve had it, and that can lead you to overestimate the number of people who have actually been exposed,” said William Hanage, an epidemiologist at the Harvard TH Chan School of Public Health.
To recruit a representative sampling for a survey in Florida’s Miami-Dade County, the research team used information from the local power utility to randomly dial residents with a recorded call from the mayor to ask if they would enrol. Even this approach is not perfect, said Erin Kobetz, a professor of medicine and public health sciences at University of Miami, who is helping run the survey. Some people, for example, won’t have transportation to get to the testing site or will have other reasons to participate or not.
She said researchers needed to acknowledge the limitations of their studies. “This is important information for public health planning, because it’s telling us what’s occurring at a population level,” she said. “Is it perfect? No. Is it better than nothing? Yes.”
Hanage said that conducting a survey with more participants and in a community that has had wider spread of the virus will produce more reliable results. Greater exposure makes it more likely the random sample will include people who have had serious illness as well as mild or asymptomatic illness, so the survey can “determine the real spectrum of severity of disease,” he said. Otherwise, the flaws in the tests in producing false positives or false negatives will be exacerbated. (More on that later).
Do the results pass the smell test?
Far more people have been infected by the coronavirus than official counts reflect. Given the range of symptoms, many people never felt sick enough to seek out a test, and many who did couldn’t get one. In countries including the US, testing has been limited by problems with the rollout and capacity.
But when the team that led the Santa Clara County study reported that the case count could be 50 to 85 times higher than the number of confirmed cases, that raised scepticism. If that were truly the case, then local hospitals would have been far more inundated with patients than they had been, outside researchers argued. Other estimates have pegged total cases as 10 to 20 times higher than confirmed cases.
Similarly, that higher infection rate would drive the fatality rate (which is established by dividing deaths over total infections) far lower than what’s been seen throughout the pandemic. Making a claim that goes against so much other data requires a ton of evidence that outside experts said this study did not provide.
So about those false positives and false negatives?
No test is perfect. And the sheer number of antibody tests – Dutch virologist Marion Koopmans recently saw nearly 275 on a list maintained by the WHO – makes it very tough at this stage to know how good any of them actually are. The WHO is working with a number of labs trying to validate tests, said van Kerkhove, who added: “Unfortunately that takes a little bit of time.”
In particular, the rapid tests appear not to perform well at all. Koopmans, the head of virology at Erasmus Medical Centre in Rotterdam, said the Dutch national serology task force has recommended that people not use the rapid tests, because of the risk that people will get a false result and assume – if it was a positive – that they have protection they do not in fact have. Every serology test is going to produce some erroneous results. Some people who were truly sick will test negative – that’s a false negative. Some people who were not sick will test positive – that’s a false positive.
Each commercial test comes with guidance from the manufacturer about how “sensitive” it is – in other words, what percentage of true positive cases it will detect – as well as how “specific” it is, meaning how good it is at not generating false positive results.
Those estimates are especially important when the rate of infection in an area is likely low. Even a small over-estimate – say a 5% false positive rate – can vastly increase the final projection of how many people in a location had been infected.
Michael Osterholm, director of the Centre for Infectious Diseases Research and Policy at the University of Minnesota, drew up a chart to explain how different rates of sensitivity and specificity will impact a serology study in an area with 1m people, using a test that had 95% sensitivity (caught all but 5% of true positives) and 95% specificity (designated as positive only 5% of people who were actually negative).
If 5% of the population had been infected with SARS-CoV-2, there would have been 50,000 infected people. This test would find 47,500 (the true positives) but it would miss 2,500 (the false negatives). And it would detect 47,500 false positives – as many false positives as true positives. If the rate of infection in the community was smaller, the percentage of wrong results would rise.
If the rate of infection in the community increased, the errors become less substantial. If 15% of the community – 150,000 – had been infected, this test would find 142,500 true positives, 42,500 false positives, and would miss 7,500 cases – the false negatives.
Applying this knowledge to the results from New York puts the picture in sharper focus. The release from the state doesn’t disclose the sensitivity of the test used, but it does note the specificity is between 93% and 100%, a “huge range,” Ashish Jha, head of the Harvard Global Health Institute, noted on Twitter. If the test performed at the low end of that range, New York’s infection rate would be closer to 7% – half the figure Cuomo announced – and nearly one out of every two positives would have been a false positive, Jha said.
“These tests don’t perform like people think they do and so there are a lot of crazy results,” Osterholm said. “You can often find more than half of the positives you do document are actually false positives.” People who don’t understand how challenging serology testing is may assume a result is binary – positive or negative. But reading a result is nowhere near that black and white, Osterholm said.
Think that’s complicated? There’s more.
The whole goal with serological surveys is to get an estimate of the proportion of cases that testing missed. That means finding the people with mild infection. But some emerging evidence suggests at least some people with really mild or almost symptom-free infections may have very low levels of antibodies – or no detectable antibody at all. That will complicate both the testing and the interpretation of any results.
“If people with mild disease don’t have antibodies, do they then have some kind of immunity or not at all?” Koopmans wondered.
Another potential complication arises from the fact that SARS-CoV-2 is a member of the coronavirus family, which includes four viruses that cause common colds. Most of us would have been infected by at least a few of these four over the courses of our lifetimes.
Are the new serology tests good enough to distinguish SARS-2 antibodies from those generated by the other human coronaviruses? Or will some of them pick up a false signal?
“We know that it’s a potential problem,” the WHO’s van Kerkhove said, adding that’s why it is so important that tests are validated. Part of the validation process is testing the assays on blood samples drawn – before the new virus emerged – from people who have had previous coronavirus infections.
In the US, the Food and Drug Administration has taken the extraordinary step of allowing test manufacturers to market serological tests that haven’t yet undergone agency review, as long as they have undergone validation in the hands of the manufacturer. But Koopmans said sometimes this type of validation is minimal.
Natalie Dean, an assistant professor of biostatistics at the University of Florida who has been critiquing some of the serology studies on Twitter, said this approach is kind of an about-face for the FDA. “I guess one of the important things to remember is that the FDA faced all those challenges with PCR testing” – the type of testing used to confirm active infection – “and being very stringent. And so, what appears to have happened is sort of a swing in the other direction, where now they’re being quite lax,” Dean said.
Still, with all the current problems, two things are clear.
The first: Whether the study was conducted in California or Denmark, in the Netherlands or Germany, most have shown the virus has not yet infected a big portion of the screened population.
“In general, I think things have been pretty consistent. It’s only in the really hard-hit places that we’re seeing anything above a single-digit number,” said Dean. “The harder hit places have higher seroprevalence, but even the hard-hit places don’t seem to have crazy numbers. Certainly not at the herd immunity level.”
That leads us to the second thing: The world still has no idea how to interpret the import of any of these studies.
The presence of antibodies suggests a person was previously infected. But are they protected? Most experts assume there will be at least short-term protection. But how long will it last? Will people with low levels of antibodies be as protected as people with higher levels? Could countries safely issue “immunity passports,” as several have suggested, that would allow people with proof of prior infection to return to work or move about more freely?
Only more experience with this new virus will provide answers to those kinds of questions. “At the current time, there’s a lot of problems with that (idea),” van Kerkhove said about the immunity passports. “Because antibody does not mean immunity.”
[link url="https://www.statnews.com/2020/04/24/the-results-of-coronavirus-serosurveys-are-starting-to-be-released-heres-how-to-kick-their-tires/?utm_source=STAT+Newsletters&utm_campaign=d713df9ad1-Daily_Recap&utm_medium=email&utm_term=0_8cab1d7961-d713df9ad1-152512097"]Full Stat News report[/link]