By now, everyone has heard of antibody (serology) tests which have pointed at higher Coronavirus infection rates than have been officially reported based on diagnostic tests. In this diary we take a look at some of these studies and examine one aspect of the tests that has major implications on how we interpret these test results — the accuracy of the serology test kits. We also briefly list other factors which lower the confidence in the results, their applicability to other counties, states and countries and their implications for relaxing social distancing measures,
A Sample of Antibody Test Studies
Besides the now well-known studies in California and NY, there are a few others, as listed in the table below. Many more are in the pipeline.
|
#tests |
Estimated Prevalance |
Selection Criteria |
test MfgR |
Date |
Santa Clara
|
3,330 |
2.75% |
Facebook ad |
Premier Biotech |
Early Apr |
LA County |
863 |
5.23% |
Random using database |
Premier Biotech |
Early Apr |
NY state |
3000 |
13.9%
21.2% in NYC
|
Grocery store customers |
? |
Apr 20? |
Miami |
1,400 |
6% |
Random telephone requests |
? |
Apr 10 - 24 |
Chelsea, near Boston
(low income, crowded hotspot)
|
200 |
31.5% |
Pedestrians |
BioMedomics |
Early Apr? |
Company in Orange County |
415 |
10% |
All employees |
? |
Apr 18 |
Wuhan Zhongnan Hospital |
3,600 |
2.4% |
Hospital staff |
? |
April? |
Small town in Germany
|
500 |
14% |
All residents |
? |
Apr 9 |
Geneva |
417 |
5.5% |
Random |
? |
Apr 17 |
Netherlands |
? |
3% |
Blood donors |
? |
Early Apr |
Notes:
- In the U.S., only the Santa Clara study has published a (non-peer-reviewed) report.
- In all cases with selections sampled from a large population, individuals had to volunteer for the test.
- Except for the German town and Geneva, none of the studies truly sampled a random set of the target population
- The small town of Gangelt in Germany hosted a large carnival celebration in Feb which attracted thousands of outside visitors
- The prevalence values (aka seroprevalence) usually have a 95% confidence range associated with them. So, the actual value may be lower or higher than the point value shown above.
- The test results do not indicate the antibody levels present in various individuals.
As can be seen, the calculated prevalence rate exhibits a wide range across studies, probably affected by a number of different factors include selection bias, demographics, test kit accuracy and local circumstances.
Antibody Testing and Prevalence Calculation
Antibody tests are conducted on blood samples (not nasal swabs) and they detect the presence of antibodies, which indicate a current or past infection by the virus. It takes about 2 weeks after infection for antibodies to develop in detectable quantities.
In all these studies, blood was drawn from selected individuals, an inexpensive antibody test was performed in the lab, the test results were compiled (positive or negative) and the raw prevalence rate (= number of positives divided by the number of individuals tested) was adjusted for demographics and test kit parameters to arrive at a final prevalence rate. Results are presented as point values and as 95% confidence interval ranges.
In the NY state study, the reported rate seems to be the raw unadjusted prevalence rate.
Estimating Prevalence from Test Results
How is prevalence rate calculated based on raw results and antibody test kit parameters?
Here is a simple explanation of how raw test results are converted to adjusted prevalence rate. The description below shows how to perform point (average) calculations and does not take in account distributions, error bars and confidence intervals.
Each antibody test has two key parameters that characterize the accuracy of the test kit.
- The Specificity parameter (s) specifies the probability that a negative sample is correctly identified as negative. A specificity of 99.5% means that the test has 0.5% probability of misidentifying a negative sample as positive (false positive).
- Sensitivity (r) specifies the probability that a positive sample is correctly identified as positive. A sensitivity of 80.3% means that the test has 19.7% probability of misidentifying a positive sample as negative (false negative).
The following table illustrates how prevalence rate is computed from the raw test results -
|
|
SANTA CLARA |
Equation |
Test Specificity |
s |
99.50% |
|
Test Sensitivity |
r |
80.30% |
|
Total tests |
N |
3,330 |
|
Positive tests |
I |
50 |
|
Raw Infection rate |
x |
1.5% |
I / N |
Infection rate aka Prevalence |
p |
1.26% |
(I/N - (1-s)) / (r - (1-s)) |
Expected True positives |
tp |
33.56 |
N * p * r |
Expected False positives |
fp |
16.44 |
N * (1-p) * (1-s) |
The equation for p is derived by solving for p in the equation I = tp + fp = N * p * r + N * (1-p) * (1-s)
The example above uses the Santa Clara data with the s and r values from the test kit data as documented in the paper. The prevalence rate comes out to be 1.26%, which is lower than the raw infection rate because of the presence of false positives; in this example, there is a high likelihood that there are 16.44 false positives. The study actually shows an infection rate of 2.75% after adjusting for demographic data (we don’t show that here).
What if the Antibody Test Parameters are off?
For the Santa Clara study, we next compute the prevalence rate assuming slightly different values of the specificity parameter.
|
|
SANTA CLARA |
LOWER SPECIFICITY |
HIGHER SPECIFICITY
|
Test Specificity |
s |
99.50% |
98.50% |
100.00% |
Test Sensitivity |
r |
80.30% |
80.30% |
80.30% |
Total tests |
N |
3,330 |
3,330 |
3,330 |
Positive tests |
I |
50 |
50 |
50 |
Raw Infection rate |
x |
1.5% |
1.5% |
1.5% |
Prevalence rate |
p |
1.26% |
0.00% |
1.87% |
Expected True positives |
tp |
33.56 |
0.05 |
50.00 |
Expected False positives |
fp |
16.44 |
49.95 |
0.00 |
If the real specificity was just a bit lower, say 98.5%, as shown in the 2nd data column, the prevalence rate is effectively zero, since all 50 positive test results can be attributed to false positives!
If the real specificity was just a bit higher, say 100%, as shown in the 3rd column, (which is what the LA county study apparently assumed), the infection rate gets inflated by 50% to 1.87%.
We can see that small changes in specificity lead to large swings in the final prevalence rate, especially when the prevalence rate is low. For the Santa Clara and LA County studies, how accurate is the specificity value and therefore how accurate are the final results? Also, do the specificity and sensitivity numbers remain the same for tests conducted in lab settings vs those conducted by sample collection in open drive-through facilities?
In the NY state case, it appears that the 13.9% rate has not been adjusted for test kit parameters, i.e., Specificity and Sensitivity are assumed to be 100%. Given that the Specificity value has been stated as 93% to 100%, the adjusted prevalence rate would range from 7.42% to 13.9%, leading to a lower prevalence rate.
Here is a graph showing how the calculated prevalence rate changes depending on the value of Specificity. The values have not been adjusted for demographics, as is done in the study. The table above shows just 3 points of this graph.
What it shows when Specificity is low (i.e., probability of false positive is high), the calculated prevalence rate is lower than the raw rate, because many positive tests can be attributed to false positives.
Similarly when Sensitivity is low (i.e., probability of false negative is high), the calculated prevalence rate is higher than the raw rate, because many positive cases are missed.
This is how serology tests with poor performance can distort actual results. This also shows how to bias the prevalence calculation towards a higher value — by assuming a slightly higher value of Specificity and a lower value of Sensitivity.