The mortal enemy of the "virtual tie"
As happens every election cycle, but which has become much more pronounced in recent years, reporters and others breathlessly report the latest poll results and, if the numbers are close, they start informing everyone of the Margin of Error of a given poll and then breathlessly state phrases like “statistical tie” or “virtual tie” if the difference in numbers are close to or within the Margin of Error.
Most reporters, however, have the mathematical and statistical knowledge of a newt. But unless they make the effort, they don’t get better and as a result don’t understand probability and proceed to share their ignorance with everyone else. So why does a 4 point lead, a 3 point point lead, or even a 2 point lead in a poll with a 4 point margin of error not indicate a “virtual tie” or a “statistical tie”? Why, if it’s that close, does a site like 538 only give the person behind a 15% or so chance of winning?
Unfortunately, understanding this requires a little math, but worry not because this will keep it simple with only a few numbers, and some pictures. And this is going to be “lies told to children”, aka rules of thumb. Real statisticians and pollsters (of which I am not) would be able to pick this apart, but it is meant as a simplistic explanation.
When you see a poll comparing two candidates, there are only four numbers you need to be aware of:
1. The numbers for the candidates;
2. The confidence level of the poll (typically the “19 times out of 20” bit); and
3. the Margin of Error (MoE).
Let’s take an imaginary poll. To keep it simple, there’s only 2 candidates. A is polling at 50%, B is polling at 46%, there are 4% undecided/third candidate/think butterflies are evil and the MoE is (coincidentally) 4%.
If the poll had a sample size equal to the number of people who end up voting (also known as “an election”), then the MoE would be 0% and there would be total confidence that the result was A 50, B 46. However, because polling doesn’t do that, you sample the population. If your sample is sufficiently random and sufficiently big enough, then the magic of statistics allows you to make predictions of what the result would be if the election was held at that moment.
Because it would be dumb luck for random polling to hit the right number, the odds are a poll will be different from that true number. But how different? Well, there’s this:
I’m sure most of you will recognize that: that there is a bell curve. Imagine you didn’t do just one random poll, you did hundreds. If your polls were random, with enough people polled in each one, and you did them over and over you’d see clustering; most of the results for Candidate A will be around 50, and the further the result was from 50, the fewer polls would show it.
But here’s the thing: when you do a political poll, you don’t know what the true number is. That’s what you’re trying to find out. But you’d expect, if you do proper, statistically good random polling, that it’s more likely to be closer to the real number than it would be further away, but how close is it likely to be? What are the odds you’ll be within a reasonable range of the true answer? And what’s a reasonable range?
There are equations to determine the “standard error" based on the size of the sample, a statistical measure of how far you’re likely to be away from the real number. I’m not going to repeat them, but they have the neat feature that once you reach a certain size in your random sample, adding more people doesn’t really make it that much more accurate. Polling 100 people is crap, pulling 400 is much better, polling 800 better (but not by as much), polling 1000 better again (but by even less), and so on and so forth. That’s why you’ll rarely see polls with many more than 1,000 respondents: statistically, your accuracy won’t go up that much more compared to how much more money you have to spend to conduct the larger poll.
Okay, so our pollster has calculated what the standard error should be based on the number of people polled. Standard error also works on the bell curve principal. Let’s say that our calculated standard error is 2%.
Here we see the resulting graph. What that tells us is that if poll says Candidate A is getting 50% of the vote (if it were held today), and the number of people polled gives a standard error of 2%, that means there’s a 68.27% probability that Candidate A’s actual will be somewhere within the range of 48 to 52.
Now, 68% odds are good, but really, that means 1 out of 3 times it’s not going to fall in that range, which is way too high. So what if we go out two standard errors for the range? What are the odds that Candidate A will get between 46 and 54?
Now that’s much better. There is a 95.45% probability that A’s actual number will be somewhere in that range. But that 95.45% is reasonably close to 95%, and why does 95% remind us of something? Oh, that’s right. That’s equal to 19/20.
That’s why they say “19 times out of 20, the results of this poll...” What they’re actually saying is that the probability is 95% that the real number will lie within 2 standard errors of the poll result. But wait, 2 standard errors is 2 times 2%, which is 4%, and where have seen that number before? Yes, that’s the Margin of Error.
If you want to be precise, 95% confidence is 1.97 times the standard error, but 2 and 95.45 are close enough that it doesn’t really matter in this kind of thing, so we’ll round off.
So, to summarize: the MoE is, rounded off, twice the standard error.
Okay, but we have two candidates, so what does this mean? Who’s really ahead?
Let’s keep things simple: let’s say that those 4% undecided/other party voters is the real number and doesn’t change, so we only have to deal with Candidates A and B. If A gets 50%, B must have 46%. If A has 49, B must have 47, if A has 51, B must have 45, and so on.
If our poll has, as mentioned, A-50, B-46 and the MoE of 4, what are the odds they’re tied or, possibly, B might even be leading? Let’s look at it from the B’s perspective. B’s probability curve is going to look like A’s, except centered on 46.
The probability of B’s actual number being 47 or above is about 31%. But B still loses if the result is below 48, because A’s result will be above 48, so A must get 48 or better to be tied or winning. However, the odds of B getting 48 or greater are less than 16%.
So what have we determined? With a 4% margin of error and a 4% difference in the polling numbers, and all other things being equal, the candidate who is behind has less than a 16% chance of really being tied or ahead. Looking at it the other way, candidate A has an 84% chance of really being the one in the lead and, if the election results are close to the polls results, actually winning.
Well, what if they were closer? What if it was still the 4% MoE, but they were only 2 apart, 50-48 (because half the undecided got off their ass and picked B)? Well…
B has a 31% probability of getting 49 or above, which they need to reach to tie or surpass A. Nearly twice as good as they had before but still, if I told you something had nearly 70% odds of being true (roughly A’s chances of really being ahead), you’d feel reasonably confident that yeah, A is probably ahead and B is probably losing.
Once again, the caveat: this is extremely simplistic, but gives a reasonable rule of thumb. Adding more candidates makes it a bit more complicated, but since in this election you’re really only worried about 2 candidates (except Utah), it works well enough.
So what would it take to have a true “virtual tie” or “statistical tie”? I won’t add more graphs, but, with a 4% MoE, here’s a rough guide:
A leading B by 4, 16% chance B really tied or ahead.
A leading B by 3, 23% chance B really tied or ahead.
A leading B by 2, 31% chance B really tied or ahead.
A leading B by 1, 40% chance B really tied or ahead.
A leading B by 0.5, 45% chance B really tied or ahead.
A tied with B, 50% chance B really tied or ahead.
Well, look at that. In order to say, with certainty, that B has even odds of really being tied or ahead of A, the poll has to actually show them tied.
Still, we accept a 95% probability for a poll number being within the margin of error, so we shouldn't necessarily accept 50% to declare it really too close to comfortably call. I’d say 40% is a reasonable benchmark. If I told you that something had a 60% chance of being true, but a 40% chance of being false, that would be a bit too close for most people. So for a 40% chance of an actual tie or a switch in the lead, the poll has to show a difference of, at most, 1. Which is ¼ of the Margin of Error.
If the Margin of Error is another number, it will still work out to the same fraction. A 10% MoE would need them to be within 2.5 of each other for that 40% benchmark (which is why polls with 10% MoEs are crap). If the MoE is 3%, they’d have to be within 0.75 of each other. With a 2% MoE, within 0.5.
This is why a site like 538 can show Clinton “within the margin of error” behind Trump in Texas, and yet have Trump at an 84% chance of winning the state. Reporters can natter on about “virtual ties” or “statistical ties" all they want, but the polls aren’t actually showing that. If I tell you something has a 16% chance happening, you would not describe it as a good chance.
Of course, polls numbers are not set in stone unless they’re actually seen as being stable. Pulling within 3 of Trump in freaking Texas still means something even if, mathematically, she still has a ways to go.
