This is a companion piece to the story I wrote a few months ago: “Why polling is dead, dead, dead.” In that piece I addressed the changes in social conditions that now make it impossible to take a random or representative sample of the American voting population. It assumed expertise and honesty on the part of the pollster.
What I did not address, and which has come up often in discussions since then, are efforts to deliberately bias or “skew” a poll so as to obtain a desired result. This is now thought to be happening. How often is it happening? To what extent are the results we now see deliberately skewed, as opposed to incompetently skewed, or simply skewed by the changing nature of our society? We just don’t know.
It’s a judgement call, of course. A competent pollster knows how to bias a poll in ways imperceptible to the average person seeing only the results. Even a totally incompetent one may accomplish this if only the end result is shown to the ultimate consumer.
Unfortunately, only the end result is shown most of the time. We see a headline: “82% of Republicans think Trump should get a second term!” What we don’t see are the questions that were asked, how the sample was drawn, and the various analyses of the results leading to that headline. Do we see what percentage of the total sample were Republicans? No. Do we see the exact wording of the question that produced that result? No. Do we see the demographic makeup of the sample? No. Do we see anything whatsoever of the analysis procedure? No. Often we don’t even see the size of the sample, or sometimes even the response rate—especially if the response rate is abysmally low, as is often the case now. That goes equally for polls where a good faith effort was made to get an accurate result, and where the result was predetermined, and the methodology manipulated accordingly.
So how is this done? It can be heavyhanded and obvious, or subtle. Those who do it heavyhandedly are people who decide to “take a poll”, but know little or nothing about polling. The fact is, almost everybody thinks they know how to “take a poll”: you just make up some questions and then call a bunch of people and ask them those questions, right? It’s much like everybody thinking they’re an expert photographer or a really good cook.
If you’re called by someone “taking” such a poll, you can easily detect it, especially if they’re looking for a predetermined result. They’ll use “good” adjectives for the result they want, and “bad” ones for the one they don’t. Maybe they’ll refer to Trump as “President Trump” and Biden as “Joe Biden”. Or they’ll use “agree” language: “Do you agree that Joe Biden is too old for another term?” Or they might use confusing terminology: “Would you vote to re-elect President Trump?”
An extreme version of this is the “push poll”: they ask a question, then read you a “factual” statement (which probably isn’t), then ask whether that statement would cause you to change your answer. People who believe the statement may make a change, and the changed result will be the only one reported.
As the respondent on the phone, you’ll probably detect these efforts. You might decide to foil them by giving the answer they obviously don’t want, or you might just hang up. But possibly enough people won’t detect the bias to give them what they want, and can then report.
Deliberately skewed polls perpetrated by experts are an entirely different kettle of fish. Unless you know quite a bit about polling, you’re not likely to detect them. Pollsters know that it’s much more difficult to write and conduct a poll that will give accurate results than to conduct one where the result is predetermined. You need to pay attention to the exact wording of questions, and to their order, as well as to the drawing of the sample, and the analysis of the results.
For example: pollsters know that there is a bias toward answering “yes” to a yes or no question. So they can word the question so that the answer they want will be expressed as “yes”. (Well-written and administered polls will have different versions where the same answer will be expressed as “yes” or as “no”. For instance: “Do you think carrots are good for you”, on some versions, and on an equal number of versions “Do you think carrots are bad for you”?)) Another example: they know that there’s a bias toward the items at or toward the top of a list, so the items will be scrambled in different versions of a competent, honest questionnaire.
The order of questions matters. People will take into account their answers to earlier questions when answering later questions, and try to be consistent. A well-written poll will place questions in an order that takes account of this tendency. Also, people may be more likely to be disturbed or angered by some questions than others, so those questions need to be placed AFTER others on the same subject. Then, by the time those questions are asked, they already feel invested in giving their opinion on that subject.
Then there are the demographic questions. Any survey has them. Without them it would be worthless. If you can’t say, “42% of people 20-40 say X, but only 16% of people 41-80 say X”, why are you even bothering to take a survey? Yet those questions tend to be the ones at which the respondent will balk. “I don’t want to answer personal questions”, they say. If they don’t, you have to throw out their entire set of answers. (One problem lately may be that pollsters are now reluctant to do that because of terribly low response rates. But if you keep a partial set of answers in the mix, you degrade your results even further.) Because of this reluctance, demographic questions belong at the end, when even a reluctant respondent may feel too invested to give up at that point.
My husband gets a lot of polls. Maybe he’s on somebody’s list, or maybe it’s just because he answers his phone. He says that the demographic questions usually come at the beginning, and sometimes after he answers those, they say “Thank you” and hang up. Those are probably intentionally skewed polls; they don’t want people in his demographic: too old, too highly educated, the wrong ethnic group, or who knows what?
Of course another possibility is that they ask the demographic questions first because they don’t know any better. They’re trying to be efficient and not waste time on people who would refuse to answer them at the end. These would be non-professional pollsters, so presumably their poll has other problems as well.
Intentional biasing doesn’t just determine the characteristics of the questionnaire. A good deal is done when picking the sample. In fact more probably happens there than anywhere else. If you know that people in the South will give you the answers you want, over-sample the South. More complicatedly, if you know that people with a higher educational level will give you the answers you want, over-sample states that have a higher proportion of people with a BA or higher.
Then there’s “unskewing”. Apparently a great deal of that goes on now. It means deciding that some group or groups were under- or over-sampled, and correcting for that. It’s been called “hand-waving”, as in “This just doesn’t look right, let’s fix it.” It can be done with honest intentions, or not—but even if the intention is honest, it will tap into the unconscious biases of the people involved.
Finally there’s the biasing that can be introduced during analysis. This involves deciding what to cross-tabulate with what----AND it involves, after doing that, which results to report.
The only way to counter all this is to have everything out on the table for the ultimate consumer. Sampling methods and sample. The entire questionnaire. Response rate. Raw figures and “unskewed” figures, if applicable. All the tabulations. Most people won’t bother to look at all that, but they can, and having it “out there” would be a check on the worst impulses of the intentional biasers.
Not that I expect any such thing to happen.