There seems an inordinate amount of confusion here at dKos in claims and counter-claims of what the Exit polls show. This diary is my attempt to give an overview of exit polling. What it can be used for, what it shouldn't be used for and what it really really means. By way of disclaimer I am not associated with polling or statistics in any way shape or form Just an ordinary Kossack who wanted to understand a topic a little better
More below the fold.
A LITTLE HISTORY
To begin our little tour of exit polling, we need to go back to 1980. Here`s Martin Plissner`s account as published in Slate
Back in the 1960s, when the networks first began projecting winners state by state, they had to wait for real votes, even in safe states, before they made a call. In 1980, however, NBC, without any notice to the others, changed the rules. They used exit polls--scientifically sampled surveys of voters leaving their polling stations throughout Election Day--to make a bunch of calls in states where the polls closed at 8 p.m., before any votes were counted. This method enabled NBC a few minutes later to post a triumphant "Reagan elected" graphic which no competitor could match for hours. Jobs at the other networks were suddenly in jeopardy.
By the 1982 off-year elections, every network had exit polls in every state. They raised Election Night budgets to the point by the end of the decade where none of the networks could afford them on their own. Starting in 1990, they formed a succession of pool operations...
Warren Mitofsky started and directed Voter Research & Surveys from 1990 to 1993, which was the election consortium of the four major television networks, ABC, CBS, CNN and NBC. The name was subsequently changed to Voter News Service (VNS. After the debacle of Florida 2000, which resulted in networks calling Florida first for Gore, then retracting then calling for Bush and then retracting, the sh*t hit the fan. After network executives were summoned to testify to Congress, a complete overhaul of VNS was planned.
The 2002 fiasco was the final nail in the VNS coffin. As this
article . details, it was a wild ride.
Breakdown Leaves Networks Hanging
Voter News Service, which provides key data, says it "was not satisfied with the accuracy of today's exit poll analysis."
By Dana Calvo, Elizabeth Jensen and Richard Simon
Times Staff Writers
November 6 2002
WASHINGTON -- Voters and candidates anxious for early results from Tuesday's elections were left hanging like chads when a key system used to predict winners broke down.
As a result, a nation accustomed to hearing the networks predict winners as soon as the polls close was forced to do something unusual in the age of instant information: wait for ballots to be counted.
Voter News Service, the media consortium that conducts the exit polling used by the networks and Associated Press to project winners, said it "was not satisfied with the accuracy of today's exit poll analysis."
The decision not to release the data meant that many voters went to bed Tuesday night not knowing who won a number of closely fought House and Senate races.
Early in 2003, VNS was disbanded and a new service was introduced , the National Election Pool. is a consortium of ABC News, Associated Press, CBS News, CNN, Fox News and NBC News. It was formed in ..."order to provide information on Election Night about the vote count, election analysis and election projections. NEP retained the Associated Press to conduct a tabulation of the vote. NEP also contracted with Edison Media Research and Mitofsky International (Edison/Mitofsky) to make projections and provide exit poll analysis."
The exit poll data for 2004 prepared and released by NEP. More on this later.
WHAT EXACTLY IS AN EXIT POLL?
To basic answer, provided by NEP is
Exit polls are interviews with voters after they have cast their votes at their polling places. The polling places are a scientifically selected sample of polling places that collectively represent a state, or for the national exit poll they represent the nation. An interviewer gives every nth voter exiting the polling place a questionnaire to complete. There are questions about demographic such as gender, age, race, and issues related to the person' s vote and questions about the person' s vote choice in the different contests. Participation is voluntary and anonymous. Interviewing starts when the polls open in the morning, continues throughout the day until about an hour before they close at night.
For a more complete answer, lets turn to the blogosphere, specifically
Mystery Pollster aka Mark Blumenthal a professional democratic pollster.
A quick summary of how exit polls work: The exit pollster begins by drawing a random sampling of precincts within a state, selected so that the odds of any precinct being selected are proportionate to the number that typically vote in that precinct. The National Election Pool Exit Poll, which is conducting the exit polling for the six major networks today, will send exit pollsters to 1,495 precincts across the country.
One or sometimes two interviewers will report to each sampled precinct. They will stand outside and attempt to randomly select roughly 100 voters during the day as they exit from voting. The interviewer will accomplish this task by counting voters as they leave the polling place and selecting every voter at a specific interval (every 10th or 20th voter, for example). The interval is chosen so that approximately 100 interviews will be spread evenly over the course of the day.
When a voter refuses to participate, the interviewer records their gender, race and approximate age. This data allow the exit pollsters to do statistical corrections for any bias in gender, race and age that might result from refusals.
The interviewer will give respondents a 5 1/2 by 8 1/2 card to fill out that will include approximately 25 questions (see an example from the New Hampshire primary here). Respondents fill out the survey privately then put the completed survey in a clearly marked "ballot box" so they know their identities cannot be tracked and their answers remain confidential.
The biggest challenge to exit polls is logistical: How to transmit all the results to a central location quickly and accurately. In past elections, interviewers would take a 10 minute break every hour to tabulate responses. Interviewers would then call in tabulations at three approximate times during the day: 9:00 a.m., 3:00 p.m. and shortly before the polls close (disclaimer: I have no first hand knowledge of this year's procedures, which may be different).
Once the polls close, the interviewer will attempt to obtain actual turnout counts, and if possible, actual vote returns for their precinct. One of the unique aspects of the exit poll design is the way it gradually incorporates real turnout and vote data as it becomes available once the polls close. The exit poll designers have developed weighting schemes and algorithms to allow all sorts of comparisons to historical data that supports the networks as they decide whether to "call" a state for a particular candidate. When all of the votes have been counted, the exit poll is weighted by the vote to match the actual result.
This is just a small flavor of the wealth of information that he provides. Go read
here.
ARE EXIT POLLS ACCURATE?
Depends on WHICH exit poll data you're looking at. The early sets of numbers, are just that-, sets of numbers with little predictive value. Don't believe me? Ok here's Mystery Pollster's advice:
1) It is still just a survey - Even when complete, an exit poll still has the same random variation as any other survey. NEP says typical state exit polls will have a sampling error when complete of +
- 4% at a 95% confidence level, and +- 3% for the national exit poll. Even if comparable to the final numbers - which they are decidedly not - the mid-day leaked numbers would have much greater error, perhaps +/- 7% or more.
2) The mid-day numbers do not reflect weighting by actual turnout - the end-of-day exit poll used to assist the networks in determining winners will be weighted by the actual turnout of voters at each selected precinct. The weighting will then be continuously updated to reflect turnout at comparable precincts. In the past, mid-day numbers have reflected a weighting based on past turnout, so the leaked mid-day numbers may tell us nothing about the impact of new registrants or the unique level of turnout this time.
One point needs emphasis here: even in past elections, networks never called an election based on raw exit poll numbers alone. They were first weighted by a tally of the full day's turnout at each sampled precinct. This end-of-day data is (obviously) not available at 12 noon.
- Voting patterns may be different early in the day - People who work full time jobs typically vote more heavily before or after work. Even a perfect mid-day exit poll - and there is no such animal - may not be any better at picking a winner than the half-time scores in any given football game on Sunday. Also, despite what you may have heard on the West Wing, I know of no serious study showing a consistent Democratic or Republican tilt to the morning or evening hours (if anyone does, please email me).
- Early or absentee voting - As of last night, the ABC News survey estimated that 15% of all registered voters nationally had already cast absentee or early ballots. Obviously, these voters will not be available to interviewers standing outside polling places. To incorporate early voting, the National Election Pool is doing telephone interviewing in 13 states to sample the votes of those who voted early. Will these early votes be included in the mid-day leaked numbers? Who knows? I wouldn't count on it. (Good question, Andrew).
- They could be fictional - Both sides have huge armies of field workers sweating it out in the streets right now. Field workers have been known to find creative ways to boost the morale of their own troops or demoralize the other side. Might someone start a rumor by sending made up numbers to a blog? Ya think? After all, the guy most web surfers turn to for leaked exits likes to say that the information he provides is only 80% accurate. What are the chances he could be, excuse the technical term, making shit up?
.
But what about the later numbers? As more data is accumulated and weighted, the data is hopefully more predictive. As MP says "My knowledge on the specific procedures they use is weak, but the interviewer is responsible for getting hard data on turnout, especially at the end of the day (either from their own tally, from the precinct officials or both). This turnout data is critical to weighting the end-of-day poll."
We know that in some cases, the exit data with the end of day weighting is sufficient to call a state. That is exactly what happened at 7PM Eastern, when the polls in Georgia, Indiana and Kentucky closed and were immediately declared for Bush. Vermont was called for Kerry. All without the benefit of a single vote being tallied. Ah! Technology.
( For a list of when NBC called various races, this is a good resource.)
.
What about the states that aren't able to be called. Remember, a state exit poll at the CLOSE of polling still has a margin of error of plus or minus 4 %. When a race is closer, then the Networks rely on actual totals derived from key precints, turnout, trends and every other tool in their kit. When the race isn't a blowout, the exit poll is only one tool amongst many that election analysts use.
To illustrate what happens when we forget that the exit poll is not perfect consider the following
* VNS overstated the final Democratic vote in the 1992 presidential race. Many believe that this was due to Perot voters being relatively suspicious and unwilling to cooperate with the exit polls.
- In the 1989 Virginia gubernatorial election, exit polls showed African-American Democratic candidate Douglas Wilder winning by 10 percent. Since he actually won by less than 1 percent, many feel that some white voters, feeling perhaps a bit guilty, lied about voting for Wilder.
- In the Republican primaries in New Hampshire in 1992 and in Arizona in 1996, exit polls overestimated the vote for Pat Buchanan. The most likely reason for the mistakes, which resulted in misleading news coverage? Zealous Buchanan voters were more willing to participate in exit polls. In 1992, exit polls predicted a George Bush win by only a small margin over Buchanan -- Bush actually won by 16 points. In 1996, exit polls put Bob Dole a distant third after Steve Forbes and Buchanan - Dole actually came in a close second to Forbes
Here's the
entire article
WHY DID THE EXIT POLL NUMBERS CHANGE?
One of the least understood aspects of exit polls is they are designed to be changed to reflect the actual vote cast. Once the polls close, the process of weighting the exit polls to make them reflective of how the vote turned out is begun. Once again we turn to MP
As returns start to come in, the exit pollsters weight each individual precinct sample by the actual vote cast by all voters at that precinct. Thus, as the night wears on, the accuracy of the exit poll gradually improves.
Why bother with the exit poll when real votes are available? The poll helps analysts determine the size and preferences of key subgroups with increasingly greater precision. What is the vote among Independents? African Americans? Young voters? New registrants? How do those patterns compare with pre-election expectations? Knowing the answers to those questions helps guide those at the network "decision desks" in making projections.
Also, weighting the poll by the actual vote improves its accuracy for its third and most important mission: providing an analytical tool for journalists and the rest of us who want to interpret and explain the election outcome. When a final result for a state is available, the exit pollsters weight the entire sample to match the vote results (there is often a mismatch due to drawing a sample of precincts rather than the entire state). That is the reason bloggers and others noticed that exit poll results posted on CNN and other news sites changed overnight. It was not a conspiracy, just standard practice.
This weighting is an ongoing process as actual returns are made available This point really can't be emphasized enough. As soon as the polls close and real votes are available, the exit poll WILL CHANGE.
Hope this helps getting us all on the same page. For those who want to pursue this in more detail, examining what the exit poll controversy this year is all about, here are some additional links:
. Newark Star Ledger - Nov 3 - exit polls are way off - good quotes etc...here
Wash Post article - Exit polls off - denial by Edison here
Newsweek - Nov 4. Great quotes pro & con about exit polls here
The NEP FAQs about exit polls here