Skip to main content

So which pollsters were the best and which were the worst? That depends on how you want to analyze things. 

I'm not a statistician or anything, so my analysis can only be very rudimentary. But here it is.

I took only the final state polls from every pollster I had in my database*.  The most recent final poll was from the day before election, the oldest from September 13 (Quinnipiac poll of New York), but most were from October and November.  I used the pollster's spread as the metric by which to measure it against the actual election's percentage spread as of Friday morning.  I calculated a house effect for each poll (predicted minus actual) and an error for each poll (absolute value of predicted minus actual); averages were the sum of each poll's house effect / error divided by the number of final polls that pollster did.

I analyzed every pollster's final poll, but I focus on the eleven pollsters with five or more states in this analysis. 

By the lowest average error metric, Marist won, with PPP second.  Mason-Dixon was by far the worst, with Rasmussen being the second worst.


The most centrist national pollster ended up being Pharos.  Pharos was also the only pollster with a left-leaning house effect on average over all presidential and senate polls.  Everyone else tilted right of actual results, even PPP.  The most partisan pollster on the right?  ARG (4 points).



You could also say a quality pollster is the one which has the most number of guesses within some given number of points. 



The spread in every single final Marist poll was within 5 points of the actual result in both the presidential race and the senate race, which is a pretty amazing accomplishment.  By contrast, fewer than half of Mason-Dixon's polls were within five, also an amazing accomplishment really.

Quinnipiac had the best senate poll on-the-nose mark (3 of 8 within a point), Gravis (amazingly) had the best presidential on-the-nose mark (3 of 11 within a point), but PPP had the best combined on-the-nose mark (9 of 35).  Mason-Dixon was not within a point in any of its 15 final polls (president or senate), though Rasmussen's 2-for-38 mark deserves some special scorn as well just because they had so many more chances than the rest. 

Pharos was within three points of the final margin 9 out of 14 times.  For comparison, Mason-Dixon was within three points only 2 out of 15 times.




Another way you can take the measure of the best and the worst pollsters is by the number of battleground races that they call better than anyone else, because really it's the battleground races that most people focus on.  So I looked at the presidential races in the battleground 9 states plus the expanded map 3 states of Minnesota, Michigan and Pennsylvania.  I also looked at the 14 closest senate races plus Nebraska. 

The overwhelming winner by this standard: PPP.

(1 * indicates a pollster was the best or tied for the best national pollster in the given race, but not the best pollster. 2 ** indicate the best or tied for the best pollster. 1 ? indicates a pollster was the worst or tied for the worst national pollster in a given race, but not the worst pollster. 2 ?? indicate the worst or tied for the worst pollster.)







PPP was tops among national firms in 15 races: Colorado, Florida, Michigan, Minnesota, New Hampshire, Nevada, Nevada senate, Virginia, Virginia senate, Wisconsin senate, Arizona senate, Connecticut senate, Montana senate, New Mexico senate and Pennsylvania senate.  They were the worst only in the Ohio senate race (showing a 10 point Sherrod Brown win when it was only 5.2 points).

Conversely, Rasmussen was the worst national firm in 10 races, Colorado, Iowa, Virginia, Wisconsin, Wisconsin senate, Arizona senate, Massachusetts senate, New Mexico senate, North Dakota senate and Pennsylvania senate.  They were the best in only three races (New Hampshire, Pennsylvania and Indiana senate) (interesting that they were both the best and the worst in Pennsylvania: Rasmussen's 5-point Obama win was almost dead on, but the 1-point Casey win was 8 points lower than reality).

________________

So you can pick your own best and worst from this info.  It's pretty clear to me that the best pollsters were PPP and Marist, and the worst were Rasmussen and Mason-Dixon. 

I look forward to seeing Nate Silver's pollster ratings.

*my database excluded a few polls for reasons of methodology that in hindsight I wish I had included, including Mellman polls. 






(This has been cross-posted at quibblingpotatoes.blogspot.com.

EMAIL TO A FRIEND X
Your Email has been sent.
You must add at least one tag to this diary before publishing it.

Add keywords that describe this diary. Separate multiple keywords with commas.
Tagging tips - Search For Tags - Browse For Tags

?

More Tagging tips:

A tag is a way to search for this diary. If someone is searching for "Barack Obama," is this a diary they'd be trying to find?

Use a person's full name, without any title. Senator Obama may become President Obama, and Michelle Obama might run for office.

If your diary covers an election or elected official, use election tags, which are generally the state abbreviation followed by the office. CA-01 is the first district House seat. CA-Sen covers both senate races. NY-GOV covers the New York governor's race.

Tags do not compound: that is, "education reform" is a completely different tag from "education". A tag like "reform" alone is probably not meaningful.

Consider if one or more of these tags fits your diary: Civil Rights, Community, Congress, Culture, Economy, Education, Elections, Energy, Environment, Health Care, International, Labor, Law, Media, Meta, National Security, Science, Transportation, or White House. If your diary is specific to a state, consider adding the state (California, Texas, etc). Keep in mind, though, that there are many wonderful and important diaries that don't fit in any of these tags. Don't worry if yours doesn't.

You can add a private note to this diary when hotlisting it:
Are you sure you want to remove this diary from your hotlist?
Are you sure you want to remove your recommendation? You can only recommend a diary once, so you will not be able to re-recommend it afterwards.
Rescue this diary, and add a note:
Are you sure you want to remove this diary from Rescue?
Choose where to republish this diary. The diary will be added to the queue for that group. Publish it from the queue to make it appear.

You must be a member of a group to use this feature.

Add a quick update to your diary without changing the diary itself:
Are you sure you want to remove this diary?
(The diary will be removed from the site and returned to your drafts for further editing.)
(The diary will be removed.)
Are you sure you want to save these changes to the published diary?

Comment Preferences

  •  Asdf (1+ / 0-)
    Recommended by:
    sebastianguy99

    It should be noted that Sam Wang got 50/51 Presidential EV calls correct (missing Fl) but all 10 close Senate races incorrect.

    Nate got 51of 51 EV calls correct and 8 of the 10 Senate races.

    These guys are the new Gold standard.

    Sadly, everything Communism said about itself was a lie. Even more sadly,, everything Communism said about Capitalism was the truth.

    by GayIthacan on Fri Nov 09, 2012 at 11:47:33 AM PST

    •  But they aren't pollsters (0+ / 0-)

      Disclaimer: If the above comment can possibly be construed as snark, it probably is.

      by grubber on Fri Nov 09, 2012 at 11:48:51 AM PST

      [ Parent ]

    •  Nate (0+ / 0-)

      Yeah, Nate is quite remarkable and up to Nov 7, was better than a shipping container of Valium for many.

      Not to be critical, as his impressive track record speaks for itself, but I wonder if his "state effects" will be adjusted.  He was really off in both the North Dakota (93%), and to a lessor degree, Montana (66%) senate races.      

      •  The thing about both those states though (0+ / 0-)

        is that Romney won them by larger margins than the polls were predicting, indicating that there still needs to be a presidential-race state effect that goes that far right.  It just didn't apply with the senate races.

        •  Hard to Quantify (0+ / 0-)

          It is really hard to quantify the factor of "one of us" for Senate candidates versus "not one of us" for Presidential races.  In North Carolina at one time we had Terry Sanford and Jesse Helms as our Senators.  I asked an NC resident how that could possibly be and he said, "They are both good old boys that everybody knows."

          How do you quantify that?  "I really really like farmer Senator Tester and I really, really hate Obama.'  Reverse it in Iowa - they really, really like Obama more than normal for that state population.  

          At some point, your math fails you and you are just starting to go toward what the Republicans did with "unskewing" - "math says X, but our guy is the better candidate, I think he ought to win."  

  •  Very nice work (0+ / 0-)

    This is much more useful than the Fordham analysis of national PV polls.

    Disclaimer: If the above comment can possibly be construed as snark, it probably is.

    by grubber on Fri Nov 09, 2012 at 11:52:20 AM PST

  •  But Just a Hair's Breadth Worse Than NPR nt (0+ / 0-)

    We are called to speak for the weak, for the voiceless, for victims of our nation and for those it calls enemy.... --ML King "Beyond Vietnam"

    by Gooserock on Fri Nov 09, 2012 at 11:56:53 AM PST

  •  Bias, not house effect (2+ / 0-)
    Recommended by:
    grubber, quibblingpotatoes

    The standard nomenclature is for "house effect" to mean difference from typical pollsters. Difference from actual results is called "bias", although that doesn't imply intent.

    The results will shift a bit as the provisional votes trickle in, with the R biases all increasing.

    Pharos may end up standing out even more as the least biased. Unfortunately, following Nate's comments, it also seems fairly likely that they aren't real pollsters, since there's no evidence that they made any of the expensive in-person calls that they claim.

    Michael Weissman UID 197542

    by docmidwest on Fri Nov 09, 2012 at 12:10:13 PM PST

    •  Thanks! (1+ / 0-)
      Recommended by:
      docmidwest

      Told y'all I'm not a statistician!  

      It is funny that the firm that probably made up numbers (I mean, their website is terrible and no one who really has a background in polling publishes to two decimal places) is also the one that may end up with the most centrist guesses overall.  

Subscribe or Donate to support Daily Kos.

Click here for the mobile view of the site