Recently I've seen at least two rec-listed diaries that allege that the Gallup Organization has deliberately distorted its survey results. The most recent one avers, "Gallup admits: it only releases likely voter results when they favor McCain!" Before that we had: "Busted!: Gallup, CBS, USA.Today, etc. Tinkers With Party ID Again." Both diaries shot up the rec list, with comments about how Gallup was abetting election fraud. These are (or should be) serious allegations, but the supporting evidence was seriously lacking. I'll explain over the jump.
The allegations
These seem to be the major points in the two diaries I linked above:
- Gallup and other firms "can't stop the madness and [they're] still diddling around on behalf of their media masters."
- Gallup's September poll, along with other media polls, was "conducted using a higher sampling of Republican voters than in July.... [T]hese polls still decided to shift in the favor of Republicans coming out of their convention, which only gives the perception of McCain really gaining steam."
- A former Gallup executive confirms that "pollsters don't report public opinion, they manufacture it."
- The Gallup/USA Today poll released on September 8 "used a sudden, unexplained, temporary shift to the likely voter model.... This was a deliberate choice by Gallup and USA Today -- to release a poll that would maximize the impression of McCain's bounce and help shape the campaign narrative in favor of McCain-Palin's favor."
- Gallup admits that "it believed institutionally that likely voter results were less accurate than registered voter results," but "deliberately chose to release" a poll using this "admittedly less accurate method."
A matter of ethics
First, a point that some may consider painfully naive: Gallup researchers subscribe to an ethical code, and as far as I can tell, they take it seriously. Most if not all of them belong (as I do) to AAPOR, the American Association for Public Opinion Research. The AAPOR Code of Professional Ethics & Practices says in part:
We shall not knowingly select research tools and methods of analysis that yield misleading conclusions.
We shall not knowingly make interpretations of research results that are inconsistent with the data available, nor shall we tacitly permit such interpretations.
When preparing a report for public release we shall ensure that the findings are a balanced and accurate portrayal of the survey results.
If we become aware of the appearance in public of serious inaccuracies or distortions regarding our research, we shall publicly disclose what is required to correct these inaccuracies or distortions....
Alas, AAPOR can't take away the toys of researchers that violate this code, but it can and does call them out. A classic example is AAPOR's smackdown of Frank Luntz for failing to support his claims about public support for the "Contract with America." This is not the sort of attention that the folks at Gallup would like to receive. On the contrary, George Gallup was passionately idealistic about honest public opinion research, and the organization strongly values its reputation for integrity. I think it has earned that reputation.
If the allegations I've cited are true, then Gallup's reputation should be in the dumpster. If they are unsupported, then someone should set the record straight, as I will try to do here. Point by point:
1. Media masters
I don't feel like parsing exactly what the diarist had in mind in this claim, but having read a few hundred comments, many readers take it this way: the MSM has a stake in (1) contriving a close election so that ratings go up, and/or (2) throwing the election to McCain, which can be served in part by creating confusion about who is really ahead. And so, Gallup and/or other pollsters have been commissioned to serve these ends.
At least part of the premise here is reasonable: the MSM probably does benefit if the election is exciting. Beyond that... well, let's move on.
2. "...decided to shift in the favor of Republicans..."
The allegation, as I understand it, is that Gallup either deliberately oversampled Republicans, or deliberately overweighted the Republicans in their sample (or both), so that Republicans would be overrepresented and McCain would fare better than he ought.
No evidence is presented for this allegation. To be sure, the diarist links to a Huffington Post article that documents that Gallup's tracking poll, the Gallup/USA Today poll, and a CBS poll all showed a shift toward more self-identified Republicans than in past polls. For instance, in the 9/3-5 Gallup tracking poll, equal percentages of Republicans and Democrats were surveyed, compared with a 10-point Democratic advantage two weeks earlier. (Here I'm relying on the facts in the article.)
To some of us, the fact that three different polls showed a movement toward Republican identification suggests that maybe a lot of people who have been wavering between "Republican" and "independent" wavered back to Republican immediately after the convention. And it's always possible that the polls obtained biased samples -- for instance, some of them may have sampled too many Republicans because Republicans happened to be home watching the convention. It's also logically possible that pollsters grabbed the "party identification" knob and gave it a firm twist to the right. But I have to wonder: why would they even bother? If party identification is some telltale giveaway, why not produce some other numbers instead? (Of course, for someone who believes that McCain cannot have been ahead, any numbers would be implausible.)
At any rate, I've seen no evidence whatsoever that Gallup did anything to manipulate the proportions of self-identified Democrats and Republicans in these surveys. The assertion that Gallup "decided to shift in the favor of Republicans" is mere conjecture, "supported" by many quotations that have no bearing on the matter.
By the way, citing party registration figures is simply beside the point, unless a lot of context is provided. Surveys generally ask people how they think of themselves, not how they are registered -- and plenty of registered Democrats and Republicans don't think of themselves as such.
3. David W. Moore on manipulating public opinion
Full disclosure: I haven't read David Moore's latest book yet. I did, however, take a few minutes to read the Publisher's Weekly review quoted at Moore's website:
[Moore] argues that today's polls report the whims rather than the will of the people due to an intrinsic methodological problem: poll results don't differentiate between those who express deeply held views and those who have hardly, if at all, thought about an issue. Thus, respondents are compelled to provide an ill-considered, top-of-mind response because the method does not offer the option of expressing no opinion. In Moore's view, forced-choice polls not only distort public opinion, they create a legitimacy spin cycle, which damages U.S. democracy by manufacturing a public consensus to serve those in power.
That's an interesting line of criticism, part of which dates back at least to the 1940s (when Lindsay Rogers wrote, "Instead of feeling the pulse of democracy, Dr. Gallup listens to its baby talk"). But it has nothing to do with whether Gallup games its pre-election polls.
4. The "sudden, unexplained, temporary shift"
As the second diarist rightly says, the Gallup tracking poll has used a registered voter screen -- that is, anyone who says they are registered to vote is included. (My guess is that Gallup doesn't screen in North Dakota, which doesn't have voter registration, but such details don't matter here.) It's also true that the first September USA Today poll, fielded the 5th through the 7th, reported that "McCain leads Obama by 54%-44% among those seen as most likely to vote." (This statement appears in the 8th paragraph of the story. The second paragraph reports McCain's 4-point lead among registered voters.) So, is this a "sudden, unexplained, temporary shift" from registered voters to likely voters?
No, it isn't. PollingReport.com lists ten USA Today/Gallup polls between January and August. All ten of them used a likely voter screen. Here is USA Today's writeup of the previous survey, which was fielded August 21 through 23. To quote:
Obama holds a 47%-43% edge over McCain among registered voters and a 48%-45% edge among likely voters. Both leads are within the margin of error of +/—4 percentage points.
In the USA TODAY/Gallup Poll a month ago, Obama led McCain by 3 percentage points, but McCain held a 4-point lead among likely voters.
So, a late-August (pre-convention) poll that not only used the likely voter screen, but cited a July poll that did as well. Summary: the Gallup tracker has always used a registered voter screen; the USA Today/Gallup Poll has apparently used a likely voter screen all year (and has reported both RV and LV results). However, people might have one good reason to think otherwise:
The Democratic National Convention significantly boosted Americans' views of Barack Obama as a strong leader who "shares your values" and can manage the economy and Iraq, a USA TODAY/Gallup Poll taken Saturday and Sunday [August 30-31] finds.... In the head-to-head race, Obama leads 50%-43% among registered voters. In the USA TODAY Poll taken Aug. 21-23, the Illinois senator held a 4-percentage-point lead.
Aha! so USA Today apparently chose to use RV numbers, not LV numbers, to measure Obama's bounce. Here is the catch: Mark Blumenthal reported that this survey in fact "was a two-night subset of [the] Gallup Daily tracking survey." So, if anything, there seems to have been a sudden, unexplained, temporary shift away from the likely voter screen -- although it isn't hard to explain: USA Today wanted some fresh results between its regularly scheduled polls. (Below I note another point that might have confused people about the standard practice in the USA Today polls.)
By the way, one more quotation from that presumably dreadful USA Today piece:
"The Republicans had a very successful convention and, at least initially, the selection of Sarah Palin has made a big difference," says political scientist Larry Sabato of the University of Virginia. "He's in a far better position than his people imagined he would be in at this point."
However, in an analysis of the impact of political conventions since 1960, Sabato concluded that post-convention polls signal the election's outcome only about half the time. "You could flip a coin and be about as predictive," he says. "It is really surprising how quickly convention memories fade." (emphasis added)
I guess that fits right in with the idea that the media likes to hype the closeness of the race -- but then, why bother to contrive a 10-point lead at all?
5. Gallup admits...
Here is part of the quotation from Gallup Poll editor-in-chief Frank Newport that one diarist construed as a sort of smoking gun:
Second, we are at this point reporting likely voter estimates on only an occasional basis. We feel that the trends among registered voters give us the best way to track election preferences in our daily poll, in part because many voters are not yet in a position to accurately estimate their chances of voting on Election Day. But from time to time, we do estimate (and report) likely voter results to give us a feel for the potential difference turnout could make in November. [emphasis in original]
(This statement might have given some people the impression that the USA Today/Gallup polls hadn't previously reported likely voter estimates -- although Newport is primarily talking about the daily tracking poll.)
Did Newport "admit" believing that likely voter results are "less accurate"? No. He said that registered voter trends are, or may be, the best way to track daily trends. This could be true if, at any given time, a spike in Obama supporters' or McCain supporters' enthusiasm might create a transitory spike in the topline results, which would not reliably reflect underlying trends. (I'm not splitting hairs here: the only way to tell whether the likely voter results are "less accurate" would be actually to conduct a snap election -- i.e., there actually is no way to tell.) There is some evidence for Newport's view: Erikson et al. concluded that in 2000, the likely voter results in the CNN/USA Today/Gallup tracking poll fluctuated more than the registered voter results. Probably influenced by that analysis and their own internal research, Gallup decided not to use the likely voter model in its tracker -- but USA Today decided to report both likely-voter and registered-voter results. As Newport noted in late July, earlier in the year the likely voter numbers had tended to favor Obama!
Perspective
It's reasonable to question whether these likely-voter numbers are worth reporting at all, and if so, how they should be reported. Newport wrote back in July that the likely-voter result "months before an election shows the potential that voter turnout can have on the popular vote outcome, but is not a predictor of what that turnout will look like on Election Day." I think that USA Today's report (which, again, led with the RV numbers and presented the LV numbers six paragraphs later) was in that spirit -- and, as an information hound, I'm grateful that USA Today presented both results -- but someone could reasonably think that we would have been better off without the likely voter "information."
But I can't find the reason in the specific allegations of professional misconduct lodged against Gallup, in these two diaries and many of the comments attached to them. I hope Kossacks will think twice and check facts carefully before posting or recommending diaries that impugn people's integrity. It seems like the right thing to do.