« previous post | Main | next post »
January 31, 2005
explaining the exit polls?
Paul F. Velleman: January 31, 2005
In the 1970’s and 80’s I worked on election nights for a major network as part of a team of statisticians making “calls” in statewide races (President, Senator, and Governor). Eventually, the team was disbanded because exit polls were so accurate that our expertise was no longer needed.
But in the past election, the exit polls differed from the recorded vote by an unprecedented amount. Nationwide, exit polls predicted that Kerry had won by 3%, but the final tally showed Bush ahead by 2.5%. Errors in some key states were even larger. As a statistician, I have been concerned that the errors were unexplained. Last week, Edison Media Research and Mitofsky International, who conducted the exit polls on contract for the major networks and news services, released a report on the errors in their polls. Now an organization called US Count Votes has released an analysis of the E/M data that raises serious questions about E/M’s proposed explanations for the exit poll biases.
All who have considered the problem agree that there are three plausible explanations:
1. Chance error,
2. Bias in the exit polls, and
3. Inaccurate election tallies, or (say it softly) election fraud.
Edison/Mitofsky and all who have examined the data agree that (1) is not plausible. The errors were so extraordinarily beyond what could occur simply by chance that we can safely exclude this possibility. Anyone who has taken a freshman Statistics course can do the calculations.
The E/M analysis arbitrarily ignores (3) and considers only possible biases. In one sense, this is understandable. Their job was to predict the final vote totals and they failed miserably. Some commentators were made to sound foolish when they started talking as if Kerry had won by about 7pm on election night. Clearly E/M’s clients didn’t get their money’s worth.
But as citizens, our job is somewhat different. If the bias in the exit polls can be explained by errors in E/M's methods or by other factors, it would reduce the concern for the vote itself. This analysis of the E/M data makes it clear that the E/M report fails to provide such an explanation. And, of course, if the exit polls were not themselves flawed, that would raise questions about the honesty of the vote itself.
I have looked at the E/M data and participated in the discussion as the US Count Votes document was refined. I have signed their report as a member of the team that reviewed the work. I am posting this commentary here to direct readers of the blog to that report. I think their points warrant serious consideration and that they clarify the need for further analyses of the voting and polling data.
The underlying scientific consideration is that any theory or model that claims to account for patterns in data must make sense no matter how we view the data and must be consistent with other external information. Specifically, if there are biases inherent in the E/M exit polls, we should expect them to follow their own consistent patterns. For example, it isn’t plausible to posit different biases in different places without offering any account of the differences. Such ad hoc explanations are not scientifically or statistically supportable.
Nor is it plausible that a bias present in some locations should inexplicably be absent in others. Conversely, if there is a pattern to the biases that makes sense, the very existence of such a pattern makes the proposed explanation more plausible.
For example, one early explanation of the exit poll errors was that (more Democratic) women tend to vote early, while (more Republican) men vote later. Such an explanation could be supported by finding trends in the biases during election day (E/M reported poll results in three waves). But no such pattern is found, and E/M do not propose that explanation. They analyze data from the end of the polling day, and it is those data that show the biases they are trying to explain.
So what do they propose? E/M’s explanation is simply that Bush voters were substantially more likely than Kerry voters to decline to be interviewed. (Specifically that 56% of Bush voters but only 50% of Kerry voters declined.) E/M offer no evidence of this other than the obvious fact that the polls don’t match the recorded vote and the unstated fact that they can find no other explanation. But there are no underlying patterns to support their explanation. For example, E/M look for patterns in refusals and find none. In fact, as the US Count Votes report points out, the response rate of voters willing to be polled is actually a bit larger (although probably not significantly so) in states that voted more strongly for Bush—the opposite of the pattern that would support E/M's hypothesis.
Does the E/M hypothesis account for other patterns in the data? They propose no such explanations and admit to some patterns that their hypothesis fails to explain. For example, states that voted with paper ballots showed only small random errors between exit polls and votes, well within statistical error. States that used automated systems showed large errors fairly consistently biased toward Kerry. It doesn’t seem plausible that voting method would influence a voter’s willingness to talk with pollsters (nor do E/M claim that this happened).
Let me be very clear. I do not assert there was extensive fraud. I would prefer not to think that, and I had hoped the E/M report would reveal a systematic flaw in their methods that accounted for the errors. But it hasn’t, and the issue is still open. The E/M report does not account for the biases in a manner that would support explanation (2). The exit polls may well have been flawed, but we have yet to see a plausible account of how or why.
The data released thus far beg for a more thorough analysis. E/M have not released precinct-level data, which would be necessary to determine whether voting technology is a factor. I hope that they will do so soon. I also hope that the news media report this story so that the public can be widely informed about it. I recognize that if significant problems with the reported vote are found, Republicans will feel that the effort was somehow directed against them. But honest voting is a value that should be supported equally by both Left and Right. We cite discrepancies between exit polls and votes in elections in other countries as evidence of problems. Especially when we have been called to spread liberty and democracy throughout the world, it behooves us to make our own democracy as open and honest as we can.
TrackBack
TrackBack URL for this entry:
http://www.typepad.com/services/trackback/6a00d834536ae669e200d83457cf8f69e2
Listed below are links to weblogs that reference explaining the exit polls?:
» Exit Poll Post Mortem from E-Voting News and Analysis, from the Experts
Paul Velleman at left2right has an interesting analysis of the exit poll data from the November presidential election.
His conclusions:
(1) Discrepancies between exit polls and votes cannot be explained by random chance, so the discrepancies m... [Read More]
Tracked on Jan 31, 2005 8:43:01 AM
» Explaining the Exit Polls from Moonage Political Webdream
Left2Right does a lengthy job doing what most media is trying to do, understand what went wrong with the exit polls.The exit polls may well have been flawed, but we have yet to see a plausible account of how or [Read More]
Tracked on Jan 31, 2005 10:46:20 PM
» Evaluating the Exit Polls from PBS Watch
...it is probably easier and cheaper to rig the poll results than to rig the election results. ... I reject the notion that the lack of a purely statistical explanation for exit poll bias necessarily implies election fraud. [Read More]
Tracked on Feb 4, 2005 10:24:00 PM
» Remaining Questions About the US Election from SIVACRACY.NET: Siva Vaidhyanathan's Weblog
One of my favorite professors at Cornell taught statistics, and recently he started blogging about something important: Explaining the exit polls? Posted by Paul Velleman at Left2Right on January 31, 2005 In the 1970’s and 80’s I worked on ... [Read More]
Tracked on Feb 6, 2005 4:17:45 PM
Comments
Posted by: Tom Perkins
For various reasons I don't care to elaborate right now, the night of Nov. 1st I drove to Columbus Ohio from northern Viginia to work the phones for the Republican GOTV campaign, the morning of the 2nd I drove back to work. I've never done anything like that before--I'd never given money to any political campaign, only to to the Libertarian party. I do not at this time anticipate supporting the Republicans or any other party with like efforts in the future, I'll wait and see what and who they come up with. I was on the phones from about 7am to 7:15pm. I can tell you from personal experience that at about 5pm and on, I increaqsingly called teenage children and babysitters who reported that, "Mommy and Daddy went to vote for the President, they're not hear right now."
That's nothing I heard even once before 5pm. I do believe the earliest polls exagerated support for Kerry. It was a sampling error, likely caused by the work or other personal habits of the people voting that early in the day.
It is no mystery to me that early exit polls were wrong, the Bush voters were working for their living, they voted late. At 7:15, I was still telling them (by voice messages, gennerally they weren't home) on the phone that if they got to the polls by 7:30, they (the poll workers), had to let them vote.
I know lots of Republicans in rural areas waited in line to vote in cold wet weather as late as 11:30--it was not a problem unique to heavily Democratic urban areas.
That's all there is, there's no conspiracy to steal OH from Kerry.
I wouldn't be so adamant about it if I wasn't there to see some of it for myself.
You lost. GET THE FUCK OVER IT!
Yours, TDP, ml, msl, & pfpp
Posted by: Tom Perkins | Jan 31, 2005 10:23:43 AM
Posted by: Tom Perkins
I should add that I am sure that each party did their level best to win, and that I do not think that there were more shenanigans than usual in this election.
But. I agree that voter verified paper records of votes are required for electoral integirty to be maintained. This is obvious.
It is also obvious to me that the Democrats resist equally obvious measures required for the integrity of the electoral process.
So I tell you what (although I think agreement should be untroublesome to all parties of honest intent), we'll trade you voter verified triplcate records for voter registration mechanisms which ensure legaly valid voters vote only once and in their precincts of legal residence, and only if they are of good legal standing (not felons, are citizens).
How does that strike you?
Yours, TDP, ml, msl, & pfpp
Posted by: Tom Perkins | Jan 31, 2005 10:36:43 AM
Posted by: Paul Deignan
Exit polls rely on voluntary participation and exposes the respondent to costs and privacy concerns that the actual vote does not. Furthermore, the exit polls were conducted by one firm at a particular time and particular locations.
Now, consider that some other polls were wildly off on the eve of the election -- esp. Zogby. Other polls tended to be consistent over the run up to the election such as Pew.
Note also that there is a persistent trend among some to systematically make very very poor predictions, e.g. the outcome of the Iraqi elections, Afghanistan elections, Nicaragua, El Salvador, the USSR's demise etc. There is a trend here.
Furthermore, it has been found that there is a statistically significant discriminant between the characteristic information processing methods (personalities) of conservatives and liberals in such a way as would explain the systematic failure of liberals to predict events.
The question is, "Are we ready to deal with reality?" This is a very good time for examination and analysis. The US functions best with two functional political parties.
Posted by: Paul Deignan | Jan 31, 2005 10:46:23 AM
Posted by: Paul Deignan
Here is a link to polling leading up to the election -- there are internals here as well: Polls
Posted by: Paul Deignan | Jan 31, 2005 10:50:58 AM
Posted by: noah
How about random bias in favor of Kerry on the part of the exit-poll takers? I have read that there is evidence of such bias among the younger cohort.
There is much more palpable evidence that the Washington governors race was stolen by the Democrats...why does this not concern you? HMMM!?
Posted by: noah | Jan 31, 2005 10:52:37 AM
Posted by: CDC
Fraud is possible, but not likely. Too many people with loose tongues would have to be involved and too many LIBERAL DEMOCRAT reporters would be hot on the trail. It positively would come out and the feeding frenzy would make any other political scandal in American history seem trivial. The potential downside so outweighs the potential upside, and the odds of success are so small, that a fraud of this type would be stupid to attempt. Rove isn't stupid.
I think it's much too early to give up on case 2.
Posted by: CDC | Jan 31, 2005 10:55:42 AM
Posted by: Tom Perkins
Ah shoot, that should read, morning of the 3rd. Right around where 522 cuts south from I70, I was starting to get a little blurry headed. If Bush pulls of some lese majeste with respect to the scared cow of Social Security, it'll be well worth it. Yours, TDP, ml, msl, & pfpp
Posted by: Tom Perkins | Jan 31, 2005 10:58:26 AM
Posted by: pedro
Very informative post, Paul V. It's interesting and dispiriting to witness how the problem of making sense of the discrepancies between exit polls and vote tally--a problem which requires technical statistical knowledge to be tackled sensibly--is easily explained away, with formidable simplitude, by people with little knowledge of the statistics involved, let alone of the data.
Posted by: pedro | Jan 31, 2005 11:09:46 AM
Posted by: Tom Perkins
Pedro, I was there on the phones. Were you? Yours, TDP, ml, msl, & pfpp
Posted by: Tom Perkins | Jan 31, 2005 11:13:15 AM
Posted by: pedro
So, I'm to rely on your anecdotes, Tom? Do you really think I'm naive enough to dismiss the observations of professional statisticians simply because you worked the phones for the GOP?
Posted by: pedro | Jan 31, 2005 11:55:24 AM
Posted by: D.A. Ridgely
As one of those formidably simplitudinous folks who can’t tell a chi square from a chi rho and whose knowledge of statistics doesn’t extend much beyond confidence in the belief that 49% of all statisticians are below average, put me down for “all of the above.”
I would never rule out the possibility of some voter fraud in any election. If anything, I think I would be more inclined to rule out the possibility of no voter fraud. Even so, Mr. Velleman doesn’t seem to be considering that there could be multiple factors contributing to the disconnect between the polls and the results. Moreover, the fact that plausible explanations have not yet been forthcoming is hardly the same thing as there being no such plausible explanations.
In any case, and I say this as someone who is sufficiently innumerate that I actually do break down and buy a lottery ticket from time to time, can’t a penny randomly turn up heads a hundred times in a row? I mean, don’t flukes happen?
Posted by: D.A. Ridgely | Jan 31, 2005 11:58:47 AM
Posted by: CDC
Pedro: "...a problem which requires technical statistical knowledge to be tackled sensibly--is easily explained away, with formidable simplitude, by people with little knowledge of the statistics involved..."
Thank you for your concern but the report seemed pretty straight forward.
Posted by: CDC | Jan 31, 2005 11:58:57 AM
Posted by: Tom Perkins
My anecdote would I think be repeated as the experience of all present.
You can conclude we are all in on the conspiracy or conclude the initial exit polls were at best accurate for the people having voted at the time--who were not nearly everyone who was going to vote that day.
What says Prof. Velleman with regard to my personal experience. Has he interviewed other GOTV workers? Would he regard the results of those interviews as credible?
And what would he say about both parties multilaterally disarming with respect to their putative favored vote fraud methods? A trade of voter verified paper records for systems making it very difficult for someone to vote more than once and then only if they are citizens legally able to vote in the precinct where they show up, and actually prosecuting fraudulent voters?
Yours, TDP, ml, msl, & pfpp
Posted by: Tom Perkins | Jan 31, 2005 12:17:49 PM
Posted by: Will Curtis
I think the US Counts Vote brief is interesting and their hypothesis is plausible. But if I were a betting man I would propose the following hypothesis of my own:
Instead of suggesting that their was a nationwide effort to systematically shift the vote in favor of Bush, why not suggest that there was a systematic nationwide effort to shift the exit poll data.
This hypothesis is more likely, in my view, because whereas vote counting is overseen and scrutinzed in the various states and counties by people *from both parties*, the collection of exit poll data is not done with any overt counter-fraud measures. Thus, it is the *exit poll* fraud which scares me, not the much less likely vote-count fraud.
How much easier would it be to skew exit poll results? Think about it.
Posted by: Will Curtis | Jan 31, 2005 12:20:26 PM
Posted by: mikec
As a libertarian who voted for neither Bush nor Kerry, I saw a huge difference in attitude between Bush and Kerry supporters. Kerry supporters assumed I was one of them. They started conversations with statements that would have been insulting had I been a Bush supporter: implying that Bush was stupid and evil, and that if I supported him, I was, too. Bush supporters were much more reticent about their views; in fact, they usually didn't state a preference at all unless they got some encouragement.
My guess is that, by election day, quite a few Bush supporters were tired of being insulted and demeaned and decided to keep their mouths shut.
Posted by: mikec | Jan 31, 2005 12:48:28 PM
Posted by: pedro
"You can conclude we are all in on the conspiracy or conclude the initial exit polls were at best accurate for the people having voted at the time--who were not nearly everyone who was going to vote that day."
Or you can refrain from jumping to conclusions before the data is examined properly. Nobody is arguing that it is pertinent to conclude right now that there was significant fraud in the election. But it is perfectly reasonable to seriously scrutinize all possibilities, including the possibility that exit polls coincidentally happened to lose accuracy in precincts with a particular kind of voter technology.
In the aftermath of the 9-11 attacks, a number of financial experts speculated that it would be possible to detect risk of attacks by examining patterns of unusual option market activity, in particular fluctuations in the case of put/call volume ratios. A normal put/call ratio is in the neighborhood of 1, yet in the days before 9-11, both put/call ratios for American and United were exceptionally high. Does it make sense to analyze the data so as to have predictors of risk of attacks? Yes, it does, regardless of how promising the method may be.
Similarly, it behooves us to analyze patterns in the voting data to evaluate risk of fraud. Whether systematic, widespread, significant fraud occurred is very hard to know, but I'm inclined to disbelieve it. But a careful examination of the data by professional statisticians, and a careful evaluation of the voting technologies by professional engineers is entirely appropriate.
Posted by: pedro | Jan 31, 2005 12:54:55 PM
Posted by: Paul Deignan
I perceive a political bias in the MSM that makes me wary of pollsters. Why? Generic fear of contributing to an effort that might misrepresent me such as the fear of being misquoted and scandalized by unethical partisan reporters. Unfounded? Unfortunately not. We can read the pages here (pedro -- you are guilty of this). The fact that this tactic is employed by one party disproportionately is a function of their internal dynamics (see the blog).
I will bet, that there is a politically related dependency between response rates and party affiliation/voting preference. In fact, there are reports that tend to support this proposition.
There is a reason for the polarization -- that is not by chance either as the red-blue maps should make clear to all.
Posted by: Paul Deignan | Jan 31, 2005 12:57:16 PM
Posted by: Paul Deignan
Here is one link: Exit Poll
Posted by: Paul Deignan | Jan 31, 2005 1:14:13 PM
Posted by: Paul Deignan
BTW, for those interested, the reaon for the intimidation is directly related to the personality shift between liberals and conservatives. Liberals feel a need for agreement and ratification while conservatives are more comfortable with rational disagreements.
The results are here: Personality Study Results
Posted by: Paul Deignan | Jan 31, 2005 1:20:57 PM
Posted by: Tom Perkins
pedro, it is only prudent to expend resources scrutinizing serious possibilities. There is as yet only thin statistically derived evidence of any vote fraud/irregularity in OH, and I can with personal experience provide testimony that would tend to explain such thin statistical evidence as resulting from something other than fraud.
The put/call ration for the corporate stock you mention would first have to be shown to be at least a few standard devaitions away from the delta of such ratios in like circumstances before I would call it predictive.
Will Curtis' mention of "exit poll" fraud deserves at least as much attention as investigations of "vote" fraud.
And in the vein of that meme, "a careful evaluation of the voting technologies by professional engineers is entirely appropriate" Prof. Velleman certainly may be a professional statistician, by appearing in this forum in this fashion, he also declares himself to be a partisan one. Curtis' possibility would I think have been possibility number 4 in a list of explanation compiled by a perceptive, uninvolved person.
This list:
"1. Chance error,
2. Bias in the exit polls, and
3. Inaccurate election tallies, or (say it softly) election fraud."
Is incomplete.
Yours, TDP, ml, msl, & pfpp
Posted by: Tom Perkins | Jan 31, 2005 1:21:16 PM
Posted by: pedro
Paul D.: as usual, you make little sense to me. (If only you took it upon yourself to write more clearly.) You accuse me of behaving like an unethical partisan reporter, yet you provide no argument to support your ridiculuous accusation. I imagine you must be reading between the lines.
Posted by: pedro | Jan 31, 2005 1:22:35 PM
Posted by: CDC
Pedro: "But a careful examination of the data by professional statisticians...is entirely appropriate."
Fine. Then publish it.
Posted by: CDC | Jan 31, 2005 1:24:12 PM
Posted by: Doug
Dr. Velleman, I had read a (to this layman) plausible analysis of the exit poll failures on Mystery Pollster a few weeks back. The systemic factor he identifies is young, undertrained exit poll workers. He provides a couple of hypotheticals about how a poorly-trained college student might unwittingly pick a non-random sample or undercount voters with which she disagreed.
Do you agree with that analysis? Or if you don't feel it's supported by the data you examined, why not?
Posted by: Doug | Jan 31, 2005 1:24:35 PM
Posted by: AlanC9
The issue isn't that the exit polls underreported one side or the other. That could have a lot of a reasonable explanations. After all, Bush and Kerry voters are different from one another.
There's an issue if the variation between the reported vote and the exit polls is related to the voting technology. That shoudn't happen, unless there's some sort of demographic distinction between districts with electronic voting machines and disctricts without them.
Is there? Or is the variation not really related to the voting technology? D.A., Paul, tom.... any explanation for me? (Believe me, the last thing I want is to actually have to think there's a problem here.)
Posted by: AlanC9 | Jan 31, 2005 1:42:48 PM
Posted by: Paul Deignan
AlanC9
That shoudn't happen, unless there's some sort of demographic distinction between districts with electronic voting machines and disctricts without them. Is there?
Yes, yes, yes, yes, yes,yes, yes, yes, yes!
There is a statistically significant intrinsic difference between partisan democrats and republicans -- personality!
This factor is exactly the right factor in the right proportion to exactly explain what we are seeing here. Believe it or not, this is an original observation (the fact that the difference is measurable and significant).
Posted by: Paul Deignan | Jan 31, 2005 1:55:32 PM