« previous post | Main | next post »
January 31, 2005
explaining the exit polls?
Paul F. Velleman: January 31, 2005
In the 1970’s and 80’s I worked on election nights for a major network as part of a team of statisticians making “calls” in statewide races (President, Senator, and Governor). Eventually, the team was disbanded because exit polls were so accurate that our expertise was no longer needed.
But in the past election, the exit polls differed from the recorded vote by an unprecedented amount. Nationwide, exit polls predicted that Kerry had won by 3%, but the final tally showed Bush ahead by 2.5%. Errors in some key states were even larger. As a statistician, I have been concerned that the errors were unexplained. Last week, Edison Media Research and Mitofsky International, who conducted the exit polls on contract for the major networks and news services, released a report on the errors in their polls. Now an organization called US Count Votes has released an analysis of the E/M data that raises serious questions about E/M’s proposed explanations for the exit poll biases.
All who have considered the problem agree that there are three plausible explanations:
1. Chance error,
2. Bias in the exit polls, and
3. Inaccurate election tallies, or (say it softly) election fraud.
Edison/Mitofsky and all who have examined the data agree that (1) is not plausible. The errors were so extraordinarily beyond what could occur simply by chance that we can safely exclude this possibility. Anyone who has taken a freshman Statistics course can do the calculations.
The E/M analysis arbitrarily ignores (3) and considers only possible biases. In one sense, this is understandable. Their job was to predict the final vote totals and they failed miserably. Some commentators were made to sound foolish when they started talking as if Kerry had won by about 7pm on election night. Clearly E/M’s clients didn’t get their money’s worth.
But as citizens, our job is somewhat different. If the bias in the exit polls can be explained by errors in E/M's methods or by other factors, it would reduce the concern for the vote itself. This analysis of the E/M data makes it clear that the E/M report fails to provide such an explanation. And, of course, if the exit polls were not themselves flawed, that would raise questions about the honesty of the vote itself.
I have looked at the E/M data and participated in the discussion as the US Count Votes document was refined. I have signed their report as a member of the team that reviewed the work. I am posting this commentary here to direct readers of the blog to that report. I think their points warrant serious consideration and that they clarify the need for further analyses of the voting and polling data.
The underlying scientific consideration is that any theory or model that claims to account for patterns in data must make sense no matter how we view the data and must be consistent with other external information. Specifically, if there are biases inherent in the E/M exit polls, we should expect them to follow their own consistent patterns. For example, it isn’t plausible to posit different biases in different places without offering any account of the differences. Such ad hoc explanations are not scientifically or statistically supportable.
Nor is it plausible that a bias present in some locations should inexplicably be absent in others. Conversely, if there is a pattern to the biases that makes sense, the very existence of such a pattern makes the proposed explanation more plausible.
For example, one early explanation of the exit poll errors was that (more Democratic) women tend to vote early, while (more Republican) men vote later. Such an explanation could be supported by finding trends in the biases during election day (E/M reported poll results in three waves). But no such pattern is found, and E/M do not propose that explanation. They analyze data from the end of the polling day, and it is those data that show the biases they are trying to explain.
So what do they propose? E/M’s explanation is simply that Bush voters were substantially more likely than Kerry voters to decline to be interviewed. (Specifically that 56% of Bush voters but only 50% of Kerry voters declined.) E/M offer no evidence of this other than the obvious fact that the polls don’t match the recorded vote and the unstated fact that they can find no other explanation. But there are no underlying patterns to support their explanation. For example, E/M look for patterns in refusals and find none. In fact, as the US Count Votes report points out, the response rate of voters willing to be polled is actually a bit larger (although probably not significantly so) in states that voted more strongly for Bush—the opposite of the pattern that would support E/M's hypothesis.
Does the E/M hypothesis account for other patterns in the data? They propose no such explanations and admit to some patterns that their hypothesis fails to explain. For example, states that voted with paper ballots showed only small random errors between exit polls and votes, well within statistical error. States that used automated systems showed large errors fairly consistently biased toward Kerry. It doesn’t seem plausible that voting method would influence a voter’s willingness to talk with pollsters (nor do E/M claim that this happened).
Let me be very clear. I do not assert there was extensive fraud. I would prefer not to think that, and I had hoped the E/M report would reveal a systematic flaw in their methods that accounted for the errors. But it hasn’t, and the issue is still open. The E/M report does not account for the biases in a manner that would support explanation (2). The exit polls may well have been flawed, but we have yet to see a plausible account of how or why.
The data released thus far beg for a more thorough analysis. E/M have not released precinct-level data, which would be necessary to determine whether voting technology is a factor. I hope that they will do so soon. I also hope that the news media report this story so that the public can be widely informed about it. I recognize that if significant problems with the reported vote are found, Republicans will feel that the effort was somehow directed against them. But honest voting is a value that should be supported equally by both Left and Right. We cite discrepancies between exit polls and votes in elections in other countries as evidence of problems. Especially when we have been called to spread liberty and democracy throughout the world, it behooves us to make our own democracy as open and honest as we can.
TrackBack
TrackBack URL for this entry:
https://www.typepad.com/services/trackback/6a00d834536ae669e200d83457cf8f69e2
Listed below are links to weblogs that reference explaining the exit polls?:
» Exit Poll Post Mortem from E-Voting News and Analysis, from the Experts
Paul Velleman at left2right has an interesting analysis of the exit poll data from the November presidential election.
His conclusions:
(1) Discrepancies between exit polls and votes cannot be explained by random chance, so the discrepancies m... [Read More]
Tracked on Jan 31, 2005 8:43:01 AM
» Explaining the Exit Polls from Moonage Political Webdream
Left2Right does a lengthy job doing what most media is trying to do, understand what went wrong with the exit polls.The exit polls may well have been flawed, but we have yet to see a plausible account of how or [Read More]
Tracked on Jan 31, 2005 10:46:20 PM
» Evaluating the Exit Polls from PBS Watch
...it is probably easier and cheaper to rig the poll results than to rig the election results. ... I reject the notion that the lack of a purely statistical explanation for exit poll bias necessarily implies election fraud. [Read More]
Tracked on Feb 4, 2005 10:24:00 PM
» Remaining Questions About the US Election from SIVACRACY.NET: Siva Vaidhyanathan's Weblog
One of my favorite professors at Cornell taught statistics, and recently he started blogging about something important: Explaining the exit polls? Posted by Paul Velleman at Left2Right on January 31, 2005 In the 1970’s and 80’s I worked on ... [Read More]
Tracked on Feb 6, 2005 4:17:45 PM
Comments
Posted by: Tom Perkins
For various reasons I don't care to elaborate right now, the night of Nov. 1st I drove to Columbus Ohio from northern Viginia to work the phones for the Republican GOTV campaign, the morning of the 2nd I drove back to work. I've never done anything like that before--I'd never given money to any political campaign, only to to the Libertarian party. I do not at this time anticipate supporting the Republicans or any other party with like efforts in the future, I'll wait and see what and who they come up with. I was on the phones from about 7am to 7:15pm. I can tell you from personal experience that at about 5pm and on, I increaqsingly called teenage children and babysitters who reported that, "Mommy and Daddy went to vote for the President, they're not hear right now."
That's nothing I heard even once before 5pm. I do believe the earliest polls exagerated support for Kerry. It was a sampling error, likely caused by the work or other personal habits of the people voting that early in the day.
It is no mystery to me that early exit polls were wrong, the Bush voters were working for their living, they voted late. At 7:15, I was still telling them (by voice messages, gennerally they weren't home) on the phone that if they got to the polls by 7:30, they (the poll workers), had to let them vote.
I know lots of Republicans in rural areas waited in line to vote in cold wet weather as late as 11:30--it was not a problem unique to heavily Democratic urban areas.
That's all there is, there's no conspiracy to steal OH from Kerry.
I wouldn't be so adamant about it if I wasn't there to see some of it for myself.
You lost. GET THE FUCK OVER IT!
Yours, TDP, ml, msl, & pfpp
Posted by: Tom Perkins | Jan 31, 2005 10:23:43 AM
Posted by: Tom Perkins
I should add that I am sure that each party did their level best to win, and that I do not think that there were more shenanigans than usual in this election.
But. I agree that voter verified paper records of votes are required for electoral integirty to be maintained. This is obvious.
It is also obvious to me that the Democrats resist equally obvious measures required for the integrity of the electoral process.
So I tell you what (although I think agreement should be untroublesome to all parties of honest intent), we'll trade you voter verified triplcate records for voter registration mechanisms which ensure legaly valid voters vote only once and in their precincts of legal residence, and only if they are of good legal standing (not felons, are citizens).
How does that strike you?
Yours, TDP, ml, msl, & pfpp
Posted by: Tom Perkins | Jan 31, 2005 10:36:43 AM
Posted by: Paul Deignan
Exit polls rely on voluntary participation and exposes the respondent to costs and privacy concerns that the actual vote does not. Furthermore, the exit polls were conducted by one firm at a particular time and particular locations.
Now, consider that some other polls were wildly off on the eve of the election -- esp. Zogby. Other polls tended to be consistent over the run up to the election such as Pew.
Note also that there is a persistent trend among some to systematically make very very poor predictions, e.g. the outcome of the Iraqi elections, Afghanistan elections, Nicaragua, El Salvador, the USSR's demise etc. There is a trend here.
Furthermore, it has been found that there is a statistically significant discriminant between the characteristic information processing methods (personalities) of conservatives and liberals in such a way as would explain the systematic failure of liberals to predict events.
The question is, "Are we ready to deal with reality?" This is a very good time for examination and analysis. The US functions best with two functional political parties.
Posted by: Paul Deignan | Jan 31, 2005 10:46:23 AM
Posted by: Paul Deignan
Here is a link to polling leading up to the election -- there are internals here as well: Polls
Posted by: Paul Deignan | Jan 31, 2005 10:50:58 AM
Posted by: noah
How about random bias in favor of Kerry on the part of the exit-poll takers? I have read that there is evidence of such bias among the younger cohort.
There is much more palpable evidence that the Washington governors race was stolen by the Democrats...why does this not concern you? HMMM!?
Posted by: noah | Jan 31, 2005 10:52:37 AM
Posted by: CDC
Fraud is possible, but not likely. Too many people with loose tongues would have to be involved and too many LIBERAL DEMOCRAT reporters would be hot on the trail. It positively would come out and the feeding frenzy would make any other political scandal in American history seem trivial. The potential downside so outweighs the potential upside, and the odds of success are so small, that a fraud of this type would be stupid to attempt. Rove isn't stupid.
I think it's much too early to give up on case 2.
Posted by: CDC | Jan 31, 2005 10:55:42 AM
Posted by: Tom Perkins
Ah shoot, that should read, morning of the 3rd. Right around where 522 cuts south from I70, I was starting to get a little blurry headed. If Bush pulls of some lese majeste with respect to the scared cow of Social Security, it'll be well worth it. Yours, TDP, ml, msl, & pfpp
Posted by: Tom Perkins | Jan 31, 2005 10:58:26 AM
Posted by: pedro
Very informative post, Paul V. It's interesting and dispiriting to witness how the problem of making sense of the discrepancies between exit polls and vote tally--a problem which requires technical statistical knowledge to be tackled sensibly--is easily explained away, with formidable simplitude, by people with little knowledge of the statistics involved, let alone of the data.
Posted by: pedro | Jan 31, 2005 11:09:46 AM
Posted by: Tom Perkins
Pedro, I was there on the phones. Were you? Yours, TDP, ml, msl, & pfpp
Posted by: Tom Perkins | Jan 31, 2005 11:13:15 AM
Posted by: pedro
So, I'm to rely on your anecdotes, Tom? Do you really think I'm naive enough to dismiss the observations of professional statisticians simply because you worked the phones for the GOP?
Posted by: pedro | Jan 31, 2005 11:55:24 AM
Posted by: D.A. Ridgely
As one of those formidably simplitudinous folks who can’t tell a chi square from a chi rho and whose knowledge of statistics doesn’t extend much beyond confidence in the belief that 49% of all statisticians are below average, put me down for “all of the above.”
I would never rule out the possibility of some voter fraud in any election. If anything, I think I would be more inclined to rule out the possibility of no voter fraud. Even so, Mr. Velleman doesn’t seem to be considering that there could be multiple factors contributing to the disconnect between the polls and the results. Moreover, the fact that plausible explanations have not yet been forthcoming is hardly the same thing as there being no such plausible explanations.
In any case, and I say this as someone who is sufficiently innumerate that I actually do break down and buy a lottery ticket from time to time, can’t a penny randomly turn up heads a hundred times in a row? I mean, don’t flukes happen?
Posted by: D.A. Ridgely | Jan 31, 2005 11:58:47 AM
Posted by: CDC
Pedro: "...a problem which requires technical statistical knowledge to be tackled sensibly--is easily explained away, with formidable simplitude, by people with little knowledge of the statistics involved..."
Thank you for your concern but the report seemed pretty straight forward.
Posted by: CDC | Jan 31, 2005 11:58:57 AM
Posted by: Tom Perkins
My anecdote would I think be repeated as the experience of all present.
You can conclude we are all in on the conspiracy or conclude the initial exit polls were at best accurate for the people having voted at the time--who were not nearly everyone who was going to vote that day.
What says Prof. Velleman with regard to my personal experience. Has he interviewed other GOTV workers? Would he regard the results of those interviews as credible?
And what would he say about both parties multilaterally disarming with respect to their putative favored vote fraud methods? A trade of voter verified paper records for systems making it very difficult for someone to vote more than once and then only if they are citizens legally able to vote in the precinct where they show up, and actually prosecuting fraudulent voters?
Yours, TDP, ml, msl, & pfpp
Posted by: Tom Perkins | Jan 31, 2005 12:17:49 PM
Posted by: Will Curtis
I think the US Counts Vote brief is interesting and their hypothesis is plausible. But if I were a betting man I would propose the following hypothesis of my own:
Instead of suggesting that their was a nationwide effort to systematically shift the vote in favor of Bush, why not suggest that there was a systematic nationwide effort to shift the exit poll data.
This hypothesis is more likely, in my view, because whereas vote counting is overseen and scrutinzed in the various states and counties by people *from both parties*, the collection of exit poll data is not done with any overt counter-fraud measures. Thus, it is the *exit poll* fraud which scares me, not the much less likely vote-count fraud.
How much easier would it be to skew exit poll results? Think about it.
Posted by: Will Curtis | Jan 31, 2005 12:20:26 PM
Posted by: mikec
As a libertarian who voted for neither Bush nor Kerry, I saw a huge difference in attitude between Bush and Kerry supporters. Kerry supporters assumed I was one of them. They started conversations with statements that would have been insulting had I been a Bush supporter: implying that Bush was stupid and evil, and that if I supported him, I was, too. Bush supporters were much more reticent about their views; in fact, they usually didn't state a preference at all unless they got some encouragement.
My guess is that, by election day, quite a few Bush supporters were tired of being insulted and demeaned and decided to keep their mouths shut.
Posted by: mikec | Jan 31, 2005 12:48:28 PM
Posted by: pedro
"You can conclude we are all in on the conspiracy or conclude the initial exit polls were at best accurate for the people having voted at the time--who were not nearly everyone who was going to vote that day."
Or you can refrain from jumping to conclusions before the data is examined properly. Nobody is arguing that it is pertinent to conclude right now that there was significant fraud in the election. But it is perfectly reasonable to seriously scrutinize all possibilities, including the possibility that exit polls coincidentally happened to lose accuracy in precincts with a particular kind of voter technology.
In the aftermath of the 9-11 attacks, a number of financial experts speculated that it would be possible to detect risk of attacks by examining patterns of unusual option market activity, in particular fluctuations in the case of put/call volume ratios. A normal put/call ratio is in the neighborhood of 1, yet in the days before 9-11, both put/call ratios for American and United were exceptionally high. Does it make sense to analyze the data so as to have predictors of risk of attacks? Yes, it does, regardless of how promising the method may be.
Similarly, it behooves us to analyze patterns in the voting data to evaluate risk of fraud. Whether systematic, widespread, significant fraud occurred is very hard to know, but I'm inclined to disbelieve it. But a careful examination of the data by professional statisticians, and a careful evaluation of the voting technologies by professional engineers is entirely appropriate.
Posted by: pedro | Jan 31, 2005 12:54:55 PM
Posted by: Paul Deignan
I perceive a political bias in the MSM that makes me wary of pollsters. Why? Generic fear of contributing to an effort that might misrepresent me such as the fear of being misquoted and scandalized by unethical partisan reporters. Unfounded? Unfortunately not. We can read the pages here (pedro -- you are guilty of this). The fact that this tactic is employed by one party disproportionately is a function of their internal dynamics (see the blog).
I will bet, that there is a politically related dependency between response rates and party affiliation/voting preference. In fact, there are reports that tend to support this proposition.
There is a reason for the polarization -- that is not by chance either as the red-blue maps should make clear to all.
Posted by: Paul Deignan | Jan 31, 2005 12:57:16 PM
Posted by: Paul Deignan
Here is one link: Exit Poll
Posted by: Paul Deignan | Jan 31, 2005 1:14:13 PM
Posted by: Paul Deignan
BTW, for those interested, the reaon for the intimidation is directly related to the personality shift between liberals and conservatives. Liberals feel a need for agreement and ratification while conservatives are more comfortable with rational disagreements.
The results are here: Personality Study Results
Posted by: Paul Deignan | Jan 31, 2005 1:20:57 PM
Posted by: Tom Perkins
pedro, it is only prudent to expend resources scrutinizing serious possibilities. There is as yet only thin statistically derived evidence of any vote fraud/irregularity in OH, and I can with personal experience provide testimony that would tend to explain such thin statistical evidence as resulting from something other than fraud.
The put/call ration for the corporate stock you mention would first have to be shown to be at least a few standard devaitions away from the delta of such ratios in like circumstances before I would call it predictive.
Will Curtis' mention of "exit poll" fraud deserves at least as much attention as investigations of "vote" fraud.
And in the vein of that meme, "a careful evaluation of the voting technologies by professional engineers is entirely appropriate" Prof. Velleman certainly may be a professional statistician, by appearing in this forum in this fashion, he also declares himself to be a partisan one. Curtis' possibility would I think have been possibility number 4 in a list of explanation compiled by a perceptive, uninvolved person.
This list:
"1. Chance error,
2. Bias in the exit polls, and
3. Inaccurate election tallies, or (say it softly) election fraud."
Is incomplete.
Yours, TDP, ml, msl, & pfpp
Posted by: Tom Perkins | Jan 31, 2005 1:21:16 PM
Posted by: pedro
Paul D.: as usual, you make little sense to me. (If only you took it upon yourself to write more clearly.) You accuse me of behaving like an unethical partisan reporter, yet you provide no argument to support your ridiculuous accusation. I imagine you must be reading between the lines.
Posted by: pedro | Jan 31, 2005 1:22:35 PM
Posted by: CDC
Pedro: "But a careful examination of the data by professional statisticians...is entirely appropriate."
Fine. Then publish it.
Posted by: CDC | Jan 31, 2005 1:24:12 PM
Posted by: Doug
Dr. Velleman, I had read a (to this layman) plausible analysis of the exit poll failures on Mystery Pollster a few weeks back. The systemic factor he identifies is young, undertrained exit poll workers. He provides a couple of hypotheticals about how a poorly-trained college student might unwittingly pick a non-random sample or undercount voters with which she disagreed.
Do you agree with that analysis? Or if you don't feel it's supported by the data you examined, why not?
Posted by: Doug | Jan 31, 2005 1:24:35 PM
Posted by: AlanC9
The issue isn't that the exit polls underreported one side or the other. That could have a lot of a reasonable explanations. After all, Bush and Kerry voters are different from one another.
There's an issue if the variation between the reported vote and the exit polls is related to the voting technology. That shoudn't happen, unless there's some sort of demographic distinction between districts with electronic voting machines and disctricts without them.
Is there? Or is the variation not really related to the voting technology? D.A., Paul, tom.... any explanation for me? (Believe me, the last thing I want is to actually have to think there's a problem here.)
Posted by: AlanC9 | Jan 31, 2005 1:42:48 PM
Posted by: Paul Deignan
AlanC9
That shoudn't happen, unless there's some sort of demographic distinction between districts with electronic voting machines and disctricts without them. Is there?
Yes, yes, yes, yes, yes,yes, yes, yes, yes!
There is a statistically significant intrinsic difference between partisan democrats and republicans -- personality!
This factor is exactly the right factor in the right proportion to exactly explain what we are seeing here. Believe it or not, this is an original observation (the fact that the difference is measurable and significant).
Posted by: Paul Deignan | Jan 31, 2005 1:55:32 PM
Posted by: pedro
Tom: unsurprisingly, some research has indeed been conducted regarding the violation of put/call parity in the leadup to 9-11. I'm not sure whether there's any predictive value in looking at substantial deviations from put-call parity, but the issue is one which competes the relevant professionals to evaluate, and it seems like they have evaluated it. Allen Poteshman, for example, has been studying how rare violations of put-call parity are in the stock market.
As for the possibility that only pollers in precincts with certain type of voting technology chose to commit fraud on the public by reporting biased numbers, I wouldn't find it very compelling. But I don't wish to speculate on the data, because I don't have it. Precinct-level data will help statisticians evaluate the possibilities.
Posted by: pedro | Jan 31, 2005 1:56:23 PM
Posted by: AlanC9
Paul, interesting study on your blog. Be nice if you broke it down by [i]types[/i] of liberals and conservatives, since those two labels aren't all that coherent. Maybe next time.
Posted by: AlanC9 | Jan 31, 2005 1:56:53 PM
Posted by: AlanC9
Paul, what does that have to do with the voting method in the district? Are majority-conservative districts more likely to have touch-screen voting?
Posted by: AlanC9 | Jan 31, 2005 1:58:48 PM
Posted by: Tom Perkins
AlanC9,
Actually, the issue IS "explaining the exit polls."
Prof. Velleman writes: "E/M have not released precinct-level data, which would be necessary to determine whether voting technology is a factor." So that is not an issue being examined by anyone yet, I suppose.
I personally am unaware of any assembled data at all breaking down precinct results vs exit poll discrepancies by polling technology for OH. Numerous examples are available for FL on this page, although they mention "by county" not "by precinct" data.
http://yalefreepress.blogspot.com/2004_11_01_yalefreepress_archive.html
Perhaps Prof. Velleman, or Dr. if he prefers, will able to supply that complete with any fudge factors or coefficients used in calculations, along with the data and equations/methods used, if and when it is available.
Yours, TDP, ml, msl, & pfpp
Posted by: Tom Perkins | Jan 31, 2005 1:59:39 PM
Posted by: AlanC9
(that's what I get for going back and forth between boards with UBB code and HTML boards)
Posted by: AlanC9 | Jan 31, 2005 1:59:48 PM
Posted by: Paul Deignan
AlanC9,
It was necessary to have only a single output dimension in order to have statistically significant results from that sample. A future study involving thousands under better control should include a political ideology questionnaire as you suggest.
I don't think any correlation with voting technology is causal in the exit polling discrepancies. I would, however, expect that the discrepancy be nonuniform and dependent on the subcultures of the areas and also upon the pollster demographic (there is probably insufficient data to draw any such relationship).
Do expect the discrepancy to be nonuniform and locality dependent.
Posted by: Paul Deignan | Jan 31, 2005 2:04:50 PM
Posted by: Paul Deignan
Pedro,
Paul D.: as usual, you make little sense to me. (If only you took it upon yourself to write more clearly.) You accuse me of behaving like an unethical partisan reporter, yet you provide no argument to support your ridiculuous accusation. I imagine you must be reading between the lines.
Just waiting for you to bite:
Here is your post: Pedro's post
Posted by: Paul Deignan | Jan 31, 2005 2:13:46 PM
Posted by: pedro
Paul: my post was a response to this.
Posted by: pedro | Jan 31, 2005 2:31:05 PM
Posted by: Chris
When Lou Holtz was head coach at Notre Dame they played someone (was it Michigan?) and they won the game. However, during the post game press conference the reporters being the expert statisticians that they are were quizzing Holtz on the fact that the opposing team had more first downs, more total yardage and the like. Holtz’s reply was that he thought it was the score that mattered and not the number of first downs and that if the winner was based on first downs then he would have made sure that he had more first downs than points.
Posted by: Chris | Jan 31, 2005 2:34:17 PM
Posted by: pedro
Incidentally, Paul: I would agree with someone who contests the idea that homosexuality is like being tall. My point was simply that even people who choose to be homosexual ought to have the same rights as those who choose otherwise.
Posted by: pedro | Jan 31, 2005 2:39:29 PM
Posted by: Paul Deignan
Pedro,
We do not claim that our interpretation (false or true) of another's statement is their own. Your intent was clear. I have seen it used before. It is a defense mechanism, advertisement of group membership, and warning to others.
Note here that you are directly addressing me in a show and tell/ask manner. This is the normal mode of rational disscussion when the purpose is to exchange information. Here we drop our defenses and open ourselves to ideas. "Do you see what I see? How do you think that? Why?"
The two methods of communication have different objectives. We can communicate on either level, but a mismatch of methods leads to noise and polarization. This is an implication of the personality study.
Posted by: Paul Deignan | Jan 31, 2005 2:58:46 PM
Posted by: Paul Velleman
A few quick responses to questions and observations raised here:
1)Tom P: The E/M data are from their third wave of polling, accumulated at
the end of the day. Any observations about who voted early or later are
irrelevant and cannot explain the patterns. Read the E/M report: they make
that clear.
2) Paul D: There is indeed some evidence about differences between
Republicans and Democrats in responding to polls. Pew Research has twice
performed the experiment in which they execute a standard 4 or 5-day
telephone poll and at the same time execute an intensive 3-month effort to
contact a parallel sample and ask the same questions. The first poll gets
completed questions from about 25%, the intensive one from about 75% of
the sample. They find that the telephone polls match the more thorough
survey with few differences, but one difference was that Republicans were
*more* likely to respond to the first poll -- by about 6%. If you know of
a study showing otherwise, I'd like to see it. This is the best study I
know of thus far.
3) Noah, Doug (and others): E/M looked for bias patterns vs age of the
interviewer (their interviewers coverd the full range of ages) and found
none. See their report.
4) Paul D: As I pointed out (and see the USCountVotes report) the only
pattern against party affiliation goes slightly in the *wrong* direction
for the E/M explanation. There's no evidence otherwise that I know of.
5) Exit poll "fraud" is an interesting thesis, but again, it calls for
some corroborating pattern. Should this have happened equally in all
states? More in Red or Blue? Could it possibly be related to voting
technology? I don't see any patterns in the data that support this and
some that seem to contradict it.
6) Voting technology. Yes, the E/M report includes a table (repeated in
the USCountVotes report) showing that voting technology has a strong
relationship with the bias. That is the most suspicious result in their
data both becuase it suggests fraud and because it contradicts most of the
alternative explanations.
7) Could the errors we've seen have happened by chance? Basically, no. As
a statistician, it is the virtual impossibility of this happening by
chance that caught my attention and that demands explanation. And E/M
readily acknowledge this in their report. It is the failure of the E/M
explanation (or others such as those proposed by discussants here) that
concerns me and raises the question of other causes.
Posted by: Paul Velleman | Jan 31, 2005 3:13:06 PM
Posted by: John F. Opie
Hi -
Gee, only three choices?
I don't mean to be snarky, but there are at least two others:
4) Incompetence on the part of the exit polling companies, their pollsters and statisticians
and
5) deliberate miscalculation to negatively impact the election results.
Now, I don't know whether this is the case: but ignoring these possibilities is at least as politically biased as your 3.
And having a great reputation doesn't protect you from incompetence and politically motivated bias. Even the best of companies can make mistakes.
Personally, I think it was bias in the exit polls. But if the exit polls were biased, this raises the question of 4 or 5: someone screwed up.
And after reviewing the article in question, I think that you are perhaps missing the point: exit polls don't tell you for whom people voted, it tells you what people are telling exit pollers for whom they voted. That's perhaps the fundamental flaw.
John
Posted by: John F. Opie | Jan 31, 2005 3:23:57 PM
Posted by: No Labels Please
The argument cited in the article that the sampling error should go the other way [response rate sightly higher in pro bush distrincts] is badly flawed.
What's important is not the absolute response rate for all voters, but the relative response rate between Bush and Kerry voters.
For example, it's easy to hypothesize that in smaller, more rural areas [more RED/BUSH areas] all voters are more amenable to being polled than voters in NYC and Los Angeles [I don't know that this is true - but it passes the smell test]. ***This however, would not imply anything about the relative repsonse rates of Kerry/Bush voters in these areas or in any other areas.***
And therefore, this overall increase in resposiveness in "Bush" states wouldn't necessarily lead to any bias at all in the calculation done.
This does not reflect favorably on the "statisticians" who wrote this article.
Comments, Mr. Velleman?
Posted by: No Labels Please | Jan 31, 2005 3:27:45 PM
Posted by: No Labels Please
I'd also like to propose an alternative explanation for the data:
Many people who actually voted for Bush lied and said they voted for Kerry in the exit poll.
It seems to me that supporting Bush became socially stigmatized for many people - ie extremely unpopular and "uncool". This was a result of the widespread unpopularity of the war and the virulence of the anti-Bush opposition
For example, many of us probably noticed that although the pre-election polls were about 50/50 Bush/Kerry, you would think that Kerry would win in a landslide if you observed the vocal anti-Bush sentiment at social gatherings, in academia, in the press, with the Europeans, etc...
My suspicion is that many people voted for Bush who may have not wanted to admit it in an exit poll becasue of latent fear of social stigmatization or actual genuine embarassment as their support was lukewarm.
Posted by: No Labels Please | Jan 31, 2005 3:42:55 PM
Posted by: frankly0
Paul Velleman,
My problem with the notion that it is election fraud that lies behind the exit poll discrepancies lies on the other end, in the inherent plausibility of your 3). I just don't see how election fraud could have been pulled off on such a massive scale that it wouldn't leak out.
Were partisan Republicans truly in positions where this scope of fraud might be perpetrated? How might this have been carried out? If you can't come up with a scenario whereby this might have happened, why should we entertain it? It has been primarily a failure to come up with such an account that has put me off from considering the possibility of election fraud.
I understand the political sensitivity of the point, but, honestly, I don't think it's quite right to simply suggest the possibility without providing some sketch at least of how it might have been made to work.
Posted by: frankly0 | Jan 31, 2005 3:45:13 PM
Posted by: Paul Deignan
2) Paul D: There is indeed some evidence about differences between Republicans and Democrats in responding to polls. Pew Research has twice performed the experiment in which they execute a standard 4 or 5-day telephone poll and at the same time execute an intensive 3-month effort to contact a parallel sample and ask the same questions. The first poll gets completed questions from about 25%, the intensive one from about 75% of the sample. They find that the telephone polls match the more thorough survey with few differences, but one difference was that Republicans were *more* likely to respond to the first poll -- by about 6%. If you know of
a study showing otherwise, I'd like to see it. This is the best study I know of thus far.
One such link was provided in post immediately following and directly relates to this election.
4) Paul D: As I pointed out (and see the USCountVotes report) the only pattern against party affiliation goes slightly in the *wrong* direction for the E/M explanation. There's no evidence otherwise that I know of.
One cannot draw this conclusion from the trend showing higher exit polling turnout in nominally Republican precincts. We need to know who responded and who balked -- cannot assume that they were representative of the precint unless under these conditions. That is a grave error.
Since the exit polls are voluntary, it is impossible to measure an unmeasurable. You need to capture a random sample -- not a self selected sample. This is always one of the greatest problems with these type of polls.
Posted by: Paul Deignan | Jan 31, 2005 4:16:23 PM
Posted by: Tom Perkins
While I have only scanned this document online:
http://uscountvotes.org/ucvAnalysis/US/USCountVotes_Re_Mitofsky-Edison.pdf
I find no data here about when a third wave of exit polling took place, nor any comparison about differences between inaccuracies of exit polls taken at different times of the day.
This document:
http://exit-poll.net/election-night/EvaluationJan192005.pdf
has such information, but it perforce cannot say whether my experience was coincidental or material. It can only state what the error was, and does not show any correlation between polling technology and error, other than to show that the electronically recorded ballots had a much higher error than the maligned paper ballots--but the error corresponded strongly to the size of locality wherein the precinct was located. Larger locales had larger errors, and these were larger across the board, not correlated to polling technology. If this pattern held as strongly across strongly blue and strongly red states, it militates against such a correlation being related to fraud--unless suspicious persons are willing to credit a VERY large conspiracy with members almost having to come from differing parties.
There is also no evidence the error was particularly large in especially close states, or is states of an electoral college size enablng them to decide the election, had they gone another way.
Weather had almost as large an effect on the WPE (within precinct error) as whether a state was a swing state--not something I would expect if fraud was at issue. You'd think if someone were to commit a felony of this magnitude, they's eleiminate weather as a variable.
Also, the average error within precinct for OH and for CA were the same(10.9), and SC (10) was very similar. In each case, a state is either uncontestedly Dem (CA) or Rep (SC) or a swing state. Why are there errors so similar? Why is this pattern repeated over many election cycles? Why is UT consistently a low error state, and can that performance be duplicated?
Again, this is a first impression after skimming the report.
Exit poll errors generally increased in the year 2004, and did so without regard to the advantage to any party in any state.
Something has changed and the work done in the 70's and 80's to make exit polls more accurate needs to be done again.
And not to take an off topic track without good reason, but would Dr. Velleman endorse both voter verified paper records and rigorous mechanisms to prevent people from voting illegally?
Does Dr. Velleman think my experience of a sharp rise people voting for Bush after 5pm (many of whom would not have exited a polling place until long after 7:30pm, owing to the lines) is coincidental? Or irrelevant?
Yours, TDP, ml, msl, & pfpp
Posted by: Tom Perkins | Jan 31, 2005 4:26:13 PM
Posted by: Paul Deignan
Paul V.
Note that the Pew poll was telephonic – not a face to face interaction. I will wager you that the discrepancies were especially bad in the precincts where a young pollster was stationed (I have not seen the data). Not that the pollster was necessarily inexperienced, but that the respondent identified the pollster with partisans Democrats. If Democrat -- a higher response rate, if Republican -- a higher balk rate.
So, perhaps if we knew the balk rate and the age/gender of the pollster we might make some use out of the nominal composition of the precincts. Of course, we would also need to normalize for time of day.
I'll also wager that we have insufficient data to say much here.
Posted by: Paul Deignan | Jan 31, 2005 4:28:45 PM
Posted by: Paul Deignan
So on that Pew poll -- the more face to face, the more likely the Republican to balk.
This is the type of results that support my hypothesis of a social interaction in balking by political affiliation.
Posted by: Paul Deignan | Jan 31, 2005 4:45:24 PM
Posted by: Paul Deignan
Here is that link again: http://www.wm.edu/news/?id=4027
It also lends support to the social interaction theory (not otherwise). Right now, we have a pattern.
Posted by: Paul Deignan | Jan 31, 2005 4:47:49 PM
Posted by: oliver
Actually, in theory the biases could be due to chance, although as Prof. V says that's really really unlikely. What they can't be the result of is a fluke. "Fluke" suggests the thing defies analysis. But for every event bigger than the quantum mechanical movement of electrons all there is to deal with is simple cause and effect. Statistics isn't the sharpest tool in the drawer, but analysis is far from hopeless.
Posted by: oliver | Jan 31, 2005 5:52:21 PM
Posted by: oliver
Well, I guess there are big non-linear systems like the atmosphere that give rise to mathematical chaos, which looks kinda flukey. I retract.
Posted by: oliver | Jan 31, 2005 5:59:37 PM
Posted by: oliver
We might be able to exclude chaos as a cause of these biases based on the short time interval over which they happen. Chaos I think would require that people's votes depended on each other, which of course is true, but your friends probably don't change your mind in the course of an hour on election day. Yeah, I think we can rule out chaos along with quantum mechanics.
Posted by: oliver | Jan 31, 2005 6:05:55 PM
Posted by: Paul Shields
My theory is that there is a critical mass of shy Republicans
Posted by: Paul Shields | Jan 31, 2005 6:08:31 PM
The comments to this entry are closed.