December 31, 2004


Paul F. Velleman: December 31, 2004

The juxtaposition of news about the disaster in the Indian Ocean basin and the prices of the Presidential inauguration parties presents too great a clash for me to ignore. According to the NY Times, the Presidential Inaugural Committee expects the tab to top $40 million. Tickets are selling in the range of $2,500 per person.

I don't want to belabor this. The moral is obvious. It's not that we're the richest country in the world; somebody's go to be that. Nor even that our contribution to assist the victims, large though it may be, is far from the size it could be. But to spend millions on a gaudy party in the name of the President while so many are in such dire need seems to me to be the epitome of everything we should not stand for.

So maybe we shouldn't stand for it.

This isn't--or shouldn't be--a Left/Right issue. People of good will are horrified by the scale of this disaster, and don't think of it in terms of domestic politics. The scale is unprecedented in our lifetimes. Here is an opportunity to both do great good and to be seen as doing great good. And that's something the U.S. could use a bit more of these days. But, I suppose that any suggestion that the inaugural parties be canceled or scaled down and that the funds be sent where they could do real good will be seen as partisan.

That would be a shame.

So I'll open this for discussion:

  • Should we try to start a groundswell (dare I say tsunami?) of popular support for scaling down the parties and sending the money to those in dire need?
  • How can this be discussed and advocated free of partisan positions as something that active workers--those who really do get things done--from both Left and Right could accomplish together?
  • Indeed, can it be done?

January 31, 2005

explaining the exit polls?

Paul F. Velleman: January 31, 2005

In the 1970’s and 80’s I worked on election nights for a major network as part of a team of statisticians making “calls” in statewide races (President, Senator, and Governor). Eventually, the team was disbanded because exit polls were so accurate that our expertise was no longer needed.

But in the past election, the exit polls differed from the recorded vote by an unprecedented amount. Nationwide, exit polls predicted that Kerry had won by 3%, but the final tally showed Bush ahead by 2.5%. Errors in some key states were even larger. As a statistician, I have been concerned that the errors were unexplained. Last week, Edison Media Research and Mitofsky International, who conducted the exit polls on contract for the major networks and news services, released a report on the errors in their polls. Now an organization called US Count Votes has released an analysis of the E/M data that raises serious questions about E/M’s proposed explanations for the exit poll biases.

All who have considered the problem agree that there are three plausible explanations:

1.  Chance error,
2.  Bias in the exit polls, and
3.  Inaccurate election tallies, or (say it softly) election fraud.

Edison/Mitofsky and all who have examined the data agree that (1) is not plausible. The errors were so extraordinarily beyond what could occur simply by chance that we can safely exclude this possibility. Anyone who has taken a freshman Statistics course can do the calculations.

The E/M analysis arbitrarily ignores (3) and considers only possible biases. In one sense, this is understandable. Their job was to predict the final vote totals and they failed miserably. Some commentators were made to sound foolish when they started talking as if Kerry had won by about 7pm on election night. Clearly E/M’s clients didn’t get their money’s worth.

But as citizens, our job is somewhat different. If the bias in the exit polls can be explained by errors in E/M's methods or by other factors, it would reduce the concern for the vote itself. This analysis of the E/M data makes it clear that the E/M report fails to provide such an explanation. And, of course, if the exit polls were not themselves flawed, that would raise questions about the honesty of the vote itself.

I have looked at the E/M data and participated in the discussion as the US Count Votes document was refined. I have signed their report as a member of the team that reviewed the work. I am posting this commentary here to direct readers of the blog to that report. I think their points warrant serious consideration and that they clarify the need for further analyses of the voting and polling data.

The underlying scientific consideration is that any theory or model that claims to account for patterns in data must make sense no matter how we view the data and must be consistent with other external information. Specifically, if there are biases inherent in the E/M exit polls, we should expect them to follow their own consistent patterns. For example, it isn’t plausible to posit different biases in different places without offering any account of the differences. Such ad hoc explanations are not scientifically or statistically supportable.

Nor is it plausible that a bias present in some locations should inexplicably be absent in others.  Conversely, if there is a pattern to the biases that makes sense, the very existence of such a pattern makes the proposed explanation more plausible. For example, one early explanation of the exit poll errors was that (more Democratic) women tend to vote early, while (more Republican) men vote later. Such an explanation could be supported by finding trends in the biases during election day (E/M reported poll results in three waves). But no such pattern is found, and E/M do not propose that explanation.  They analyze data from the end of the polling day, and it is those data that show the biases they are trying to explain.

So what do they propose? E/M’s explanation is simply that Bush voters were substantially more likely than Kerry voters to decline to be interviewed. (Specifically that 56% of Bush voters but only 50% of Kerry voters declined.) E/M offer no evidence of this other than the obvious fact that the polls don’t match the recorded vote and the unstated fact that they can find no other explanation. But there are no underlying patterns to support their explanation. For example, E/M look for patterns in refusals and find none. In fact, as the US Count Votes report points out, the response rate of voters willing to be polled is actually a bit larger (although probably not significantly so) in states that voted more strongly for Bush—the opposite of the pattern that would support E/M's hypothesis.

Does the E/M hypothesis account for other patterns in the data? They propose no such explanations and admit to some patterns that their hypothesis fails to explain. For example, states that voted with paper ballots showed only small random errors between exit polls and votes, well within statistical error. States that used automated systems showed large errors fairly consistently biased toward Kerry.  It doesn’t seem plausible that voting method would influence a voter’s willingness to talk with pollsters (nor do E/M claim that this happened).

Let me be very clear. I do not assert there was extensive fraud. I would prefer not to think that, and I had hoped the E/M report would reveal a systematic flaw in their methods that accounted for the errors. But it hasn’t, and the issue is still open. The E/M report does not account for the biases in a manner that would support explanation (2). The exit polls may well have been flawed, but we have yet to see a plausible account of how or why.

The data released thus far beg for a more thorough analysis. E/M have not released precinct-level data, which would be necessary to determine whether voting technology is a factor. I hope that they will do so soon. I also hope that the news media report this story so that the public can be widely informed about it. I recognize that if significant problems with the reported vote are found, Republicans will feel that the effort was somehow directed against them. But honest voting is a value that should be supported equally by both Left and Right. We cite discrepancies between exit polls and votes in elections in other countries as evidence of problems. Especially when we have been called to spread liberty and democracy throughout the world, it behooves us to make our own democracy as open and honest as we can.

March 29, 2005

A self-referential poll?

Paul F. Velleman: March 29, 2005