EmailThe American Association for Public Opinion Research today released an exhaustive report on the 2008 presidential primary polls, focusing to a large degree on the pre-primary polling meltdown in the New Hampshire Democratic primary.
AAPOR’s news release says its investigators have isolated four possible factors in the error in the New Hampshire polls – ending too early, failing to chase down hard-to-reach respondents, under-representing some groups (possibly because they were hard to reach) and variations in likely voter models.
Other possible factors were discounted, including the so-called (and now generally discredited) Bradley effect, the exclusion of cell-phone-only respondents from some samples, the practice of asking recalcitrant respondents which candidate they leaned toward and a late decision by independent voters to cast their ballots in the GOP primary.
Some fit with the theories and limited evidence we’ve seen before – see my blog items here and here exploring some of these issues. The first, lack of timely polling, surprises me, given both the few final-night polls (which also got it wrong) and the exit poll results, which did not support the notion of a swing among late deciders. But not to prejudge: I’m crashing on other work today and won’t have time to review the full report for several hours. You can see it here, and I’ll post further comments as warranted after review.
One item I would note is that AAPOR’s announcement indicates that seven polling organizations released their full pre-primary datasets for the committee’s review – but many others did not. In its foreword, the report says, “The fact that many pollsters did not provide us with detailed methodological information about their work on a timely basis is one reason we will never know for certain exactly what caused the problems in the primary polling that we studied." That’s a troubling lack of disclosure on the part of the organizations in question.
The report – 123 pages long – looks to be an impressive piece of work, just the sort of thing that, immediately after the New Hampshire fiasco, I suggested was essential. The volunteer AAPOR committee members who produced it, led by Prof. Michael Traugott of the University of Michigan, deserve our great thanks.