MOE and Mojo

Dec 3, 2007 10:58am

The Des Moines Register’s new poll, released Sunday, has Barack Obama 3 points ahead of Hillary Clinton in Iowa, which it characterizes as a “lead” for Obama. Our own ABC/Post poll two weeks ago had Obama 4 points ahead – and we called it “close,” not a lead.

What gives?

The answer is that it’s all about how far you’re willing to push the envelope. To the Register, we’re probably being too conservative. To us, the Register is going overboard. But there’ll be more of this to come in the weeks ahead, so it’s worth understanding how we get here.

A poll is not laser surgery; it’s an estimate. The reliability of the estimate is (in part) a function of its sample size. This is expressed as the margin of sampling error, and it’s customarily given at the 95 percent confidence level. For a poll of 500 likely voters, which was the sample size in both the ABC/Post and the DMR polls, that’s plus or minus 4.5 points. This means a candidate would need a lead of 9 points or more for us to say with 95 percent confidence that it’s statistically significant. Neither poll comes close.

However, the estimate a poll gets is in fact the likeliest true value, and the likelihood decreases as we move toward the extreme ends of sampling error. In fact we can calculate the level of confidence we can have that Obama really leads in either of these polls.

The answer: With Obama +4 vs. Clinton in the ABC/Post poll, we could have said with 77 percent confidence that – all else equal – he really had a lead. In the DMR poll, with Obama +3, the confidence level is 64 percent. Apparently 64 percent confidence is good enough for the Register to call it a “lead.” It’s not for us; nor was the 77 percent probability in our last poll.

Why not? One reason is that the customary confidence level in survey research is 95 percent, not 77, or 64. Another is that these probabilities only hold if all else is equal – and it isn’t. These estimates also are subject to non-sampling error, the likeliest cause of which is their estimate of who in fact qualifies as a likely voter. In our tighter likely voter model, more closely approximating caucus turnout in 2004, we didn’t have Obama +4, we had him +2, with 28 percent to Clinton’s 26 percent. The probability of that being a statistically significant lead was only 37 percent. Thus the prudent course was to call it close, which is what we did, and what we think it is.

Now on the Republican side, with 400 interviews, we had Mitt Romney with 28 percent support, Mike Huckabee with 24 percent. Huckabee had all the mojo – he was the guy making the move, as our analysis two weeks ago made clear. The probabilities still had us call the race close.

That Huckabee mojo looks to be continuing; the Register now has him at 29 percent support, to Romney’s 24 percent. The Register, again, calls Huckabee the leader. The confidence level 88 percent. Is that enough to call it a “lead”? It’s tempting. But before going there we’d want to see what turnout their likely voter model anticipates and what their other models (if any) show. Meanwhile, in our book, this one, too, is close.

As you can see, there's a bit of judgment in all this. The shorthand approach used by the AP is to say that when a candidate's numerical advantage doesn’t exceed sampling error, but is at least half of what sampling error demands, it can be called a "slight" lead. That's of course not what it is; it's really a possible lead in which we cannot be wholly confident.

All this underscores one of the fundamental points about pre-election polls: They are estimates. Even with good-quality methodology, the notion of pinpoint accuracy is a myth. And the reason we do them is not simply to try to puzzle out who's ahead – but to understand how and why the voters are coming to their choices.

You are using an outdated version of Internet Explorer. Please click here to upgrade your browser in order to comment.
blog comments powered by Disqus