It’ll be tempting to assess the outcome of the presidential election in Afghanistan on the basis of pre-election polls there. That is not a good idea.
Final estimates from good-quality pre-election polls conducted very close to election day typically do predict election outcomes accurately. But the two pre-election polls we saw out of Afghanistan appear so problematic – both in terms of what we know about them, and what we don’t know – as to take them off the table as reliable estimates.
Sadly, nonetheless, both were widely reported by media outlets here and abroad.
One of these polls was released Aug. 9 by an outfit called Glevum Associates, which appears from its website to be a military contractor engaged in producing psychological operations data as part of a U.S. Army counterinsurgency program, the Human Terrain System. The New York Times identified its poll as “financed by the United States government,” with no details; AP and Reuters did similarly.
Glevum did not return our repeated calls and e-mails for details, both methodological and in terms of the survey’s sponsorship, at the time of its release. Its methodological statement sounds satisfactory, but leaves some questions unanswered. Its poll was done July 8-17, more than a month before the election. Its full questionnaire, essential in discerning bias, was not released (rather, we got a PowerPoint summary).
It appears to have reported vote preferences among all Afghans who were registered to vote (79 percent of the population in its data), then to have taken steps to “factor out” the 20 percent who were undecided or declined to answer – apparently by simply percentaging them out of the base, a procedure based on the dicey assumption that these respondents ultimately would divide along the same lines as those who did express an opinion. A better approach, not taken here, is to ask undecideds which way they lean.
Glevum also looked at those who said they were certain to vote (74 percent of registereds, and therefore 58 percent of all Afghans), with essentially identical results. Its estimates were 36 or 37 percent support for Hamid Karzai among all registered or likely voters with the undecideds/refusals included, and 45 percent for Karzai among registered or likely voters with the undecided/refusals factored out. All other candidates trailed.
The other pre-election poll was released Aug. 14 by the International Republican Institute, described in the Times’ pickup of the poll as “a nonprofit pro-democracy group affiliated with the Republican Party and financed by the American government.” The IRI puts out a fair amount of polling, and has steadily refused our requests over the years for greater details about its methodology, questionnaires and results. (It has a new p.r. person, though, who today did promise to pass along our questions.) Its interviews were conducted July 16-26; the election was Aug. 20.
IRI’s release does not indicate that it asked respondents whether or not they were registered to vote – seemingly an important criterion – yet goes on to identify a “likely voter” population. This, however, seems to comprise 90 percent of the total population; actual turnout appears to have been vastly lower (current estimates say 40 to 50 percent). IRI did a much better job getting answers – 3 percent undecided or refused, vs. Glevum’s 20 percent. It had Karzai at 44 percent support (up 13 points from May). Items of interest, beyond more detailed methodological disclosure, would include how that looked among a much more tightly defined likely voter group.
Both polls produced estimates of support for Karzai, in early to mid-July, in the mid-40s. But they got there by such different means – one with 90 percent turnout, the other as low as 58 percent; one with 3 percent undecided/refused, the other 20 percent, allocated; and without adequate disclosure on other fronts – as to leave open the question of whether that’s confluence, or mere coincidence.
These polls, as noted, were broadly reported. We see references to the IRI poll by The Associated Press, the Los Angeles Times, The New York Times, USA Today, The Washington Times and the Guardian in the U.K.; and to the Glevum data by AP, The New York Times, Reuters, Toronto’s Globe and Mail and the Telegraph in the U.K.
Having been covered as meaningful pre-election estimates, these polls might now be used as part of the assessment of the election outcome itself. Doing so would require confidence in the estimates – something that should be based more than simply on a release of results, but on full disclosure of sponsorship, methodology, questionnaire and relevant data, with a willingness by the data providers to answer follow-up questions. That hasn't been our experience to date.