Polling in Afghanistan: An Antidote to Anecdote

Jan 14, 2010 1:46pm

A piece in the Huffington Post today quotes four doubters of our latest poll in Afghanistan. The concerns they raise are unwarranted, in some respects internally inconsistent and in others downright irresponsible (headline included). But the basic question is a fair one: Is it possible to conduct reliable research in these difficult conditions?

It is – and we have.

Our latest poll is our fifth in Afghanistan since 2005, conducted with the same high level of methodological standards as those that came before. Like our polling in Iraq, we see these surveys as an essential source of independent information on conditions on the ground, unfiltered by government or military authorities, or, frankly, by the perceptions of others with an ax to grind or a viewpoint to promote. We conduct these polls to obtain directly the views of Afghans themselves, and we believe our knowledge would be vastly poorer without them.

The criticisms can be boiled down to three broad themes: You couldn’t, or didn’t, go to the right places to conduct these interviews; where you did go, you didn’t interview the right people; and it doesn’t matter anyway, because they must have lied.

No meaningful evidence is cited for any of these, and none bears scrutiny.

Methodology aside, the underlying complaint seems to be that the individuals quoted simply don’t like the survey’s results. Most prominent is Matthew Hoh, a former State Department employee turned critic of U.S. efforts in Afghanistan, who says our findings on support for the central government don’t square with what he’s heard from Afghans he’s met.

With all respect to Hoh, anecdote is one thing, random sample survey research, quite another.

Field work for our Afghanistan surveys is managed by ACSOR, the Afghan Center for Socio-Economic and Opinion Research, a subsidiary of D3 Systems in Vienna, Va. ACSOR maintains staffs of trained interviewers in each of Afghanistan’s 34 provinces; they travel in separate male and female teams to randomly selected sampling points, from there follow a random route to select households and then randomly select a resident to interview.

The first criticism is that travel in much of Afghanistan is impossible given Taliban control of “large swaths of the country.” The reality is that ACSOR continuously updates district- and settlement-level assessments of where its interviewers can and cannot travel safely. As noted in our methodological report, 11 of the 194 originally sampled districts were replaced for security reasons. One critic calls 90 percent coverage “simply implausible.” It is so nonetheless.

This critic, Prakhar Sharma, is identified as having done “a large amount of public-opinion research work in Afghanistan” (without details, or reference to any such output). He says he’s “worked directly” with ACSOR and found “data falsified for insecure provinces,” a serious accusation.

ACSOR tells us it has no record of ever working with Sharma. Moreover, as we’ve reported, 31 percent of the interviews in our survey were directly monitored or back-checked by supervisors, a high level of field supervision; and all data undergo strict quality control review.

Another serious accusation, again with no explanation or evidence, comes from Anand Gopal, a Wall Street Journal reporter, who’s quoted as saying interviewers “can’t very well go door to door” and therefore “usually find participants by polling their friends and family.” He is clearly unfamiliar with the random selection procedures employed in ACSOR polls.

Hoh, for his part, says, “If you look at it, the polling was conducted in built-up areas, in urban areas … primarily off the major highways.” The fourth critic, Christian Parenti, a reporter for The Nation, is paraphrased as saying “he doesn’t believe the researchers got to the relevant areas.” In fact 80 percent of our interviews were conducted in rural areas. Our sample was randomly selected proportionately to the distribution of the country’s population. Check our photos from the field – not exactly what you’d call urban settings.

There are other unsupported complaints. Sharma says he’s never been able to get close to a 50-50 split in male and female interviews; therefore, neither could we. Interviewing women in Afghanistan requires special efforts, and as we’ve reported, there are three sparsely populated provinces in which we were unable to accomplish it. But by separately selecting sampling points for female-only interviews, and sending out female-only interviewer teams (escorted by men as needed), ACSOR interviewed 730 women, as well as 804 men, for our latest survey.

Sharma also describes Kish grid selection – a respondent selection procedure in which household residents are listed by age and sex – as “simply not feasible.” He offers no explanation. ACSOR successfully uses Kish grid selection.

ACSOR, I’d add, is the first registered survey research firm in Afghanistan. It’s completed more than 200,000 face-to-face interviews for hundreds of surveys there since 2003. We’ve spent many hours vetting its methodology, attending interviewer training, reviewing sampling plans and parsing the data. Afghanistan is a difficult place to carry out this kind of research. ACSOR does so as well as is humanly possible – with no little risk to the interviewers involved.

It’s interesting that the critics who doubt the methodology of our new poll voiced no such complaints about, say, our survey a year ago, with its bleaker outlook. Hoh, as noted, reveals his basic complaint to be that he doesn’t want to believe the results. Parenti is paraphrased as saying the level of support we found for the central government “doesn’t ring realistic to him.” Sharma says answers “are calculated to protect and benefit the respondent’s family and village.” Gopal says Afghans “tell the surveyor what he wants to hear,” adding, “there’s a heavy pro-government and pro-coalition bias in the surveys.”

These assertions are hard to square with our actual results. Fifty-nine percent of Afghans rate the performance of the United States in Afghanistan negatively, nearly twice as many as did so in 2005. Sixty-two percent rate the performance of NATO forces negatively. Thirty-six percent directly blame Western forces for civilian casualties; 61 percent blame either Western forces mainly or Western and anti-government forces equally. Favorability ratings of the United States, a tepid 51 percent overall, are down from 83 percent in 2005, and drop to 35 percent in the East and 29 percent in the South. This is pro-coalition bias?

In other results, anywhere from 83 to 95 percent of Afghans call official corruption a problem at the local, provincial and national levels. That, too, hardly seems a biased pro-government result.

Other results are better for the United States and NATO, and we find a rally behind Hamid Karzai as he’s asserted power after the disputed election. Their critics may not like it, but there are plenty of logical reasons for our results, all described in our full analysis. (Consider, for example, the external validity provided by the latest U.N. report of a drop in civilian casualties attributable to U.S. and NATO forces – a finding echoed in our data.)

Parenti, further, is paraphrased as doubting that Afghans “all of a sudden” support the central government. In fact that support, 61 percent, has not come out of the blue, but simply has roughly regained its 2006 and 2007 levels, 64 and 59 percent, respectively. It remains below its level in 2005.

On Sharma’s suggestion that respondents provide the answers interviewers want, for this to be so the interviewers have to telegraph a preferred response. Avoiding bias can be a challenge for Westerners in Afghanistan. Our local Afghan interviewers are trained to conduct their research neutrally. And they’re not told the identity of the sponsor of the survey they’re carrying out.

If respondents colored their responses to benefit their village, presumably they’d have tried to draw greater assistance by saying things are in terrible shape and getting worse. In fact we see better ratings of local conditions. Larger numbers of Afghans report roads and clinics built or repaired and electricity supplied. Forty percent give a positive rating to the availability of jobs and economic opportunities – still far from ideal, but up from 26 percent in 2007.

Better development is one of several factors that have improved the outlook of ordinary Afghans for their own and their country’s future. While there’s been progress in some areas, there are significant challenges ahead. We can understand them best not through the representations of officialdom, nor through punditry, conventional wisdom and anecdote, but through independent, reliable and direct measures of the Afghan public’s attitudes and experiences. That’s why we poll there.

You are using an outdated version of Internet Explorer. Please click here to upgrade your browser in order to comment.
blog comments powered by Disqus