Whatever profoundly negative things people might think about Barack Obama, a new poll out today demonstrates splendidly how not to measure them.
It nails the negativity, all right; this project purports to tote up responses to a list of harsh criticisms of the president – e.g., that he’s “anti-American,” “a racist,” “wants… an excuse to take dictatorial powers,” “is doing many of the things that Hitler did” and “may be the Antichrist.”
Hot words, those. The survey, done by Harris Interactive, apparently was designed to test the theories in a book claiming the “lunatic fringe is hijacking America.” The purpose seems to have been to see how many people the pollsters could get to agree to pejorative statements about Obama. Quite a few, it turns out – but with what I see as a highly manipulative approach to questionnaire design.
I’ll lay off the sampling, though this survey was done among people who sign up to click through questionnaires via the Internet in exchange for points redeemable for cash and gifts – not a probability sample. Been there before. This time let’s just look at what it asked.
The poll starts by telling respondents “here are some things people have said about President Obama,” then asking if they think each is true or false. Fifteen statements follow, with all (excluding “he is a Muslim”) unrelentingly negative. “True” answers run from a high of 40 percent, for “he is a socialist,” to a low of 13 percent, for “he wants the terrorists to win.”
The problems are fundamental. “Some people have said” is a biasing introductory phrase; it imbues the subsequent statements with an air of credibility – particularly when you don’t note that others say something else. (That approach can have problems of its own; the “some people” vs. “other people” format implies equivalence.)
The subsequent statements, for their part, are classically unbalanced – there’s no alternative proposition to consider. A wealth of academic literature, neatly summarized here, demonstrates that questions constructed in this fashion – true/false, agree/disagree – carry a heavy dose of what’s known as acquiescence bias. They overstate agreement with whatever’s been posited, often by a very substantial margin. (This reflects avoidance of cognitive burden, which tends to happen disproportionately with less-educated respondents, as is reflected in Harris’ results.)
Using all negative statements, rather than a mix of negative and positive ones, reflects another non-standard approach, one that can further bias responses. (The ordering of items, unclear in the Harris release, can be troublesome as well.)
Another problem, which I discuss here, is the challenge of over-literalism in evaluating survey results of this type. Rather than answering disparaging poll questions literally, people who are ill-disposed toward the subject may simply use these questions as an opportunity to express their general antipathy – not as a thought-out endorsement of the specific posit. And the use of hot-button invective is ill-advised in its own right; respondents may just blow it back.
Admittedly it’s a challenge to measure these sorts of sentiments. Unless carefully crafted, with balance and an approach that encourages due consideration and probes for meaning, simply asking the question can turn into little more than the old reporter’s trick of piping quotes. It’s a shopworn use of true/false and agree/disagree questions, one long overdue for retirement.
Harris indeed goes the next step by reporting its results as what its respondents’ “believe” and as opinions they “hold,” as if they themselves came up with these notions, rather than having them one-sidedly set before them on a platter. Call me what you will – and I know it can get nasty out there – but from my perspective, this is not good polling practice.
Next-day update: Nice supporting material for this post comes from Michael Scherer at Time.com’s Swampland, who notes that Pew Research Center polls in October 2008 and March 2009 alike found 17 percent of Republicans saying they think Obama is a Muslim, vs. 57 percent in Harris Interactive’s approach. Pew’s question was as follows: "Do you happen to know what Barack Obama’s religion is? Is he Christian, Jewish, Muslim, Buddhist, Hindu, atheist, agnostic or something else?" (Even here, note my comments on literalism vs. generalized antipathy, and my previous posts on the problem of measuring "knowledge.")