Polling on Presidential Pejoratives
EmailWhatever profoundly negative things people might think about Barack Obama, a new poll out today demonstrates splendidly how not to measure them.
It nails the negativity, all right; this project purports to tote up responses to a list of harsh criticisms of the president – e.g., that he’s “anti-American,” “a racist,” “wants… an excuse to take dictatorial powers,” “is doing many of the things that Hitler did” and “may be the Antichrist.”
Hot words, those. The survey, done by Harris Interactive, apparently was designed to test the theories in a book claiming the “lunatic fringe is hijacking America.” The purpose seems to have been to see how many people the pollsters could get to agree to pejorative statements about Obama. Quite a few, it turns out – but with what I see as a highly manipulative approach to questionnaire design.
I’ll lay off the sampling, though this survey was done among people who sign up to click through questionnaires via the Internet in exchange for points redeemable for cash and gifts – not a probability sample. Been there before. This time let’s just look at what it asked.
The poll starts by telling respondents “here are some things people have said about President Obama,” then asking if they think each is true or false. Fifteen statements follow, with all (excluding “he is a Muslim”) unrelentingly negative. “True” answers run from a high of 40 percent, for “he is a socialist,” to a low of 13 percent, for “he wants the terrorists to win.”
The problems are fundamental. “Some people have said” is a biasing introductory phrase; it imbues the subsequent statements with an air of credibility – particularly when you don’t note that others say something else. (That approach can have problems of its own; the “some people” vs. “other people” format implies equivalence.)
The subsequent statements, for their part, are classically unbalanced – there’s no alternative proposition to consider. A wealth of academic literature, neatly summarized here, demonstrates that questions constructed in this fashion – true/false, agree/disagree – carry a heavy dose of what’s known as acquiescence bias. They overstate agreement with whatever’s been posited, often by a very substantial margin. (This reflects avoidance of cognitive burden, which tends to happen disproportionately with less-educated respondents, as is reflected in Harris’ results.)
Using all negative statements, rather than a mix of negative and positive ones, reflects another non-standard approach, one that can further bias responses. (The ordering of items, unclear in the Harris release, can be troublesome as well.)
Another problem, which I discuss here, is the challenge of over-literalism in evaluating survey results of this type. Rather than answering disparaging poll questions literally, people who are ill-disposed toward the subject may simply use these questions as an opportunity to express their general antipathy – not as a thought-out endorsement of the specific posit. And the use of hot-button invective is ill-advised in its own right; respondents may just blow it back.
Admittedly it’s a challenge to measure these sorts of sentiments. Unless carefully crafted, with balance and an approach that encourages due consideration and probes for meaning, simply asking the question can turn into little more than the old reporter’s trick of piping quotes. It’s a shopworn use of true/false and agree/disagree questions, one long overdue for retirement.
Harris indeed goes the next step by reporting its results as what its respondents’ “believe” and as opinions they “hold,” as if they themselves came up with these notions, rather than having them one-sidedly set before them on a platter. Call me what you will – and I know it can get nasty out there – but from my perspective, this is not good polling practice.
Next-day update: Nice supporting material for this post comes from Michael Scherer at Time.com’s Swampland, who notes that Pew Research Center polls in October 2008 and March 2009 alike found 17 percent of Republicans saying they think Obama is a Muslim, vs. 57 percent in Harris Interactive’s approach. Pew’s question was as follows: "Do you happen to know what Barack Obama’s religion is? Is he Christian, Jewish, Muslim, Buddhist, Hindu, atheist, agnostic or something else?" (Even here, note my comments on literalism vs. generalized antipathy, and my previous posts on the problem of measuring "knowledge.")
Sen. DeMint: GOP Race Could Go Until Convention 
Obama Avoids Questions on Contraception Rule
Very good points. That said, could ABC News do a more scientific poll PROPERLY testing for extreme views on both the left and right? There clearly are a lot of people with kooky views out there, and that’s important to quantify for the sake of understanding political reality in this era. But as you said, Harris did a poor job at measuring this.
Posted by: vvf2 | March 24, 2010, 9:21 am 9:21 am
This not a poll. It is fishing for predetermined responces. What is the reason for doing this and how could credible media report it has meaningful? Let’s poll on that. Good article.
Posted by: Ray | March 24, 2010, 9:47 am 9:47 am
This is all true enough. But I don’t recall seeing this analysis of the health care reform polling that shared these problems.
Posted by: jayackiroyd | March 24, 2010, 9:51 am 9:51 am
The problem with the theory is that the percentages should be about zero for blatant BS…shouldn’t it?
If the methodology has the potential to influence responses on such yes or no questions perhaps it speaks more to how easily influenced the respondents are to these propagated falsehoods than anything else.
You know like when 70% of the people thought Saddam was responsible for 911. “Some people” said it. The problem is those “some people” included VP Cheney implying it if not strongly suggesting it as we headed into a war based on the marketed mis-perception “somehow” incorrectly ingrained in a majority of the u.s. people.
So in the responses I have seen strong confirmation of notions from “some people” at Palin rallies, Palin herself, the GOP party geneerally (in “some people” but not most), and the tea party signage of “some people” in spades.
Fundamentally I have less problem with the poll than with the people who are happy to allow such misinformation to be propagated to the vilest elements representing the base of a particular political party.
What I see as a real shame is that I am an AA male that aligns in many respects with most of the moderate to conservative fundamentals of the GOP. However the refusal to purge elements from the party that lead to these ridiculous (or sad) poll results keeps me far far away from such self loathing as to actually support a party that panders to …
you poll it.
Posted by: RIckC | March 24, 2010, 10:00 am 10:00 am
Look, I don’t care HOW biased a poll might be, I’d NEVER check “yes” beside any of those insane characterizations. This is what, a big excuse for having some massive population who is so ignorant and worthless that they WOULD check “yes” next to completely batshit suggestions? To me, THAT is the whole problem illustrated by this poll!!
“Some people have said the earth is flat…” Sorry, what loser checks “I agree” next to THAT? I don’t care how “badly worded” the poll is, only a completely ignorant, uneducated, deluded person would ever click “yes”.
Posted by: ShannonA | March 24, 2010, 10:09 am 10:09 am
In truth, although the Harris Poll is indeed not a great poll, I have heard practically zero Republicans not act exactly like the poll states. I can think of two I know who are not complete unhinged whack jobs and are able to discuss politics somewhat civilly. Even my own brother, who’s a Fox News fanatic, is so insane that none of the rest of the family will talk to him or listen to him on any topic even verging on the political. He spouts Fox News talking points and hurls epithets at the President, members of Congress (Dems only, of course), cabinet members, and all of us who are Liberal and Democratic, including our 93 year old mother! Unhinged may have described a certain sector of the left during Bush’s maddening Presidency. But it certainly describes pretty much the entire Republican Party since January 20, 2009.
Posted by: Susan L. | March 24, 2010, 10:15 am 10:15 am
Bias in how the questions are phrased simply doesn’t matter in this case. If a survey that asks, “Some people have said that 1 = 2. Do you agree or disagree with this statement?”, you would be a moron for checking anything but “disagree”. These questions are not matters of opinion like most polls; most of them are matters of well-established fact.
Posted by: Tacroy | March 24, 2010, 10:33 am 10:33 am
We’ve never had major media outlets (Fox News and sister radio and print) that have deliberately worked to misinform their followers. A say followers because they provide a filtered false reality a segment if society prefers over the actual news. Imagine a mental hospital that worked to re-enforce the patients delusions. I’m sure the patients would never want to leave. It would do very well financially. But the government would shut them down.
Posted by: Kevin | March 24, 2010, 10:51 am 10:51 am
Polls mean very little. Most are poorly designed, many have ulterior motives, and few do good sampling of the public.
There is one thing for certain, however, and that is that Obama is heading for that all-too-familiar territory, where more than half the citizenry finds him in an unfavorable light, and moving very consistently towards the low popularity levels of GW Bush.
The difference between Bush and Obama, will be that history will be kinder to Bush, than it will be to Obama.
Posted by: Rick McDaniel | March 24, 2010, 11:55 am 11:55 am
What poll would that be CND Fox? I have never heard those questions asked in any poll.
I’ve pointed out horrible provisions in various versions of the health care bill (like jail for non-compliance) to liberals who call me a liar and are too lazy to read the bill.
Where do y’all keep the kool-aid at? Talk about an alternate universe.
Posted by: Kate | March 24, 2010, 1:08 pm 1:08 pm
One doesn’t need a poll to verify these results – all we need to do is look at the hate-filled signs that the chanting fanatics waved during the health-care “debate”.
Posted by: Al in SoCal | March 24, 2010, 1:10 pm 1:10 pm
Mr. Langer, thanks for the post, but could you go a useful step further and direct your readers to one or more polls with methodologies that you DO approve of (follow good polling practice) that tell us the extent of negative opinions, beliefs, etc. toward the President today (preferably including a range of negative opinions)?
Posted by: David | March 24, 2010, 1:31 pm 1:31 pm
Polls asked if you wanted “this” health care bill to pass. Right wingers said no to keep in step with what they’ve been told to do. Left wingers said no because they wanted it to do more. Since most Republicans are extreme right wingers and a small number of Democrats are extreme left wingers, you got the poll saying that 60% didn’t want this health care bill. Duh!
Posted by: sacyli3 | March 24, 2010, 1:37 pm 1:37 pm
Gary, why do you count “he is a socialist” as “unrelentingly negative”? There are, in fact, a decent number of us out here, and our numbers could be worse, all things considered.
Posted by: JoeC | March 24, 2010, 3:00 pm 3:00 pm
One can imagine a similar poll done during the Bush administration, when anti-war protesters routinely (and unprovoked) carried home-made signs equating Bush to Hitler — not because some pollster suggested “some people” think so, but because their fellow idealogues said and believed it. Go to tea party anti-tax protests and find me a comparable number of signs saying Obama is the antichrist. Heck, find me one. Good luck!
Posted by: Kelly J | March 24, 2010, 3:15 pm 3:15 pm
These pollsters got verballed.
I have deliberately put in wacked responses on obviously silly polls before.
Either this is yet another attempt to portray all who disagree with the Obama nation as racist extremists, or it is just another useless poll by less-than-professinoal poll designers.
It could also easily be a combination of both of the above.
Posted by: hunter | March 24, 2010, 3:49 pm 3:49 pm
Kelly J,
As horrible as the unilateral military invasion of Iraq was, with its consequent tens of thousands of deaths among civilians and wrecking of national infrastructure, etc., I agree with you that equating Bush with Hitler went too far. Still, there’s something very different between white President Bush who aligned his government with one religion, overlaying religious polarization into the political scene, and a black President Obama, that adds a unique meaning to “antichrist” (or “anti-Christ” if you see the other blog comment at
in today’s discourse, few though the handwritten signs may be. The lock-step Republican antagonism to many Obama-led initiatives (or Democratic initiatives) in Congress and in general is also quite different from the support that Bush received from Democrats. But in general, I agree with what I think is the spirit of your posting, that there should be more acceptance, more dialogue, more building on common concerns, and more positive building by bipartisan effort, and less attention and reaction to inflammatory extremism.
Posted by: David Boyle | March 24, 2010, 3:50 pm 3:50 pm
This is a highly unbalanced critique of interactive polling–completely lacking in professional objectivity. I’m not ready to throw all the way in on interactive polling, but there is accumulating evidence that the Harris methodology is producing good validity and reliability numbers.
Posted by: Martin | March 24, 2010, 3:56 pm 3:56 pm
Kate…in response to your 1:08 post…it was a Harris Poll and the article it was in was from John Avalon, a moderate, who runs the moderate website The Daily Beast. John has also written a good book on the subject: “Wingnuts: How the Lunatic Fringe is Hijacking America”. See, if you consulted something besides Faux News….
Posted by: CND FOX | March 24, 2010, 5:13 pm 5:13 pm
Can anyone explain to me where my original post went? What is it with you ABC? You really do not want honest dialogue do you?
Posted by: CND FOX | March 24, 2010, 5:21 pm 5:21 pm
No need to paper over this, Gary – we LIKE who we are. Obama IS a socialist, and possibly the anti-christ, and the Harris poll proves that most of America agrees with us!
Posted by: Erick Erickson | March 24, 2010, 6:53 pm 6:53 pm
I agree that this poll is ridiculous, but I wonder why Mr. Langer draws attention to it, while neglecting to point out similar glaring problems with much-more-widely publicized polls.
Yesterday, USAToday/Gallup released a statement on the health care bill stating that “48% calls it ‘a good first step.’” Well:
First, nobody “called it” anything; this implies volunteered responses even more than the language in the above poll.
Second, people were given only four widely divergent choices to select from; consequently we have no way of knowing how precisely this comment reflects their views;
Third, “a good first step” was not actually a choice; it is only a short phrase pulled out of the much-longer response which was actually offered for respondents to select.
Where’s Mr. Langer on this?
Posted by: W in NJ | March 24, 2010, 6:54 pm 6:54 pm
Kelly J.-Pres. Obama hasn’t turned the White house into a mosque, Tea Party is run by the KKK and you’re about as uninformed as Sarah Palin. Her dumb ass wants 2 run 4 Pres. 4 them.
Once the public sees how much BS the GOP is pushing to get more seats in Nov. and that its revealing their ignorance, not much is changing except Obama’s approval rating.
States w/R-Gov.’s(i.e. Romney-MA before ’07) created this same plan w/out cures for SSN. They force people to get Health Ins. They’re suing now 2 stick together w/the GOP in DC or just 2 get stimulus money!!!! That’ll kill their cases, prob., w/ the USSC!!!
Posted by: Orus | March 24, 2010, 7:59 pm 7:59 pm
As I was saying about the GOP-The Ins. Comp.’s were doing the serious lobbying 2 keep theirselves involved and make it like a state type plan. And just like the GOP are playing after they lost, US citizens are playing ignorant & stupid!!!!!
So many states have this setup!!! All states with cars and a decent amt. w/ both cars & health. This President pretty much fixed SSN for the nation.
Blame that socialism mindset on the ignorant, stupid GOP & lobbying of the HMO’s!!! That’s the part of the state plans. Romney’s dumb self is in the stupid state of CA at a book-signing basically knocking a plan he got going 4 MA. The GOP and a lot of others are mad that a Right, White(McCain)man isn’t in there signing a bill they created.
Posted by: Orus | March 24, 2010, 8:16 pm 8:16 pm
I’m willing to stipulate that the Harris poll was flawed, but this is the second major poll (the other by Survey America) that found that some rather extreme views of Obama were widely held among Republicans.
Maybe that poll was methodologically flawed, too, but it does seem like something ugly is going on here.
Instead of criticizing the polls of others, maybe Mr. Langer and ABC could do their own survey on this subject — using whatever safeguards they think are appropriate?
Given that large chunks of the Republican Party appear to have been driven completely around the bend by health care reform, this would seem like a newsworthy project.
Posted by: Peter Principle | March 24, 2010, 9:48 pm 9:48 pm
Good Blog Mr. Langer.
Rasmussen published a similar poll stating that a third of Democrats believed George Bush knew something of the 9/11 attack i.e. “truthers”.
I’ll keep this poll in mind the next time I hear Harris referenced.
Posted by: Robert McNutt | March 24, 2010, 10:19 pm 10:19 pm
The headline should have read “Poll For What Obama Has Made Worse”
Posted by: Reality | March 25, 2010, 11:01 am 11:01 am
However you want to look at it, the Harris poll shows, if nothing else, the power of the right wing trigger words. When Beck or Limbaugh refers to Obama in these terms they trigger exactly the responses you complain about, Mr. Langer.
So, as well as bashing the Harris poll, start a crusade to kill the right wing trigger word bashing of Obama.
OTOH, thanks for pointing that out, now we can use it against the dittoheads and such.
Posted by: bobfromdistrict9 | March 30, 2010, 8:45 pm 8:45 pm
OBAMA QUOTE:
“…my administration will make all information available to the American people so that they can make informed judgments and hold us accountable”
PLEASE ACCOUNT FOR YOUR BIRTH CERTIFICATE SIR. The time has come for the President to perform his duty to THIS country and show us his birth certificate. No certificate? Then all of these constitutional crimes go away; the czars, healthcare, all of it. There are too many questions, too many doubts and he has been dodging far too long. Where there is smoke there is fire. The more people that begin asking for the proof, the less he and the major broadcast networks will be able to ignore us as they have done with so many other issues. If there is the slightest doubt in your mind then you too need to be asking. We have let the seemingly small things go and look where that has taken us. I want to talk about it because they are in cover-up mode. It would be such an easy thing for any one of us to show our birth certificate. Why is it so difficult for him? Why does he treat this country as if he is ashamed of it? Why does he treat our allies as if they are enemies? Why does he encourage Congress to ignore the will of the people? Why did his wife refer to his home country as Kenya? Why will he not release his own school records? Why does he disgracefully bow to the leaders of other countries? Why does he admire dictators from other countries? THIS IS OBAMA’S WATERGATE. There is no need to impeach someone who is not qualified to be president. Everything he has signed, ordered, or acted upon is illegal. So I ask you Obama: WHERE IS YOUR BIRTH CERTIFICATE?
Posted by: DontTreadOnMe | April 21, 2010, 8:25 pm 8:25 pm