Understanding Answers

Pollsters spend a lot of time thinking about what to ask people. It’s at least as important for us to think about how to understand their answers.

Here’s a suggestion in that direction: In literalism, at least in some data analysis, there’s considerably more vice than virtue.

Some of this notion’s been prompted by our latest results on global warming, but some comes, as well, from evidence across a range of other measures. In all it suggests that while people tell us exactly the message they intend to send, we’re not always tuned in to hearing it. Some, sometimes, are talking about the forest, while we’re looking for trees.

Last week we found an 8-point drop in belief global warming’s occurring, from 80 percent in July 2008 to 72 percent now. Other results have found larger declines; down 14 points, for instance, in a Pew poll in October, leading a pair of writers at Yale to propose “apocalypse fatigue” – the notion that “the louder and more alarmed climate advocates become… the more they polarize the issue, driving away a conservative or moderate for every liberal they recruit to the cause.”

Not a bad theory, perhaps, but a pinched one all the same. It tells us nothing about a range of other results that’ll strike those who differ as inexplicable: The extent of belief that Barack Obama wasn’t born in the United States, for instance, or that Saddam Hussein backed al Qaeda and harbored a stack of never-found frightful weapons, or that men never really landed on the moon.

I don’t mean in any way to equate these views, or a list of others I could add, but rather to place them under a broader concept: that people don’t always answer our questions within the narrow confines of what we’re asking, but rather in the broader context of the answer they want to give.

Imagine, for instance, that some of those who really don’t like President Obama will take just about any opportunity to offer a negative opinion of him – not to sign on literally to the specifics of the question, but rather not to miss the chance to express their antipathy toward the man.

Ditto with Saddam. Ditto, in the other direction, with Obama’s most enthusiastic boosters, some of whom will go for the positive, virtually whatever positive it may be. Suppose that some of those who express doubt about the moon landing, for their part, are taking the opportunity to voice their skepticism about the veracity of government more broadly – not, specifically, to propose a desert-based hoax, complete with Klieg lights and a tinfoil LEM.

We know that answers, to varying degrees, are informed by predispositions. Some people who are predisposed to like or dislike Obama will use that predisposition in obtaining and evaluating information about him, and likewise in answering questions about him. When his – or any president’s – overall approval rating goes down, it tends to go down across a range of issues – the economy, the deficit, health care, foreign affairs and more. Ratings of the vice president and of Cabinet officials also tend to move in tandem with the president’s. There is differentiation across these issues and individuals. But there’s also an element of simply thumbing up or down – expressing a broader assessment, not solely the narrow, specific one we think we’re measuring.

We see this, as well, in views of the economy. Consumer sentiment often improves among Democrats and worsens among Republicans when there’s a Democratic administration in the White House, and does the opposite when the president’s a Republican. Some people do make independent, strictly non-partisan economic assessments. But others – particularly when ambiguity in the economy’s condition opens the door to other judgments – use their partisan predispositions, at least to some extent, to inform those views.

If ambiguity surrounding a particular issue opens the door wider to political, ideological and other predispositions, so do the questions we ask – particularly when we confuse opinions with something firmer, like beliefs, or knowledge. Most of the people who “believed” Saddam supported al Qaeda also said, on probing, that this was their suspicion only, not that there’d been hard evidence of it. Among those who “believe” in UFOs, half, on probing, say they’re unexplained natural phenomena, not saucers flown around by little green men. Most people who believe in heaven also say, if you bother to ask, that it’s incorporeal. And so on in a thousand iterations. Attitudes are layered – probe, and you learn more.

Back to global warming. It’s a clue your measurement is less straightforward than you think when different questions, aiming at the same target, get different results. We found 72 percent saying global warming “is probably happening.” Pew found a lot fewer, 57 percent, saying there’s been “solid evidence” of it. That suggests a perception of some ambiguity or uncertainty in the science – enough to allow for predispositions to get a nose under the tent.

And voila: In our data, the decline during the last 18 months in the expressed view that global warming is probably happening occurred solely among conservatives, Republicans and Republican-leaning independents. All are groups that disproportionately oppose proposed policy changes to address the issue.

Indeed another fascinating result in our data is the fact that the change since summer 2008 took place exclusively – again, exclusively – among people who oppose the policy changes we tested, either cap and trade or other undefined, unilateral U.S. actions.

This raises the possibility that some people who oppose policies to deal with climate change – and who perhaps see them as more likely under the Obama administration – are taking the opportunity to express their discomfiture by saying they doubt global warming is occurring in the first place. We tend to think that assessments of the problem influence expressed views on the solution; for some people, I’m suggesting, it may work exactly the other way around.

The concept’s worth exploring not just in this but in many of the issues we measure. Our measurements are good ones, I firmly believe; even with the filtering of predispositions, there’s rich evidence that survey respondents honestly express their views. Our challenge – simple to say, but sometimes more complex to achieve – is to understand them.