If You Disagree, Press 2
EmailI spend a major chunk of my days locked in mortal combat with data that don't meet our standards for validity and reliability. One of today’s entries cuts pretty close to home – so close that a response may look like I’m carrying water for my employer. Stick with the evidence, though, and see what you think.
This item comes from an outfit called Public Policy Polling, which produces robopolls, in which calls are made by an automated dialer and a recorded voice asks people who go along to press buttons on the telephone keypad to indicate their views. (I’ve heard that some regard the phrase “robopoll” as pejorative. Sorry.)
In our ABC News polling standards we don’t regard autodialed, pre-recorded polls as valid and reliable survey research. Our concerns include noncoverage, nonresponse, lack of respondent selection, nonvalidation, opaque weights and the absence of a persuasive, independent literature evaluating the approach. Some other news shops – the few that, in my view, take coverage of public opinion seriously – share these judgments.
The PPP poll at hand asks registered voters to rate news organizations: “Do you trust ABC News? If yes, press 1. If no, press 2…” It finds greater trust for Fox News (49 percent, or 58 percent when you winnow down to those who have an opinion) vs. 31 percent for my employer, ABC News (40 percent among those with an opinion).
OK, ouch. But honestly my problem’s not with the results, but with the methodology. And I see support for my concerns in other data included in PPP’s report – on items such as partisanship, ideology, 2008 vote, the age and even the sex of participants.
Thirty-five percent of respondents in this survey pressed “2” to say they’re Republicans. Good for Fox, since as PPP notes, trust in Fox spikes among Republicans. But in our own polling, the number of registered voters who describe themselves as Republican is far lower – 25 percent in our latest poll.
Don’t like ABC/Post polls? Look at the Republican component of any of a long list of other independent good-quality surveys. The range among the general public (not registered voters, but there’s very little difference) is from 19 percent to 28 percent in recent polls by NBC/Wall Street Journal, CBS, AP-GfK, the Kaiser Family Foundation, the Pew Research Center, USA Today/Gallup and others. Just one came close to 35; another robopoll.
Try ideology. Fourteen percent in the PPP data pressed “1” to identify themselves as liberals. In our ABC/Post data, again among registered voters for the strictest comparison, it’s 25 percent. And our data on liberal self-identification have been steady for years.
There’s also 2008 vote. In the PPP poll, 47 percent pressed “2” for having voted for Barack Obama in 2008. In our last poll, among registered voters, it was 54 percent. Among actual voters, Obama, you might recall, got 53 percent of the vote.
Fifty-seven percent of the PPP participants pressed “2” to indicate they were women. Among registered voters in our data, 53 percent are women. And just 9 percent pressed “1” to indicate they were age 18 to 29. Sixteen percent of registered voters in our results are that age. A problem here may be the apparent absence of cell-phone-only respondents. I see no reference as to whether they’re included in the PPP sample, but it is illegal to autodial cell phones.
Some defenders of robopolls will point in another direction, at those that have accurately predicted election outcomes. From my perspective, accurate modeling is not an adequate stand-in for good polling. For reliability and validity alike, what matters are sound methods. And what matters next are substantive measurements, not horse-race bingo.
I could pick further. The questions PPP asked didn’t include the alternative proposition – even just minimal balancing, such as “or not?” That could have biased all the results in favor of trust, something my employer may not love hearing, but the case nonetheless. What it does indicate to me is a lack of awareness of the basic niceties of question wording.
We also have other data on hand, not asking the same question but measuring basic views of the TV news media. In a Pew poll last October, among people who had an opinion of ABC News, 39 percent called it “mostly liberal,” 17 percent “mostly conservative” and 44 percent “neither in particular.” For Fox the numbers were 16-55-28 percent. Fewer perceived Fox as avoiding an ideological slant than any of the other five networks tested.
Pew, back in May 2008, had another approach, a 1-4 scale of believability. Again among those with an opinion, local TV news and CNN did the best, both rated as believable (a 3 or 4) by 70 percent. ABC News came in at 64 percent; Fox, 59 percent.
There are plenty of ways to measure trust in the media, and in any research, the questions matter. But by my lights, methodology always comes first.
Rick Santorum Sweeps 3 States 
Pentagon to Open Additional Jobs to Women
First, ABC’s numbers are obviously flawed. Second, the robocall polls are also flawed.
The trouble is, many of today’s potential respondents, absolutely despise anyone calling them to solicit anything, including those with whom they actually do business.
Marketers, con artists, and rip off artists, have made people hate the telephone, bottom line, irregardless of how much they are on the up and up. Cell phones on the other hand, actually encourage people to talk without end, about absolutely nothing of importance. I guess you can call that the “technology factor”, which makes the telephone always accessible.
Next, you failed to mention the percentage of people who actually lie to you, in order to deceive you. That happens in all polls. Call it the “human jerk” factor, if you will, or taken a step farther, the Dem factor.
Yes, ABC is pro-Dem, and no, no one thinks your network is “objective” very much.
I still recall Stossel, when he was with ABC on 20/20, making a big deal, about how we don’t have too many people, because we could fit all our world population into Texas. The problem with that, of course, is that doesn’t provide for feeding those people, providing clean, safe drinking water for those people, nor does it explain how you will keep all those people from killing one another, in huge numbers, from too much proximity.
You see, lying with statistics, is an ages old art form, but people with common sense know when you are lying to them.
Finally, if you would simply look at what sections/topics you actually allow free responses to, you will notice that most of the reports on what Obama says, do not permit that, while your own reports from your own reporters, permit that. That is just a little discriminatory….know what I mean? That doesn’t go unnoticed.
The fact that open responses are not permitted on a lot of things, says you choose to control what the public says, and of course, that is the mark of dictatorship…..now isn’t it?
I have been totally stymied in any effort to register to comment, in the more restricted sections, by the way, by your own systems, and can only make responses through the open sections. No attempts to resolve that problem were successful. Kind of like trying to contact Microsoft…..when after getting no response whatsoever, any other way, I wrote to “customer service” about my problem, only to have the query returned unopened, because it wasn’t addressed to a “specific person”.
I shall not be sorry to see Microsoft got the way of buggy whips.
I may not be sorry to see ABC go that way, either, given the lack of journalistic ethics, the biases, and the downright lies, being published as “news”.
Posted by: Rick McDaniel | January 27, 2010, 2:33 pm 2:33 pm
This is very interesting because although as a conservative, i do trust Fox news to cover things that other news organizations seem would rather “cover up” such as congress’s recent attempt to raise the national debt ceiling by $1.9 trillion. Strange hunch that if Bush had tried this, it would have made the front page, but ABC News chose to lead with John Edward’s affair. I have often wondered about the discrepency of the polls, Rasmussen seems to be the most reliable polls as far as elections go, and it seems that many other polls are conducting their surveys in big cities or large population areas which are in fact stacked with Obama voters and liberals. So i agree with you about your polls having a much more liberal participation. But, is that because of the geographic demographic? Possible.
Posted by: gear | January 27, 2010, 3:21 pm 3:21 pm
Am I the only one who thinks the stand-out number in this article is that 16% in last October’s Pew poll thought FOX News was “mostly liberal”?
I think that 16% need to find a dictionary.
On a more serious note, I don’t fully understand the criticisms regarding sampling: if 35% of respondents identify as Republicans, surely you can use a statistical algorithm to adjust the polled sample to match pre-identified statistical data for the general population?
Also, aren’t robopolls more likely to get respondents who actually intend to vote in elections, or who are at least more interested?
Until recently I didn’t trust robopolls at all, but they’ve been very close to the mark the last couple of elections, so I’m beginning to wonder. And it also made me think, in my experience (and I do have a little), people often lie when answering poll questions asked by a person: they’re more likely to give an answer that’s more ‘socially acceptable’, e.g. calling themselves “moderate” instead of “liberal”. Not everyone, but there’s usually a few, which throws the numbers off. I don’t think this would be the case for robopolls, so I wonder if they could ultimately be more accurate if there was proper sampling etc.
Just a thought.
Posted by: Breandan | January 27, 2010, 4:44 pm 4:44 pm
CNN had made more of an effort lately to cover the real stories while ABC continues it’s left leaning slant. Fox leans right but seems to be unique in exposing the raw economic figures that the rest of the media tries to mute. We tend to watch a balance of CNN and Fox for accurate coverage of the big picture.
Posted by: Margaret | January 27, 2010, 4:51 pm 4:51 pm
gear | Jan 27, 2010 3:21:55 PM: Sorry to inform you that GWB raised the ceiling in 2002, and avoided raising it again by borrowing against social security in 2004, just before the election… Still, there is less and less integrity in polling, mainly because there is so much polling for any mundane subject that people are refusing to answer “real” polls. Since pollsters have to try polling 10 people before getting 1 usable answer, many outfits go for automated or instant polls as a cost-cutting measure, making any data suspect…
Posted by: treblig56 | January 27, 2010, 5:08 pm 5:08 pm
I not only think that particular poll is valid,I believe the numbers would have an even greater disparity if administered to all avid news watchers.
Posted by: gatoralum63 | January 27, 2010, 5:26 pm 5:26 pm
Who cares about “polls?” The only pollster who routinely gets elections right is Rasmussen anyway.
Posted by: Pappadave | January 27, 2010, 6:17 pm 6:17 pm
Some of the little yes-no robopolls with running totals that one encounters, on web sites dealing with political matters, totally defy reality. On one right-leaning site that I visit, it’s obvious that the result is skewed against Democrats in general and Obama in particular. Reminds me of a publisher I once worked for, who made up and announced greatly inflated results from relatively few mail-in ballots. I spotted him pretty quickly, as I picked up the mail at the post office each morning.
In any case, among the major TV networks, whatever their leanings, I pretty much trust ABC, NBC, CBS and CNN in their treatment of the news. I don’t trust Fox News in the least–or much of anything else owned and controlled by Rupert Murdoch.
I see that Margaret (above) says that “Fox leans right.” Anyone who thinks Fox tends to be liberal can’t be watching it much.
Posted by: Tom Camfield | January 27, 2010, 6:40 pm 6:40 pm
I never took Journalism 101, but I suspect that it would say to give the facts as best you can and let the reader decide. As an Independent voter I think that Fox does it best.Others are nothing more than Obama shills and should be ashamed of themselves. Incidentally, I voted for Obama, but “fool me twice”….you know the rest.
Posted by: Carl Benander | January 27, 2010, 8:00 pm 8:00 pm
Your Most Recent Poll has:
Obama Job Approval/Disapproval: 53/44
Unemployment has risen from 7.2% to 10.0% on his watch after he promised his stimulus would keep it below 8.0%.
He has pushed a health care package that is so awful it cost the Democrats Ted Kennedy’s seat in Massachusetts.
I could go on and on with other examples, but the bottom line is this:
Obama received 53% of the vote in 2008 and the ABC poll says that no one has defected from the ranks of people who voted for him.
That is simply impossible and incoherent.
Your ABC polling sample is very, very flawed.
If not for the fact that there are many other polls to refute the poor quality of the ABC poll, it would be a real problem as far as disinformation goes, but people will just tune out the poll at this point.
Posted by: Angelo | January 27, 2010, 10:55 pm 10:55 pm
Has anyone figured out why on-line “comments” sections are dominated by looney-tunes, Obama-hating right wingers?
Posted by: Tom K | January 28, 2010, 12:20 am 12:20 am
That 93% poll about congress not focusing on what people want you know has to be due to lobbying!! That’s why the HCR bill isn’t Obama’s and the GOP won’t touch the banks!!! You saw they all almost went into cardiac arrest when he mentioned stopping lobbying!!!
Posted by: ODD | January 28, 2010, 12:34 am 12:34 am
I was there 4 a minute after the STU and people were giving Pres Obama props!!! The lobbyists were probably peeking in on us!!!!
Posted by: ODD | January 28, 2010, 12:41 am 12:41 am
@Tom K:
“Has anyone figured out why on-line “comments” sections are dominated by looney-tunes, Obama-hating right wingers? ”
For the same reason Tom, that the comments sections all across the Internet were bloated with far left, incoherent liberal Bush haters for the last 6-7 years he was POTUS.
As far as the intent of this particular article. Nice hit piece Gary Langer. I think the reality is that FOX news, for better or worse is crushing you guys and you simply can’t take it. Period. I’m an independent. I vote on issues. I vote republican some times, democrat others.
I have never, ever seen the kind of sycophant pandering from once proud and reliable news org’s, like yours as I have witnessed since Obama won the democrat nomination for POTUS.
Have any of you guys stopped for a moment and thought that maybe the reason why a news org like FOX is doing so well while you guys are dying a slow death is because the American people know BS when they see it?
Go back to just doing the news bud. If you want to plant you nose in some liberal POTUS’s behind, do it in private.
Posted by: Philo | January 28, 2010, 1:47 am 1:47 am
The attempt to paint Fox as a right wing shill is bogus. The percentage difference of admitted liberals, who vote democrat, yet claim to be unbiased is about 15% (depending on the poll). Between ABC, NBC,CBS, CNN, in the mid 80% versus Fox at around 70%, only makes Fox look to lean right. In reality it’s only slightly less left, but they do a better job of controlling their bias. Now you guys have to do a dance to make any poll that shows Obama losing ground look suspect. If you guys hadn’t been totally in the tank for him all along you wouldn’t be so interested in trying to twist the data to cover your ass. The public is waking up to the fact that you guys helped pull the wool over their eyes, which is why your numbers continue to plummet.
Posted by: Joe | January 28, 2010, 4:04 am 4:04 am
Rasmussen can be a lousy polling organization and still predict “outcomes” accurately, because predicting outcomes is not the largest part of what they do. They have an interest in predictive methodology when they are actually predicting. Most of the time they are doing issue polling; that’s when they get really cute with the methodology because they can. If they weight their poll to prove most Americans think Obama is a citizen of Kenya and born out of wedlock, there’s not going to be a referendum on that issue to expose them. Election predictions are the bait for the public, while all these issue polls are the switch. They’re weighted toward a Republican sample purely to give the impression that Americans have all joined the tea party. As different as the Wall Street Journal’s reporters are from its loony editorial writers, that’s how different its pre-electoral polling is from its issue polling.
Posted by: ducdebrabant | January 28, 2010, 10:34 am 10:34 am
So let me get this straight? You don’t like more accurate robopolls because their samples aren’t in accord with your less accurate live survey polls? Hmm. I’ll have to think about that a bit.
Posted by: robert | January 28, 2010, 11:10 am 11:10 am
“irregardless”? “irregardless”? What the heck is “irregardless”? An erroneous word that means exactly the opposite of what is intended. Get an education!
Posted by: Badger | January 28, 2010, 11:49 am 11:49 am
If accurate predictions of election results aren’t a good independent test for the reliability of various polling methods, then what would be a good test? The argument above is circular (your methods are better because they are your methods): robopolls have methodology problems, but so do live interviewer polls. Rasmussen’s issue polls use obviously biased question wording, and no one should defend them. But PPP is a Dem firm– the attacks on them are mostly blaming the messenger. R. and PPP, and SurveyUSA, have had as good or better predictive value (as scientists say) than live-interviewer polls in recent election. That suggests that the issues confronting live interviewer polls (social desirability bias, in particular) might be worse than the issues confronting robocalls. I wish PPP got different results than they ones they get, and so does Tom Jensen, who runs PPP. But I think his methods have proved their worth.
Posted by: Steve | January 28, 2010, 12:00 pm 12:00 pm
2
Posted by: anonymous | January 28, 2010, 2:50 pm 2:50 pm
The worst part about this article is I think this guy actually believes what he is saying. I personally will take the polling firm that is accurate most of the time over one that THINKS they have a good methodology. It is like thinking you are the cream of the crop and finding out you are the cream of the crap!
Posted by: Gary | January 28, 2010, 4:30 pm 4:30 pm
Well, I guess it’s a good strategy, in a way, deciding that accuracy doesn’t matter in judging election poll quality. That way, you can just pretend your election polls are the only ones out there, and no one will notice how wildly inaccurate yours are by comparison. Makes sense to me!
Posted by: Research Rants | January 28, 2010, 8:19 pm 8:19 pm
I don’t trust any net work that spreads hate and discontent by making you look like an idiot if you don’t agree with the Propaganda they demand, if you don’t agree the world is coming to an end. Like the republican Fox net work and they call themselves the Moral Majority spreading hate and discontent, where do They attend church?
I like ABC at least they allow you to disagree, write your opinions most of the time, Thanks
Posted by: THWTCO | January 29, 2010, 10:34 am 10:34 am
When your competitors have a better predictive record than you have, its time to rethink your methodology. PPP is getting it correct. ABC is not.
Simple as that .
Posted by: Bri | January 29, 2010, 11:22 am 11:22 am
Polls. Yes, No, maybe, and Other. Suggest polls provide the pros and cons
to each poll so the people can make an informed and logical judgement vice the above Yes, No, Maybe, other.
would it make any difference? It might.
Second, I would like to see a sort of Bell Curve dipicting the info by party and news organization input. Something like that.
Posted by: Carl | February 27, 2010, 10:59 am 10:59 am