A Polling Fairy Tale

Once upon a time an utterly inconsequential poll made the rounds – but one that underscored a highly consequential question in my corner of the world, and, ultimately, yours: What kind of standards news organizations apply to the data they report.

In the last few days U.K. newspapers including The Independent, the Telegraph, the Daily Mail, the Scotsman – and here at home, The Christian Science Monitor and a New York Times blog item – covered what’s said to be a poll of 3,000 British parents by an outfit called TheBabyWebsite.com. Among the findings: “Three-quarters of mums and dads try to avoid stories which might give their children nightmares.”

Cutesy stuff, headline-wise. But are there wolves in the woods?

TheBabyWebsite itself offers comprehensively no information about how its data were collected. But when we asked, we were referred to OnePoll, a British “online market research company.” Its come-on couldn’t be clearer: “Join today and start earning serious cash,” says the banner ad atop OnePoll’s homepage.

Here’s the deal: “Register using our simple sign-up form and start earning cash right now,” OnePoll says. “We pay you serious cash, ranging from 10p to £1 a survey, and cash prizes for as much as £1,000. Other prizes include gift vouchers, spa treatments and electrical goods. Sign-up today and earn an instant £2.50.”

There even are testimonials. “I love doing the OnePoll surveys,” the site quotes Paul Kendall, whose photo shows a bearded and bespectacled fellow with a somewhat sour look. “They're quick and can be done whilst I have a cuppa. Thanks OnePoll.”

And thank you, Paul, for the chance to talk about this stuff.

These things are called opt-in online panels. People sign themselves up to click through internet-based questionnaires in exchange for cash or gifts – you might have seen the pop-up come-ons. Results are not based on a probability sample of respondents, but what’s called a convenience sample. They have no calculable error margin, because they live outside the realm of inferential statistics on which scientific surveys are based.

Beyond sampling, there are other potential problems. One company’s study (not OnePoll) found that the vast majority of its opt-in panel surveys actually were filled in by a very small number of panelists – semi-professionals, if you will. Since participants join for the compensation, one might wonder whether they pay real attention to the questionnaires or just click through them. (“The more surveys you participate in, the more you can earn,” says OnePoll. “Typically we will have 20 to 30 active polls at any one time.”) There’s the question of whether participants in some opt-in panels might create multiple assumed identities in order to enhance their prize-winning potential. Different sites have widely different levels of controls, and I haven’t seen much talk about validation.

Look at this e-mail, forwarded to me by a friend at Princeton University, recruiting students to join one such poll-taking club. Is this the kind of data (using that word advisedly) that you want to use to inform your judgment?

Whoa, you might say: For a poll about bedtime stories, who gives a hoot?

We all should. The reality is that surveys cover every possible subject you can imagine, from the frivolous, like this week’s example, to the profound. For each one, big and small, I think it’s the responsibility of news organizations to establish serious standards for what survey research they will and will not report – and to hold themselves to those standards.

What we see as the lack of reliable representativeness in these opt-in panels rules them out for us; our policy is not to report them. But I’m not playing gotcha; I have no doubt you’ll find polls reported by ABC News that by our standards shouldn’t have made air. (The New York Times, among others, has established standards much like our own.) Policing it’s a bear; some reporters just can’t deny themselves a compelling little item.

Often, of course, that’s the point. Polling is the new public relations; produce a “poll” with a catchy headline and media pick-up is almost guaranteed. Look again at the OnePoll website: “Solid research can be the catalyst for national news exposure,” it advises. One client is quoted as saying it’s “our first choice when planning a survey campaign to generate national news coverage.” The site says OnePoll “has worked for and on behalf of over 150 PR agencies.”

Reporting PR campaigns, though, is not our purpose; reporting meaningful, reliable news is. That means differentiating between what is and isn’t good-quality, representative survey research. Otherwise we’d be in a different business: telling fairy tales.