A paper published by the U.S. Census Bureau reinforces serious questions about the reliability of surveys filled in by volunteer participants on the internet – and extends those concerns in a new direction.
Past research has thrown doubt on the ability of so-called “opt-in online panels” to produce results that accurately reflect the views of the broader population. The new study not only reinforces that evidence, it also calls into question whether such data are reliable for two other key purposes, evaluating changes over time and differences among groups.
The paper was written by Josh Pasek, now an assistant professor of communication studies at the University of Michigan, and his Ph.D. adviser, Prof. Jon Krosnick, of Stanford University. They had unusually well-suited data with which to work: A set of parallel surveys commissioned by the Census Bureau in 2009 and 2010 to investigate the public’s willingness to fill out the 2010 Census. One was conducted by the Gallup Organization via traditional, random-sample telephone interviews; the other, by a company called E-Rewards, via an opt-in online panel.
Pasek and Krosnick have two bottom lines: The first, replicating previous work, is that data from these two sources are notably different in terms of representing population values and attitudes alike. They report:
• “The telephone samples were more demographically representative of the nation’s population than were the Internet samples after post-stratification.”
• On attitudinal questions, the proportions of most-common responses in the two datasets differed by 13 points on average and more than 30 points in some cases. Controlling for “no opinion” responses, which differed as a function of how the surveys were done, didn’t change the conclusion.
But the authors go further; while distributions of opinions and behaviors were “often significantly and substantially different across the two data streams,” they say, so were two other items: “relationships between the variables and change over time in the variables.”
That’s a heck of a kicker. Some researchers say OK, even if opt-in online sampling shouldn’t be used to estimate population values, it’s still fine for use in examining differences between groups and changes over time. Pasek and Krosnick’s work suggests otherwise.
• “Changes over time in the two data streams diverged more than would be predicted by chance in almost two-thirds of the cases examined.” This, they say, created instances “where the two data streams told very different stories about change over time.”
• Among 11 attitudinal variables that predicted intent to complete the Census, eight were significantly different, in terms of the strength of their predictive power, between the two datasets; among nine predictive demographic variables, five differed significantly in strength of predictive power; and among seven attitudinal variables that showed change over time, the patterns of three differed significantly.
The authors say different relationships among variables “suggests that respondents in the Internet surveys were systematically different from or approached their tasks differently than did the telephone respondents.” In all, they report: “Differences between data streams were sufficiently common that a researcher’s choice of a data stream to use would affect the conclusions he or she would reach.”
This cuts to one of the foundations of a multi-billion dollar business, the use of opt-in online data in designing marketing campaigns. The Census Bureau bought survey data to evaluate how to encourage people to fill out the Census. Pasek and Kronsnick suggest its efforts to achieve that goal would have differed significantly based on which data source it consulted.
The probability sample, they note, is the one based on established sampling principles, and comes closer to known population values.
A few provisos: Non-probability samples are commonly used in other areas not addressed in this paper, e.g. medical or psychological studies. And this is a single study with data on a specific issue; as with any such work, the findings that are substantively new need replication in further research.
That said, it’s rigorous work from researchers at the forefront of academic evaluations of survey data – and their conclusions are striking: “This investigation revealed systematic and often sizable differences between probability sample telephone data and non-probability internet data in terms of demographic representativeness of the samples, the proportion of respondents reporting various opinions and behaviors, the predictors of intent to complete the Census form and actual completion of the form, changes over time in responses, and relations between variables.”
For all its importance, this should not be an astonishing result. Opt-in online panels are comprised of people who sign up to click through questionnaires on the internet in exchange for points redeemable for cash and gifts. The use of these panels is vast, especially in market research, because it’s cheap and fast. But it’s also problematic, because the nature of opt-in panels violates the most basic principles of probability sampling.
We ruled out reporting opt-in online panels at ABC News more than a decade ago. In 2008 David Yeager, then another student of Krosnick’s, along with Krosnick and several of their colleagues, wrote a groundbreaking paper questioning the accuracy of opt-in online data. And a year ago the American Association for Public Opinion Research issued a report saying such panels should not be used to represent population values, should not be described as representative and should not claim a margin of sampling error.
Still, the AAPOR report left the door open a crack for the purpose of evaluating relationships among variables. It said, “a good deal of research is focused on improving our understanding of how personal characteristics interact with other survey variables such as attitudes, behaviors and intentions. …Under these and similar circumstances, especially when budget is limited and/or time is short, a nonprobability online panel can be an appropriate choice.”
Pasek and Krosnick’s paper encourages a rethink of that view.
Surely it’s coming; these researchers’ findings deserve the same close review and comment that accompanied the 2008 Yeager/Krosnick paper. That study, it should be noted, has stood the test: It recently was accepted for publication in the leading peer-reviewed academic journal in the field, Public Opinion Quarterly.
Full disclosure: Jon Krosnick is a senior adviser to my company, Langer Research Associates; and I’m a longtime member of AAPOR.