Good things sometimes come from bad data.
OK, rarely. But it happened the past week, after a program of the Robert Wood Johnson Foundation put out some thoroughly unreliable results on teens and dating. Once alerted, the foundation pulled back the purported poll – and said it’s revamping its procedures to ensure it sticks with the good stuff in the future.
The release came from an RWJF-funded group called “Start Strong Teens”; the headline, “New Survey Finds Majority of Kids Ages 11-14 Are Dating.” A mommy blogger picked it up, one of our producers at Good Morning America caught wind and we were off to the races in our poll-vetting mode, checking out just how the survey was done.
The answers, as they came in, were depressingly familiar. Though described repeatedly in the release as a “poll” and a “national survey,” this project in fact was a collection of questionnaires filled out at nine of 11 Start Strong sites around the country, with no random selection of respondents, nor any rationale to suggest why participants in the program, even if randomly selected, would represent all teens nationally. Start Strong’s publicist maintained, lamely, “We never claimed in the methodology that this was a nationally representative sample.” True – but that plainly was implied by the headline, characterizations in the release and its use of the terms “poll” and “national survey.”
The questionnaire itself was written with the involvement of teens involved in the Start Strong program, under the supervision, it was said, of a professional pollster. Not a good one, I’m afraid; as if the absence of sampling weren’t enough, the instrument itself violated almost every cardinal rule of questionnaire design.
We see this kind of thing 10 times a week, but not from the likes of RWJF, a major, research-focused foundation with a strong reputation for probity. And indeed what saved this from ending up as just another dreary day in the bad-data salt mines was the response of the foundation itself.
“As a practice, ALL research that we disseminate should be vetted by an internal research and evaluation officer,” who “gives it an initial ‘smell test,’ reviews the methodology and gives the sign-off that something is ready for public dissemination,” the RWJF’s public affairs director, Adam Coyne, told me by e-mail. “Clearly that had not happened in this case.”
Coyne agreed that the release “does imply a nationally representative sample, which we did not have. … We are not justified in saying anything about teens dating at age 11 to 14.”
The foundation pulled the news release, “halted all media outreach around this project” and had a “serious conversation” with the p.r. firm. Better yet, he reported, the issue was elevated to senior management at the foundation, and “we are refining our internal processes to ensure that everything we disseminate is honest and accurate, meets strict methodological standards, and hopefully is of interest to the field.”
Polling, I’ve said many times, is the new p.r.; we swim in a sea of junk data, produced not to measure attitudes or behavior accurately, but to promote a product, project or point of view, validity and reliability be damned. As often as the news media (not to mention mommy bloggers) fail to do their part to check out the purported data they’re reporting, so do the sponsors, producers and and publicizers of that data bear a responsibility to represent honestly what it is.
The reason is simple, and it hits as close to home here at ABC News as I’ve learned it does at the Robert Wood Johnson Foundation: “Reputation and credibility are all that any organization has,” Coyne said, “and we take ours very seriously.”
Amen to that, and back to work.