EmailThere’s a lesson in the dispute between the polling outfit Research 2000 and the website Daily Kos that broke into a firestorm this week, but I’m not sure the commentary to date has nailed it. Transparency, to me, is not the prime issue. Due diligence is.
I’ve no idea of the merits of Kos’ claims that R2K provided it with falsified data in pre-election polls; R2K denies it. I’m sympathetic to Kos’ demands for the original datasets – the underlying records of each interview – a call that's been echoed in a statement by the National Council on Public Polls (of which ABC News is a member).
But in picking apart this controversy, that’s not the only place to look.
The NCPP said it "believes public disclosure of all the relevant information about the polls in dispute will provide a solid basis for resolving this controversy.” Maybe so; I've long maintained that disclosure is essential in evaluating survey quality. But maybe not. If a vendor stands accused of fabricating results, it can just as easily stand accused of fabricating the underlying data. Not simple, but lying never is.
Disclosure, then, as necessary as it is, does not in and of itself assure data quality. That takes another step: The need for those who fund and then promote or disseminate these data first to dig deeply into the bona fides of the product.
Indeed to my mind the delivery of methodological details, including original datasets, should be an initial and ongoing requirement of any polling provider, not a demand only when controversy arises. That points to a more basic lesson of this story: the principle of caveat emptor.
Polling is a complex undertaking that can be produced in many ways – some highly valid and reliable, some less so, some not in the least. Anyone buying it needs to take the trouble to ascertain precisely how it’s being carried out – in sampling, questionnaire design, respondent selection, interviewing, quality control, weighting and more – and to assess the appropriateness of these methods for the intended use of the research.
This too rarely happens. Many fall prey to the notion that it’s all just numbers – that this 63 percent is as good as that one. The compelling nature of data induces us to drop our critical faculties, the need to assess just how these numbers were produced. Without that assessment, running with numbers is like running with scissors.
Just as I don’t know how R2K produced its results, I likewise don’t know how far Daily Kos went in examining precisely what it was buying, branding and delivering to a data-hungry world. But I do know this: Long before we get to disclosure, the best way to obtain good data is to demand it.