EmailThe University of Iowa School of Journalism and Mass Communication and the Gallup Organization gave their 2008 Iowa Gallup Award for Excellent Journalism Using Polls to "The Numbers" blog at a ceremony in Washington last night. My remarks represented an opportunity to step back from the fray and reflect on the challenges we face in reporting survey data. I'm posting them below.
Thank you. I’m honored by this award. I want to thank my colleagues Peyton Craighill and Patrick Moynihan who are so instrumental in helping me produce the content you’re recognizing this evening. ABC News for giving me the platform to have this voice. The judges who selected our entry. And everyone at the University of Iowa and at Gallup who’ve made this recognition possible.
I especially want to remember Alec Gallup, whose involvement in the judging process in what turned out to be the final weeks of his life is an honor I’ll always treasure.
What’s so encouraging to me about this award is that it means you get what I’m trying to do with The Numbers blog. And that I’ve got company on this long slog to what we hope will be better, more accurate, more honest and more thoughtful production and dissemination of survey data.
When I talk about our work at ABC News I like to start with the story of a man who goes to his rabbi to ask forgiveness for having spread false rumors about an acquaintance. Without saying a word the rabbi takes this man outside, where he proceeds to rip up a down pillow, then stands waiting silently for a minute until suddenly a wind kicks up out of nowhere, sending these tiny feathers swirling in all directions – across the yard, over the fence, through the trees and away.
“There go your falsehoods,” the rabbi says to this man. “Go get them, and bring them back.”
At ABC News we’re privileged to speak through an enormous megaphone. Our reports go out – across the yard, over the fence, through the trees and away, around the world. Each time we are painfully aware that we have one opportunity to get it right. We can’t let this recognition paralyze us. But I want you to know that it does inform everything that we do.
Getting it right, however, is merely how we start our day. As important as it is, moving information accurately from point A to point B as if it were sheetrock off the back of a truck is wholly inadequate for our larger purpose. This is especially true in the Internet age, in which information has become commoditized, its distribution fragmented. You can get it anywhere.
It falls to us then to do something more with mere information. To elevate it into something of higher value. I call it intelligence. We transform information into intelligence through the application of the skills many of us have spent decades developing. We apply our best approaches to our craft – finding the right sources, asking the pertinent questions correctly, providing context, discerning meaning, exercising our storytelling skills. Our aim is to animate information, to bring it to life in a way that informs and on a good day maybe even enlightens.
The work of a professional journalist as I’m describing it in fact applies beyond the confines of this single industry; rather it represents the fundamental responsibilities to truth, accuracy, creativity and independence that can and should inform all of our professional lives. Whether you’re working with data or with donuts, with words or with widgets, with facts or in a factory, we are all called upon to do our best to get it right.
As universal as it is, this responsibility carries a special burden in our shared enterprise – the field of survey research. Not because of who we are but because of the unique characteristics of the material with which we work. Numbers speak with a power and authority all their own. They transform anecdote into substance; rumor into reality. Data speak to the elemental human need to define, order, and thus try to understand our world though the act of enumeration, tabulation and evaluation. You cannot read the Old Testament – and I don’t just mean the Book of Numbers – without being struck by the fundamental imperative to quantify as a means of comprehension.
We all know the phrase, “The writing on the wall.” The words it said were, “Mene, mene, tekel, uparsin” – numbered, numbered, weighed and divided. Numbered, the Lord has numbered your kingdom and come to its end. Weighed, the Lord has weighed your kingdom and found it wanting. Divided, the Lord has divided your kingdom and distributed it to your neighbors. Thus ended the days of Belshazzar, grandson of Nebuchadnezzar, in today’s Babylon.
Since that day – if not before – numbers have spoken with the weight of judgment. They carry a sense of finality, the end of debate. That I think is something to resist. Data should inform our discussions but by no means dictate their outcome. It was not the weight of Belshazzar’s kingdom that spelled its demise but rather the Lord’s determination – informed by that weight – that it was wanting.
So clearly data are powerful things, things of immeasurable value, things that imply precision and scientific method, rigor and validity and reliability. The survey data that we produce help define and explain our world in a unique and irreplaceable way.
Yet at the same time data are things of considerable danger. They can inform – or misinform. They can advance our knowledge and understanding – or cloud and confuse them. They can be like lights shining on the road to help us see our way – or like feathers flying through the air.
The power data possess makes them seductive. They pull us in, they capture our imagination in a way that encourages us to suspend our critical faculties. We want them so much that we forget to ask: Where did these numbers come from? How were they produced? Are they accurate and reliable? Do they in fact measure what they purport to measure? By what methodology, under what theoretical framework? Have they been honestly evaluated and presented?
The answers, far too often, are not good. We swim in a sea of manufactured data – produced not to independently know and assess public attitudes or behavior but rather intentionally to misinform or even disinform – to promote a policy or product, an agenda or a point of view, a candidate or a career. And even when not intentionally manipulated we see a world of polling produced on the cheap, with little or no regard to best practices, ginned up and laid on the table by everyone from hucksters, snake oil salesmen and p.r. spinmeisters who don’t know better and could care less, to some academics who for perhaps different reasons fall equally short.
Part of the problem is the media. In our reporting of survey data my generation of newsmen and -women has failed miserably in our responsibility to adhere to the fundamental requirement of journalism – getting it right. I’ve said for years that for far too long the news media have indulged themselves in the lazy luxury of being both data hungry and math phobic. It is unacceptable.
When any other alleged news comes in over the transom, we check it out before we report it, because that’s our job. But if it’s a poll – a number with a percentage sign, especially a provocative one – we want it, we’ve got to have it, we grab it and go. We make up a dozen excuses, all shameless and groundless rationalizations. The simple fact is that too much of the customary, business-as-usual news reporting of polling data constitutes no less than professional malpractice.
But it’s too easy to blame the media alone – for lack of standards, for credulity, for the wink-and-nod pact in which the media’s need for content is traded with someone else’s need for self-promotion.
Let us look as well to the purveyors and producers of these alleged data.
Let’s look to the research firms that gladly take a client’s dollar to produce a predetermined poll result.
Let’s look to the corporations, trade associations, political entities and interest groups that pay their money for the results they want.
Let’s look to the p.r. professionals who traffic in junk data like a crack dealer on the corner.
Let’s look at the convenience samplers, including the four-billion-dollar business of opt-in online panels, which while perhaps fit for market research purposes – that’s beyond my scope – has yet to enunciate a theoretical justification for what its product often is claimed to be, a representative sample of a population beyond the panel’s own self-selected membership, often gussied up with fictional claims of a calculable margin of sampling error.
Let’s look at the professional associations, thus far unable or unwilling clearly to say what does or does not constitute valid, reliable representative survey research.
And let’s look to academia. We stand on the shoulders of the methodologists and academic researchers who conceptualized our practices, tested them empirically and continue to do so. But we who now dwell in the house of inferential statistics see some who should know better moving out. The lack of budgets for quality research, the demand for provocative and thus publishable results, the pressure to promote and extend a particular thesis – these and more have the potential to push our brothers and sisters in academia into the arms of research that is not what they too often pretend it to be.
These failings, the false representations, the manipulated, manufactured and misleading data, biased questions bought and paid for, the cherry-picked results and hyped analysis – none of it excuses the news media for its failings. But the blame and the shame nonetheless are shared.
The better news is that some of us get it – and I like to think our ranks are growing. We understand the need for standards, for disclosure, for caution and honesty in the production and presentation of our work.
It’s a case I’ve been making for many years now. I’m truly pleased and encouraged to see others increasingly debating and pushing ahead on these same issues. And I’m especially grateful this evening for the Iowa/Gallup award – not simply because you like the blog, but because it means you share a commitment to continuing this conversation.
It may not always be fun, or simple, or popular. But I can promise you this: It beats chasing feathers.