Of Death and Data
Email
What do death rates in New Orleans and the availability of unused embryos for medical research have in common? Both were the subject of prominently reported news stories in the last few weeks – stories we'd advised our own people here at ABC News to avoid.
It happens much more often than I'd like. We’ve been at the forefront in establishing standards for validity and reliability in reporting survey research; too many polls are manufactured, ill-supported or misanalyzed for us to do any less. But while a few news organizations are now doing similar work, many still don’t dig into the research they’re reporting in order to establish first what's really there.
Some of these evaluative efforts are easy – the P.R. polls done on the cheap with compromised and unreliable methodology, or the advocacy and partisan surveys packed with loaded questions and cherry-picked analysis. But others – including academic and health-related studies – can be a good sight more complicated.
A day after we'd steered our radio correspondent Aaron Katersky away from the New Orleans study, for instance, there it was on page one of USA Today: Researchers had found a 47 percent increase in New Orleans' death rate after Hurricane Katrina.
Not a bad headline for a slow news day, and the researchers did have an innovative approach. Expressing concern about the reliability of offical data, they tallied death notices in the local newspaper, the Times-Picayune. They found an average of 1,317 death notices a month from January to June 2006, up from an average of 924 in 2002 and 2003. On a per-capita basis, it was a 47 percent increase.
The report, produced by the New Orleans Health Department and published in the journal "Disaster Medicine and Public Health Preparedness," described this change as statistically significant. Therein lies the problem: When our senior analyst Pat Moynihan read beyond the news release to the report itself, the data tables he saw back on page 18 indicated that the difference the researchers found wasn't statistically significant at all.
Pat then engaged in a fairly lengthy phone conversation with the report's lead author and an e-mail exchange with its second author. Final outcome: Indeed the differences in death notices reported by the study were not statistically significant. You can call us a stick in the mud, but as we see trend data, if a change isn't statistically significant, it's going to have a darned hard time being newsworthy.
That same week we looked at the embryo study, published in the online edition of the journal Science and released (presumably not by coincidence) the day George W. Bush vetoed the latest stem-cell research legislation. The survey reported that a substantial number of people who’ve undergone in-vitro fertilization would donate their unused embryos to stem-cell research, potentially vastly increasing the number of available lines. The AP – which has been much better lately at checking out survey data before jumping in – picked this one up.
We saw problems. The study was described as a "national survey," yet no sampling detail was provided beyond the fact that it'd been conducted among patients at nine fertility clinics. The phrase "national survey" implies (to us, at least) that it's a nationally representative survey. Turns out this is not the case: In conservations with us, the authors allowed that theirs was a so-called convenience sample of clinics, assembled without the benefit of random sampling procedures.
Despite this lack of representative sampling, the researchers extrapolated their results to the estimated full national population of IVF patients. That's hard for us to see as justified for attitudinal results obtained via a convenience sample, certainly not without very substantial qualifications. (We also thought the study used an awfully broad definition of propensity to donate.)
These aren't mere quibbles. I've reported previously on misreporting of scientific data in studies on subjects as disparate as autism, parenting and even naps. Whatever the subject, as studies like these enter the discourse through credible media sources they can influence our thinking and ultimately inform public policy. In deciding whether and how to report them, we need first to check them out in detail, as we do any other alleged news that comes in over the transom. As newspeople, that's our job. Or should be.
Rick Santorum Sweeps 3 States 
Pentagon to Open Additional Jobs to Women
I knew I trusted ABC news for a reason, although I originally thought it was Peter Jenning’s rugged good looks.
Thanks for upholding a standard of veracity that is frequently looked over.
Posted by: Chris | July 5, 2007, 12:17 pm 12:17 pm
Here’s a thought: when you pass up on a story for the reasons stated above, and other news outlets run with it, you should do a short news piece explaining why, when you had an opportunity to run with the story, you passed up on it. By pointing out that the analysis is flawed and, that, therefore, it is not news, it will make it clear that you are a more reliable news source than those who do run with such stories. You may argue that it doesn’t make for a good story, but I would beg to differ — how can it not be a good story for ABC to out other news organizations for providing misinformation to the public? Now that’s news!
Posted by: Troy Camplin, Ph.D. | July 5, 2007, 5:20 pm 5:20 pm
I agree with what you said. Actually, one thing that always really ticks me off is that in the Katrina problem, news crews more or less watched people get killed so they had a good shot for Joe and Jill-Bobby-sue who lick their TV screens for nourishment.
:-(
Posted by: Eponymous Fool | July 6, 2007, 5:17 pm 5:17 pm
It is clear that there were excess deaths from Hurricane Katrina. The exact number is still not clear, but it is significant. The 47% increase is our best estimate, based upon a new methodology. This study introduces and validates this methodology. However, there are three important points of the study. They are:
1. This study proves that using the local newspaper death notices is a valid methodology to estimate mortality rates in the face of a disaster. It is not only quick, but reliable. Many times the death will be posted before a death certificate is given to the state’s vital records department. It takes months for the state to get and analyze this information.
2. Our country needs to get an electronic national death registry system in place. The technology exists to make this happen. States have no way of knowing the out of state deaths. The newspaper’s notices gave us an idea, but we need to develop better methods, in light of the evacuation we had from Hurricane Katrina. The study shows that there is enough evidence to really look into not only the deaths but our system of recording deaths. Our current paper system is slow and is not able to handle a disaster like that create by Hurricane Katrina.
3. The final point is that these are excess deaths. In other words, these citizens would still be alive had Katrina not happened. The mortality data from the State of Louisiana confirmed a significant increase in the mortality rate for the first quarter in 2006 on New Orleans. However, they had no idea about the number of out of state deaths. In order to prevent these deaths, we need to study the causes of death. Death notices and even death certificates do not capture the complete picture. We need to do chart reviews as we do with fetal infant mortality reviews to determine the actionable factors leading to the deaths so we can develop prevention interventions.
This is the main purpose of the article, to get a more detailed study done to prevent citizens from dying prematurely from an evacuation in a disaster. Is it stress, diet, lack of access to healthcare services, lack of prescriptions etc? At this point, no one knows. We have to figure this out. Do we need electronic medical records, personal health records, nation prescriptions and access to health information like ATM’s in banking? Should all states be compelled to honor the Medicaid insurance from other states? If you leave your state and get sick, Medicaid will not cover your out of state medical bill. These are important issues which must be addressed nationally.
Finally, these were premature deaths. Hurricanes are here to stay, there will be many more hurricanes and evacuations, not only in Louisiana, but in the southeastern US. Primary functions of government are protection and security. The life of our citizens is the most valuable thing we need to protect. We must prevent more citizens from dying prematurely in the next evacuation and disaster.
Kevin U. Stephens, Sr., MD, JD
Director,
New Orleans Health Department
Posted by: Kevin U. Stephens, Sr., MD, JD | July 10, 2007, 10:02 am 10:02 am
Dr. Stephens, thanks for your reply. I understand and admire the aim of your work. My perhaps narrow point is that conclusions based on data are best constrained by what those data say, clearly described and fully explained. Initially in the paper (http://www.dmphp.org/cgi/reprint/1/1/15) the finding of a 47 percent increase in mortality is described as “significant”; at a later point the data are described as “suggesting a significant difference”; at another you describe “little overlap” between confidence intervals. As noted, we see elsewhere that the result is not significant at the customary 95 percent confidence level; the CIs (Table 2) are 52.31-72.02 for 2002-2003 and 56.44-126.30 for 2006.
Posted by: Gary Langer | July 11, 2007, 2:38 pm 2:38 pm
Two wrongs make it right:
Stephens should have avoided using the term “significant” because the data appear to be population data. There is no implied inference to a larger population and the differences in rates are (discounting tabulation errors) the population differences. We can ask, what is the probability of getting such a big difference had the entire population been sampled? This obviously makes no sense since Stephens *did* sample the population. Conclusion: relative death rates were higher. With 100% certainty (at least with respect to sampling). Period.
Langer then got hung up on the “claims” of significance. Since these claims were inconsistent with reported confidence intervals, Langer concluded that the results should be viewed skeptically. So one mistake created a second. Sometimes one can be too vigilant!
Posted by: Eric Plutzer | July 12, 2007, 1:53 pm 1:53 pm
Eric, the authors could have used you as a peer reviewer! As a news guy, though, I’m stuck with evaluating the paper as it is, not as you or I wish it were. They ran a sig. test on the proposition that they have a 47 percent increase in mortality, and it came out as n.s. Maybe conceptually they didn’t need to run and report the test, but they did. They describe it variously, using the language in my last post; but when we asked directly, they told us, right, it’s not significant – as the CIs in Table 2 indicate. It’s also not clear how they ran it. Dan Merkle, our decision desk director here at ABC and our most stats-ready guy, says it looks like they based the confidence intervals on the standard deviations rather than on the standard errors, which does not make a lot of sense to him. Again, none of this ultimately is to take away from the important work Dr. Stephens is doing. It’s more simply that we have a paper whose headline result, from a news perspective, is not supported by the significance test the authors themselves apply.
Posted by: Gary Langer | July 13, 2007, 11:27 am 11:27 am