"When I use a word," Humpty Dumpty said in a rather a scornful tone, "it means just what I choose it to mean -- neither more nor less."
"The question is," said Alice, "whether you can make words mean so many different things."
Even if it's not answered, a natural question that arises in almost every news story about some quantifiable phenomenon -- which we might label X -- is "How many X?"
Sometimes the question is phrased, "What is the incidence, percentage, or rate of X?"
Since stories are more newsworthy when this number or rate has changed significantly, there is a strong media tendency to look for large changes in X. Often, however, these large changes are spurious, the result of using a flexible yardstick to measure X.
Here are a few recent examples.
There was a recent spate of stories about the rise in the maternal death rate in this country. As has been widely reported, awareness that maternal deaths were not all being recorded has increased over the years. In an effort to correct this, new guidelines were instituted in 1999 and 2003.
The Centers for Disease Control report "Maternal Mortality and Related Concepts" spelled these out:
"... the [new coding guidelines] introduced new details and categories in the cause-of-death titles associated with pregnancy, childbirth, and the puerperium ... the U.S. Standard Certificate of Death was revised to ask explicitly whether any female death was associated with pregnancy, instead of relying on the person filling out the form to voluntarily provide that information."
These redefinitions of maternal mortality had the predicted effect of leading to a higher incidence of reported maternal deaths.
Not surprisingly, these upticks occurred in 1999 and 2003. Some of the increase over the last decade or so may have reflected real change, but much of it seems to have been due to the redefinition. It should go without saying that however sliced, the rate is scandalously high.
Another recent story, this time from the U.K., illustrates the same point. The Conservative Party claimed that crime in various districts had risen significantly under the Labor Party, more than tripling in one district.
As with maternal morality, there was a change in the protocols for reporting crime in 2002. One of the differences is that before this date police had decided on their own whether to classify a crime as violent.
After this date a crime was automatically classified as violent if the victim said it was, and victims are more likely than police to view any sort of criminal action against them as violent.
As Mark Easton, winner of a statistical excellence in journalism award, observed, "The numbers they use don't compare like with like, they've ignored big warnings plastered over the official statistics telling people not to try and compare the two sets of numbers."