In Autism Studies, Keep an Eye on the Definition

Oct 5, 2009 9:29am

2:15 p.m. update: I've invited thoughts on this post from Stephen J. Blumberg, one of the study's co-authors, and I'm grateful for his quick reply, which I'm pasting directly below my comments.

A new study on the level of “autism spectrum disorders” in the United States is sober and circumspect. Some of today’s reporting about it has been less so.

The report, based on the federal government’s National Survey of Children’s Health in 2007-2008, finds a higher level of autism and related conditions than a previous NSCH survey found for autism alone. What needs to be clear is that the two are not directly comparable.

The 2003 NSCH survey asked about “autism,” finding it reported in 0.55 percent of American children. The new survey, instead, asked about “autism, Asperger disorder, pervasive developmental disorder or other ASD.” It produced an estimate of 1.1 percent.

It’s entirely likely that the new, broader definition contributed to the higher count; indeed the report’s authors list “more inclusive survey questions” first among their suggested explanations (higher awareness and earlier diagnosis are the others).

There could be other reasons, as well. The survey was done among parents, asking if they’ve ever been told by a doctor or health care provider that a child of theirs had any of the disorders listed. That is not a measure of clinical diagnosis; indeed, the authors note that in 38 percent of the reported cases, parents said their child does not still have the condition now.

The study’s authors say this could mean that autism or its related disorders “may have been initially suspected… but subsequently ruled out and never truly diagnosed.”  I also wonder whether, even despite the question’s context, “pervasive developmental disorder” might have been misunderstood by some parents as developmental disorder of any type, perhaps also contributing to the higher estimate. 

It would have been a good approach for this survey to measure each of the conditions listed by itself; it could then have aggregated them to get a net total of “autism spectrum disorder” while maintaining the estimate of autism alone for comparison to 2003. That looks like a missed opportunity.

It’s worth keeping in mind that surveys, even big ones, are not laser surgery. The difference between this measure of autism and related disorders, vs. the previous measure of autism alone, is 0.55 percentage points. That’s statistically significant given the sample sizes (the latest was done among 78,037 parents). But reporting it as a doubling of autism incidence takes advantage of a change in small numbers, as well as disregarding the different definition.

Even first-quality data, moreover, also are subject to nonsampling error. The survey was done by landline phone only; while that was customary as recently as a year or so ago, more good surveys these days include cell-phone-only respondents.

And there are the questions themselves. Surveys, fundamentally, are based on what respondents understand the question to be, and are limited by the questions asked. The sizable number of parents in this study who said they were told their child had ASD, but that he or she does not have it now, leaves questions about the questions that were asked, and suggests follow-ups that weren’t.

Other studies of autism and related disorders have produced different estimates, using different methodologies as well as, again, different questions. I reported here on one such 2007 report, a CDC study in which state estimates were used, questionably, to produce a national estimate of prevalence. In response to the NSCH report, the CDC on Friday said it’s updated that study, also producing about a 1 percent prevalence estimate. It said its detailed report was not yet ready.

That’ll be worth reviewing closely, especially in terms of its comparability to previous CDC work. As they go forward, what autism researchers seem to need is an agreed-upon definition of exactly what it is they’re measuring, as well as a consistent approach to measuring it.

The new study is published in the journal Pediatrics. One co-author, Dr. James M. Perrin, is reported to be “supported in part by a grant from Autism Speaks,” an autism awareness group. I have occasional contact on other matters with the second author, Stephen J. Blumberg, a researcher at the National Center for Health Statistics who’s also standards chair of the American Association for Public Opinion Research.

Comments from Stephen J. Blumberg:

Thank you, Gary, for your critical look at our Pediatrics article and the subsequent reporting. Our article on the results from the 2007 National Survey of Children's Health (NSCH) states that the new estimate “is higher than previous US estimates.” However, we did not draw direct statistical comparisons between the new estimates and previously available estimates from the 2003 NSCH, and we do not recommend that others do so.

You identified one reason not to make the comparison. As you note, the wording in the 2003 survey was limited to "autism" whereas the 2007 wording was more inclusive of other autism spectrum disorders (ASDs). We do not know how parents of children with Asperger disorder or pervasive developmental disorder might have responded to the 2003 survey.

There is another reason not to make the comparison. In 2007, a yes answer to the first question ("Has a doctor or other health care provider ever told you that your child had the condition, even if he/she does not have the condition now?") was followed by a second question asking the parent whether he or she believed that the child "currently" has the condition. The reported prevalence estimate (1 in 91 children 3-17 years of age) is based on responses to the follow-up question.

The follow-up question was not included in the 2003 survey. Instead, the 2003 survey simply asked "Has a doctor or other health professional ever told you that your child has the condition?"  The change to the survey was made because we learned that many parents do not believe that their children currently have ASDs, even though they have been provided with autism spectrum diagnoses in the past. We do not know how these parents might have responded to the 2003 question.

The questions were changed because we now know more about parents' experiences with autism. We believe that the new questions are improvements over the old questions. Unfortunately, changes such as these, made with the best intentions of improving current understanding of the prevalence of parent-reported autism diagnoses, prevent us from directly comparing the new results to our previous results. These are the trade-offs inherent in survey research.

Survey research with landline telephones has its limitations, including nonsampling error due to the exclusion of cell-phone-only households (though there is no evidence that households with landlines are more or less likely to include children with ASDs). Because of these limitations, I disagree with your call, Gary, for a single consistent approach to measuring autism prevalence. Nationally-representative telephone surveys of parents play one important role. CDC's Autism and Developmental Disabilities Monitoring (ADDM) network, which estimates prevalence based on studies of administrative records from a cross-section of doctor's offices and schools, offers an independent and different role (including an opportunity to collect more data than can be accomplished in a relatively brief telephone survey). When these two approaches yield similar results (approximately 1-in-100 with autism spectrum disorders), we can have greater confidence in the validity of the estimate.

You are using an outdated version of Internet Explorer. Please click here to upgrade your browser in order to comment.
blog comments powered by Disqus