Top Hospitals Fall Flat in Contested New Report

July 31, 2013— -- Consumer Reports said some of the nation's top hospitals performed the worst when it came to patient outcomes.

Massachusetts General Hospital and the Hospital for Special Surgeries, which are both ranked top in the nation for various specialties by U.S. News and World Report, got the lowest Consumer Reports Rating possible: a solid black circle.

But some doctors have questioned the way the ratings were determined.

Using hospital billing data from nearly 2,500 hospitals, Consumer Reports used length of hospital stay after surgery as an indicator of patient well-being to determine who would get its coveted solid red circle – Consumer Reports' highest rating. But many doctors said they were wary of whether this was the best way to rate hospitals. Even a Consumer Reports researcher said she did a double-take when she saw the results.

Length of hospital stay was gleaned from Medicare billing data rather than from patients' charts, and was considered an indicator of other complications. In other words, if a patient stayed in the hospital longer than what was expected for a particular surgery, researchers determined that he or she was more likely to have experienced complications.

Even if length of hospital stay was a good indicator for how well a patient fared after surgery, some doctors pointed out that certain hospitals routinely received more complex cases than others, leading to longer hospital stays. This would skew their Consumer Reports ratings, they said, punishing them for taking the tough cases.

"The problem I have with this is that teaching hospitals, which did poorly in this report, tend to have more complicated patients," said Dr. Richard Besser, chief health medical editor for ABC News. "More complicated patients will tend to be in the hospital longer. Community hospitals, which did better in this report, often refer their more complicated cases elsewhere."

Read about how hospital closures stress the health system.

The Massachusetts Hospital Association criticized the report in a statement.

"Consumer Reports' continued efforts to rate U.S. hospitals result in greater confusion rather than clarity with an oversimplification of this extremely complex and important subject."

When Consumer Reports researcher Doris Peter first saw the top hospitals performing poorly on her list, she said she asked her colleagues to take another look at the data. She said they stood by the data.

"It's not perfect," she said. "There are things that are just not in billing data that are in the clinical record."

Guess which hospitals are prescribing fruits and veggies.

The Hospital for Special Surgery in New York, for example, takes difficult cases from other hospitals, said hospital spokeswoman Shelley Rosenstock. She said Consumer Reports didn't differentiate between the different kinds of hip-replacement surgeries, which vary in difficulty and length of hospital stay.

Peter said Consumer Reports tried to get clinical data, but groups that collect it would not provide it. Since billing data often has flaws when it comes to reporting surgical complications, Consumer Reports' researchers used the length of stay data, and adjusted it to account for hospitals that had more complicated cases and searched for outliers.

"It's the best we can do with billing data," she said.

Cleveland Clinic's rating was another surprise on the list. It is nationally ranked in 14 specialties by U.S. News and World Report, but it came away with a mediocre rating on the Consumer Reports list.

Dr. Michael Henderson, chief quality officer at the Cleveland Clinic, said using billing data wasn't a good indicator of patient complications.

"Most surgical complications occur after patients leave the hospital," Henderson said.

Although most patients leave the hospital three or four days after surgery, most complications arise seven to 10 days later, he said. As such, administrative billing data, which is what Consumer Reports used, misses 61 percent of complications, Henderson said, referring to a study by the American College of Surgeons National Surgical Quality Improvement Program.

"I actually think that of the data sets, this is one of the weakest," he said. "We are in the middle, which is not terribly surprising because, again, we do get a lot of complications. There's some risk adjustment, but I don't think this methodology is very rigorous in its risk adjustment."

Peter said Consumer Reports may use readmission data the next time it does it's hospital ratings.

"Right now, it was too much for us to do all at once," she said. "We'll start with this measure and ideally pair it with a readmission measure."

Still, NYU Langone Medical Center, which is also nationally ranked by U.S. News and World Report, got the best possible rating: a solid red circle.