Top Hospitals Fall Flat in Contested New Report

July 31, 2013— -- Consumer Reports said some of the nation's top hospitals performed the worst when it came to patient outcomes.

Massachusetts General Hospital and the Hospital for Special Surgeries, which are both ranked top in the nation for various specialties by U.S. News and World Report, got the lowest Consumer Reports Rating possible: a solid black circle.

But some doctors have questioned the way the ratings were determined.

Using hospital billing data from nearly 2,500 hospitals, Consumer Reports used length of hospital stay after surgery as an indicator of patient well-being to determine who would get its coveted solid red circle – Consumer Reports' highest rating. But many doctors said they were wary of whether this was the best way to rate hospitals. Even a Consumer Reports researcher said she did a double-take when she saw the results.

Length of hospital stay was gleaned from Medicare billing data rather than from patients' charts, and was considered an indicator of other complications. In other words, if a patient stayed in the hospital longer than what was expected for a particular surgery, researchers determined that he or she was more likely to have experienced complications.

Even if length of hospital stay was a good indicator for how well a patient fared after surgery, some doctors pointed out that certain hospitals routinely received more complex cases than others, leading to longer hospital stays. This would skew their Consumer Reports ratings, they said, punishing them for taking the tough cases.

"The problem I have with this is that teaching hospitals, which did poorly in this report, tend to have more complicated patients," said Dr. Richard Besser, chief health medical editor for ABC News. "More complicated patients will tend to be in the hospital longer. Community hospitals, which did better in this report, often refer their more complicated cases elsewhere."

Read about how hospital closures stress the health system.

The Massachusetts Hospital Association criticized the report in a statement.

"Consumer Reports' continued efforts to rate U.S. hospitals result in greater confusion rather than clarity with an oversimplification of this extremely complex and important subject."

When Consumer Reports researcher Doris Peter first saw the top hospitals performing poorly on her list, she said she asked her colleagues to take another look at the data. She said they stood by the data.

"It's not perfect," she said. "There are things that are just not in billing data that are in the clinical record."

Guess which hospitals are prescribing fruits and veggies.

The Hospital for Special Surgery in New York, for example, takes difficult cases from other hospitals, said hospital spokeswoman Shelley Rosenstock. She said Consumer Reports didn't differentiate between the different kinds of hip-replacement surgeries, which vary in difficulty and length of hospital stay.

Peter said Consumer Reports tried to get clinical data, but groups that collect it would not provide it. Since billing data often has flaws when it comes to reporting surgical complications, Consumer Reports' researchers used the length of stay data, and adjusted it to account for hospitals that had more complicated cases and searched for outliers.

"It's the best we can do with billing data," she said.

Cleveland Clinic's rating was another surprise on the list. It is nationally ranked in 14 specialties by U.S. News and World Report, but it came away with a mediocre rating on the Consumer Reports list.

Dr. Michael Henderson, chief quality officer at the Cleveland Clinic, said using billing data wasn't a good indicator of patient complications.

"Most surgical complications occur after patients leave the hospital," Henderson said.

Although most patients leave the hospital three or four days after surgery, most complications arise seven to 10 days later, he said. As such, administrative billing data, which is what Consumer Reports used, misses 61 percent of complications, Henderson said, referring to a study by the American College of Surgeons National Surgical Quality Improvement Program.

"I actually think that of the data sets, this is one of the weakest," he said. "We are in the middle, which is not terribly surprising because, again, we do get a lot of complications. There's some risk adjustment, but I don't think this methodology is very rigorous in its risk adjustment."

Peter said Consumer Reports may use readmission data the next time it does it's hospital ratings.

"Right now, it was too much for us to do all at once," she said. "We'll start with this measure and ideally pair it with a readmission measure."

Still, NYU Langone Medical Center, which is also nationally ranked by U.S. News and World Report, got the best possible rating: a solid red circle.

"This is something that all hospitals really work on very hard," chief quality officer Dr. Martha Radford said, adding that coordinating care has been a key focus at NYU Langone. "It starts before the patient gets to the hospital so people know how long they're going to be expected to stay."

She pointed out that although large teaching hospitals get complicated cases, some did well.

Radford noted that the Mayo Clinic did "very well, at least in one of their hospitals."

In Minnesota, where Mayo Clinic has seven separate hospitals on the list, one did very well, one did well, three were average, one was poor and another was very poor, according to the Consumer Reports' rankings.

Radford said that even though risk adjustment could compensate for differences in populations and case complexity, it wasn't perfect. Although billing data is used in many hospital ratings and rankings, it's not as good as other hospital data, said Radford.

"Billing data are useful, but they are not as nuanced as other data," she said, adding that NYU Langone routinely pays attention to its rankings and ratings, but only those in which the methodology is transparent.

As a patient, Consumer Reports' Peter said she would start with the Consumer Reports list she and her colleagues created. Then, she would look for other data points to determine whether to undergo surgery at a particular hospital. For instance, Peter said a surgeon her mother was considering had state sanctions against him, and she found out about them by searching a database on her state's health website.

Besser said patients should also ask their doctors how many procedures they've done and ask what their complication rates are.

"This is information that they should give you if you ask," he said. "When you are in the hospital, there are things you can do to reduce the chances you will be the victim of a medical error or will acquire an infection you didn't come in with."