Hospital Rankings: Bulk and Bunk

What’s inside the sausage of hospital rankings?

Every year the U.S. New & World Report publishes its rankings of the nation’s top 50 hospitals. Hospital administrators await this top 50 report with a tension and fervor that rivals the NFL first-draft pick.

As soon as the report is released, snippets rocket their way into donor appeals, local hospital signage, and highway billboards. “Ranked X by US News and World Report” becomes a descriptor that is supposed to immediately invoke quality.

Just what is behind the US News and World Report rankings, and do they actually mean anything? Dr. Ashwini Sehgal, writing in the Annals of Internal Medicine , examined the structure of the famed ratings.

US News and World Report utilizes three components, equally weighted, to evaluate hospitals—structure, outcome, and reputation.

Structure is what you might expect—size, staff numbers (especially nurses), patient volume, facilities. Outcomes are mortality rates, adverse events, disease outcomes, patient safety issues.

Reputation is a much more vague metric. Two hundred and fifty physicians from each of the medical specialties were randomly selected from the AMA Physician Masterfile. These doctors—about half of whom responded to the survey—were asked to rank the top 5 hospitals in their specialty.  From these personal opinions, a reputation score was developed.

Of these three measures—according to the statistics in the Annals article—reputation correlated most strongly to the final ranking.

This means that this supposedly objective ranking of the nation’s hospitals depends mostly on the subjective reputation score. Not exactly the scientific, objective scale is purports to be.

On the home page for the rankings, there is a side-bar ad. “Promote Your Hospital,” it says. “Display your ranked hospital’s Best Hospitals badge on your website.” Clicking onto that link offers numerous opportunities for hospitals to “enhance reputation and visibility” via US News and World Report. One can “purchase an enhanced hospital listing” or “become a featured hospital.”

These rankings, therefore, seem mainly to be a marketing tool for US News and World Report. This shouldn’t come as a surprise to anyone. U.S. New & World Report is a commercial entity whose goal is, and justifiably needs to be, earning revenue. It is not a scientific non-profit such as the Institute of Medicine or the NIH, from whom we expect high-quality, unbiased evaluation.

However, once the U.S. New & World Report becomes a household name for hospital rankings, it does have some obligation to be as objective as possible. If it wants to be an arbiter of quality in any sort of quantitative sense, it has to demote the input of subjective measures.

The reliance of the subjective value of reputation suggests that we will be in an endless loop of self-promotion. Newer, smaller hospitals will never have a chance to be ranked at the top, even if they deliver superior care, simply because 125 doctors don’t know of them. But maybe they should…