The US News and World Report Rankings, Explained

By Benjamin Wallace-Wells | Wednesday, September 30, 1998

Editor's Note: Richard Morse is Director of Data Research for US News and World Report's College Issue. His group was responsible for the statistical information that formed the backbone of US News and World Report's college rankings.

The Dartmouth Review: In this year's US News & World Report rankings of national universities, Dartmouth was ranked tied for tenth with Columbia, Brown, and Northwestern, three schools that it had been ranked ahead of in recent years. Dartmouth was also passed this year by Cornell, Penn, and Caltech — again, all schools that we had been consistently ranked ahead of before this year.

This seems quite a dramatic drop — it seems doubtful that Dartmouth somehow got that much worse from last summer to this summer or that Penn, for example, got that much better.

Richard Morse: What you have to remember is that, significant as it may seem to Dartmouth, a drop of three places or so isn't terribly statistically significant when you consider a pool of some two hundred and twenty-nine institutions that Dartmouth is competing against. So such a change could be prompted by few small changes in any of the factors — it wouldn't take that much difference statistically to move any school up or down a few slots.

TDR: Why specifically did Dartmouth lose stature from last year to this? In what categories are the changes?

Morse: If you compare the data we published in 1997 vs 1998, you'll see a small decrease in the financial resources rank. The financial resources rank is a record of the amount an institution spends, per student, on research and instruction. Essentially what this does is measure the amount of money an institution will spend on faculty and facilities.

TDR:It seems like large, research universities like MIT or the University of Chicago would do better in this category than would smaller, undergraduate oriented colleges like Dartmouth, because the institutional resources that the research university can bring to bear are necessarily greater. Does that seem reasonable.

Morse: Yes. Keep in mind that we are limiting the list to undergraduate institutions, so graduate programs wouldn't come directly into play, but yes, indirectly there might be that effect.

TDR: Were there any other areas in which Dartmouth's ranking dropped?

Morse: The only other one was Faculty Resources. Faculty resources is a measure of five factors — in diminishing order of importance, class size, faculty salary, the proportion of professors with the highest degree in their field, student-faculty ratio, and the proportion of the faculty that is full-time.

Again, Dartmouth's drop was fairly small.

TDR: How is the 'Academic Reputation' ranking calculated?

Morse: We canvas College Presidents, Deans, and Professors across the country to ask them their opinion on colleges — to give them a rank from one to five, five being the highest. These responses are then averaged and you get the academic reputation rank [in Dartmouth's case, 4.4].'

TDR: Wouldn't research universities be at an advantage here as well? You're asking opinions and talking about reputations, so it seems to me that its going to be difficult for people, and especially academics, who have been through the graduate programs and are professionally tied to them, to mentally seperate the prestige of an institutions graduate programs from that of the undergraduate program. So, again, respect for the graduate programs of a research university might spill over into this ranking, and, again, put undergraduate-oriented schools like Dartmouth at a disadvantage.

Morse: Well, yes, it's true that it may have a halo effect. I am not disputing that claim, though I don't know how one would accurately measure if there is a halo effect and the raters are taking into account the reputation of the broader graduate programs.

TDR: Well, Dartmouth is an institution which prides itself on teaching and the centrality of the undergraduate curriculum. It seems to me as if a lot of the categories, and certainly (by design) the two in which Dartmouth dropped this year, seem skewed to favor research universities. Also the 'Academic Reputation' ranking, which tracks the final ranking more closely than any other, seems skewed towards research-oriented universities.

It seems then, conspicuous that there's no ranking anywhere, that teaching doesn't seem to be taken into statistical account at all.

And that puts institutions like Dartmouth, which pride themselves heavily on the quality of undergraduate teaching, at a disadvantage.

Morse: Well, you're right, there's no direct measurement of teaching quality on this list. There are other places where it comes into play indirectly, especially when we take into account the size of classrooms and the student-teacher ratio, but there's nowhere that comes into direct account. The problem with teaching quality is that its very difficult to measure. I mean, most colleges don't even have any comprehensive, statistically valid, published rankings by students of their own professors, so there's not even a place to start.

TDR: I know that the Princeton Review has consistently ranked quality of teaching. The last time I saw that list was during my senior year of high school, and I remember Dartmouth was first. That's part of what led me to think that a greater emphasis on teaching would help Dartmouth.

Morse: I'm sure it would, but it's too bad Princeton Review does that. I mean, do you know anything about their methodolgy?

TDR: No, is it shoddy?

Morse: I guess shoddy would be a polite term for it. They send a representative to stand in the middle of these campuses, and they hand out a questionaire to the first ten or twelve people that pass. It's anecdotal. There's no statistical validity there. You need a much bigger sample, to account for differences.

US News and World Report has had a few meetings with people from the Pew Foundation in Philadelphia. Pew is trying — and I can only call this really major — to set up an objective system which does a statistically relevent sampling. That would be great to have, but that's still quite a while away.

TDR: So would it be fair to say that US News isn't happy with the weight it gives to quality of teaching and is actively looking to find some better ways to express that?

Morse: Yes. We think the ranking model can be improved.

TDR: US News has consistently put Dartmouth in the 'national universities' category and not the 'liberal arts' category. It seems, though, that Dartmouth has a whole lot more in common with schools like Amherst and Williams that have a few professional schools but are centered on undergraduate education than it does with research monoliths like MIT or the University of Chicago.

Morse: Well, I don't think your President would agree with that. Maybe you don't think of them often, but Dartmouth does have a series of graduate schools - though they are mostly professional schools - that, even if you don't often think about them, are there. Dartmouth's caught in the middle, but I think those graduate and professional programs force Dartmouth into the national universities category. I will say that its probably better to be in the top ten of the national universities ranking than it is to be number one of the liberal arts colleges.

TDR: So any school in the top ten in the university rankings, like Dartmouth, would probably be first in the liberal arts rankings.

Morse: Probably.

The US News and World Report Rankings, Explained

Article Tools

Related Articles