3 ms·

One thing that I don't like about the US News college rankings is that they use objective criteria with arbitrary weightings. There's no credible rhyme or reaso

One thing that I don't like about the US News college rankings is that they use objective criteria with arbitrary weightings. There's no credible rhyme or reason why these criteria matter, especially not in the proportion they do. Nor is there a good reason that these weightings should subtly change from year to year. For this reason (plug coming), I created the algorithm that Parchment uses to rank colleges for the Student Choice Rankings [1]. It uses (voluntarily submitted) data from students who got into multiple colleges and chose one to attend. It treats this as a set of games in which one college "wins" and others lose. It's derived from the Elo chess ranking method [2]. As people will no doubt argue, voluntarily submitted data has its limitations. I acknowledge those but for this type of project, sample size is key, and I have yet to see another project with anywhere near the same quantity or recency of data. Perhaps in time that will change. 1 = https://www.parchment.com/c/college/college-rankings.php?page=1&perPage=25&thisYear=2012 https://www.parchment.com/c/college/college-rankings.php?pag... 2 = https://en.wikipedia.org/wiki/Elo_rating_system https://en.wikipedia.org/wiki/Elo_rating_system

milesskorpen 14y ago
It seems like this would weight financial aid rather heavily? Maybe that's a good thing, but doesn't really deal with the quality of the school.
- carbocation 14y ago
  That's correct: anything that makes a college desirable vis-a-vis its competitors will factor in here. We don't attempt to control for that (that's, currently, intentional). Avery, Glickman, Hoxby, and Metrick do control for that, as their goal is to use this to get an isolated notion of "best". For now, I prefer to answer the holistic question of, "Which schools do students prefer?" If we get to a future in which some schools are paying students admitted to Harvard $100k to attend their institution just so they can rank at the top of the Parchment rankings, maybe we'd want to reconsider.
  - ariwilson 14y ago
    We are already in that situation vis a vis schools with merit scholarships vs. Ivy Leagues.
    - milesskorpen 14y ago
      Harvard already gives financial aid to all of its lower-class and most of its middle-class students ... and they define middle class as family income in the $150k range, if I remember right. I don't think merit scholarships are really going to make anyone else directly comparable. Of course, Harvard is rather unusual in this regard. I think only Princeton and a tiny handful of others have a similarly generous policy.
eigenvector 14y ago
Not only are the weightings arbitrary, but many of the criteria bear no relevance to quality of undergraduate education. As an engineering undergrad, why would it matter that my alma mater's teaching hospitals were among the best in the world and raked billions in research grants? Rankings are a terrible way to select schools, not just because they don't even claim to rate quality of education, but also because many top schools explicitly IGNORE any measures of success that aren't included in major ranking schemes, whereas schools with no hope of ever cracking top 20 are forced to craft a different niche for themselves.
carbocation 14y ago
A follow up on ranking strategies. This approach was first used for college rankings by Avery, Glickman, Hoxby, and Metrick. I forgot to acknowledge them in the first post. What we use is an algorithm that says, "For this desirable student (objectively 'desirable' to the schools that were impressed enough to admit them), which school did they choose?" I see this as the "virtuous" approach. In a sense, in this model, schools are rewarded for admitting desirable students and winning the competition for them. This hinders gaming of the system because most things that you can do to game this generally helps students. Want to game it by admitting students who won't get admitted to your competitive peers? You'll fall in the rankings, because those students will pit you against 'weaker' opponents, giving you less of a chance to rise in the rankings even if you always win. There's another ranking approach that is fun but naughty. I'll call it the "vicious" approach. This one takes all colleges that admitted a student and all colleges that rejected them. The ones who rejected the student are modeled as the winners, and the ones who accepted them are modeled as the losers. In this way, the colleges that are most selective in admitting a student rise to the top. I've actually run this "vicious" algorithm, but we don't publish its results. (Gaming this one would hurt students: imagine you're City College. You could do well for yourself by getting all the future freshmen of your rival, State College, to apply to your institution, then rejecting all of them. The kids wasted money on their application, you probably rejected some great candidates just to game the rankings, and State College unfairly drops in the rankings.)
berberous 14y ago
Very interesting, but I think the flaw with this method is that many students will still make decisions based in large part on U.S. News. For example, I would guess that a non-negligible amount of students that get into Harvard (#1) and Columbia (#4) will choose to go to Harvard in large part because they beat Columbia in U.S. News (and thus they perceive it has a better reputation). So despite your interesting methodology, it will still be tied somewhat to the U.S. News rankings.
- carbocation 14y ago
  Yes, I agree that students' perception of how others would rank colleges is a serious factor that probably goes into their decision process. This type of ranking will tend to reflect all of those factors that go into the students' decisions. Using just the data that we and US News make public, one could probably do a pretty interesting adjustment for US News rank. (We publish the Elo points for each college, so you wouldn't be limited to just rank-based methods.)
kolbe 14y ago
I don't think teenagers are the best people to ask when it comes to determining the best institute of higher education. It's no wonder that schools with lax requirements like Stanford and Brown seem to get a boost when kids are the ones choosing.

milesskorpen 14y ago

It seems like this would weight financial aid rather heavily? Maybe that's a good thing, but doesn't really deal with the quality of the school.

carbocation 14y ago
That's correct: anything that makes a college desirable vis-a-vis its competitors will factor in here. We don't attempt to control for that (that's, currently, intentional). Avery, Glickman, Hoxby, and Metrick do control for that, as their goal is to use this to get an isolated notion of "best". For now, I prefer to answer the holistic question of, "Which schools do students prefer?" If we get to a future in which some schools are paying students admitted to Harvard $100k to attend their institution just so they can rank at the top of the Parchment rankings, maybe we'd want to reconsider.
- ariwilson 14y ago
  We are already in that situation vis a vis schools with merit scholarships vs. Ivy Leagues.
  - milesskorpen 14y ago
    Harvard already gives financial aid to all of its lower-class and most of its middle-class students ... and they define middle class as family income in the $150k range, if I remember right. I don't think merit scholarships are really going to make anyone else directly comparable. Of course, Harvard is rather unusual in this regard. I think only Princeton and a tiny handful of others have a similarly generous policy.

eigenvector 14y ago

Not only are the weightings arbitrary, but many of the criteria bear no relevance to quality of undergraduate education. As an engineering undergrad, why would it matter that my alma mater's teaching hospitals were among the best in the world and raked billions in research grants? Rankings are a terrible way to select schools, not just because they don't even claim to rate quality of education, but also because many top schools explicitly IGNORE any measures of success that aren't included in major ranking schemes, whereas schools with no hope of ever cracking top 20 are forced to craft a different niche for themselves.

carbocation 14y ago

A follow up on ranking strategies. This approach was first used for college rankings by Avery, Glickman, Hoxby, and Metrick. I forgot to acknowledge them in the first post. What we use is an algorithm that says, "For this desirable student (objectively 'desirable' to the schools that were impressed enough to admit them), which school did they choose?" I see this as the "virtuous" approach. In a sense, in this model, schools are rewarded for admitting desirable students and winning the competition for them. This hinders gaming of the system because most things that you can do to game this generally helps students. Want to game it by admitting students who won't get admitted to your competitive peers? You'll fall in the rankings, because those students will pit you against 'weaker' opponents, giving you less of a chance to rise in the rankings even if you always win. There's another ranking approach that is fun but naughty. I'll call it the "vicious" approach. This one takes all colleges that admitted a student and all colleges that rejected them. The ones who rejected the student are modeled as the winners, and the ones who accepted them are modeled as the losers. In this way, the colleges that are most selective in admitting a student rise to the top. I've actually run this "vicious" algorithm, but we don't publish its results. (Gaming this one would hurt students: imagine you're City College. You could do well for yourself by getting all the future freshmen of your rival, State College, to apply to your institution, then rejecting all of them. The kids wasted money on their application, you probably rejected some great candidates just to game the rankings, and State College unfairly drops in the rankings.)

berberous 14y ago

Very interesting, but I think the flaw with this method is that many students will still make decisions based in large part on U.S. News. For example, I would guess that a non-negligible amount of students that get into Harvard (#1) and Columbia (#4) will choose to go to Harvard in large part because they beat Columbia in U.S. News (and thus they perceive it has a better reputation). So despite your interesting methodology, it will still be tied somewhat to the U.S. News rankings.

carbocation 14y ago
Yes, I agree that students' perception of how others would rank colleges is a serious factor that probably goes into their decision process. This type of ranking will tend to reflect all of those factors that go into the students' decisions. Using just the data that we and US News make public, one could probably do a pretty interesting adjustment for US News rank. (We publish the Elo points for each college, so you wouldn't be limited to just rank-based methods.)

kolbe 14y ago

I don't think teenagers are the best people to ask when it comes to determining the best institute of higher education. It's no wonder that schools with lax requirements like Stanford and Brown seem to get a boost when kids are the ones choosing.