4 ms·

It still seems to me like something is very strange about your MIT ranking; there is probably a bug of some sort. Your own head-to-head MIT vs. Other College pa

• carbocation 16y ago
  You are right that, given the comparison to our own top 15, MIT's direct matchup numbers seem disproportionate to its Elo points. It is possible that this is bug driven. It is also possible that over the past year we have accumulated enough data that MIT is now clearly more favored by our users than previously (since the head to head matchups are realtime and the Elo points are precomputed, these two approaches are currently using quite different data). If true, this would be reflected by an increased Elo score for MIT when 2010 gets computed. There is also the admittedly odd possibility that MIT does better vs better colleges, though I find this possibility fairly unlikely. I'll look for bugs, as you suggest, since I'm going to be re-running the algorithm again in the coming weeks. Edit: After thinking more about my beliefs about MIT not really being comparable with your average college, I'll add another possibility: perhaps MIT is simply engaged in fewer matches against weaker opponents, collecting fewer "free" points than HYP?
  • robrenaud 16y ago
    The nature of Elo should mean that competitions against weak opponents are not free. You should be very likely to win them, yes, but when you lose them, you get penalized harshly for it. It's kind of like betting on a roulette table that has no zeros (eg, no 'house advantage') on every number except for 36. You are likely going to win (probability 35/36), but your winnings are very low (36/35), and so the expected value of the bet is still no gain (35 * 36) * (36 / 35) - 1. Occasionally, you lose all of your money. The really bright kid with the 1550 SAT from the middle class family decides he'd rather take the full ride to state U than go to MIT because the family doesn't really want to pay the tuition. On the other hand, the model just might not fit, and it could be over-predicting rare events, I haven't looked at the data.
    • carbocation 16y ago
      In the Elo system, losing against a lower ranked opponent is most certainly not free. But when you are Harvard and you win ~80% of cross-admit battles, losing only to Stanford, Princeton, Yale, and MIT (other high-Elo-point schools) then battles against lower ranked schools end up being free points. Contrast this with MIT, which, as pointed out above, does lose against lower ranked schools from time to time, bringing down it's ranking despite beating higher ranked schools frequently. In other words, the features that you suggest are critical, and the system already has those features.
• jsyedidia 16y ago
  You said you were interested in methodological ideas, so I'm going to add another comment. Based on the head-to-head rankings of MIT versus other top schools, it looks like MIT should have an Elo rating on your scale somewhere around 1800, which would give it a ranking somewhere around #4, which seems intuitively reasonable. But it is possible that MIT loses more than it should (if it were 1800) against much lower ranked colleges (e.g. Boston University (Elo 1434), which your site gives winning against MIT 28% of the time). What might be happening is that MIT accepts some fraction of students who are weaker academically for a variety of reasons, but those students are intimidated by MIT's reputation for being academically intense, and decide not to attend. My guess is that among strong students (e.g. students with higher SAT scores), MIT's rating would be much higher than among all students. This is also consistent with your suggestion that Avery et.al. used students from elite high schools. So if you want to get ratings that are more in line with what people think of the colleges (e.g. where is a good student who gets into both A and B likely to go) you should probably consider students who have high SATs, or alternatively students who get into a lot of top-ranked colleges, to be more "valuable" and thus to be more important for the rating.
  • carbocation 16y ago
    I like this suggestion, but I wouldn't use it to drive my primary rankings for this reason: I am truly interested in the revealed preference of students. If College X is bending their preference and basically buying kids away from Harvard, then I want to reflect that. If students pick BU over MIT because MIT scares their pants off, I want to reflect that, too. In other words, my aim is to have descriptive, not normative, rankings. However, I like your suggestion and I'll think about tryin to implement a system, in parallel, to give a different set of rankings with the 'feel' that people expect. It should be fun — thanks for the idea!