5 ms·

Your rankings of colleges by revealed student preferences seem seriously out of line, especially for MIT and Caltech, compared to the academic paper by Avery et

Your rankings of colleges by revealed student preferences seem seriously out of line, especially for MIT and Caltech, compared to the academic paper by Avery et.al. that you cite on your web-site. They give a ranking of 1. Harvard 2. Caltech 3. Yale 4. MIT 5. Stanford 6. Princeton 7. Brown 8. Columbia 9. Amherst 10. Dartmouth while you give 1. Harvard 2. Yale 3. Stanford 4. Princeton 5. Dartmouth 6. Penn 7. Notre Dame 8. Columbia 9. Georgetown 10. Berkeley (and MIT only #12, and Caltech not even on the list!). For example according to your Elo points (Dartmouth 1767, MIT 1688), Dartmouth would be expected to "win" over MIT in more than 60% of their battles for students (something I have a hard time believing, with all due respect to Dartmouth), while according to Avery et.al.'s paper, MIT has a significant edge over Dartmouth in their competition for students. Based on my anecdotal experiences, the Avery et.al. paper seems to give a much more reasonable account of student preferences. Where are you getting your data?

carbocation 16y ago
Out of line - really? I was initially floored at how similar they were, given the vast differences in survey methodology and time between the two surveys. We even see the same blips (exaggerated preference for Notre Dame, for example) that they see. The Avery data is from 1999. They collected theirs by sending out surveys to dozens of "elite" high schools and asking them to hand the surveys to their top 10% (this is from memory, probably not 100% accurate but close). My data is collected from the 50,000 students who have used my site to track their ~250,000 applications to college -- the bulk of which are from 2009 & 2010. It's not surprising to me that my results look a bit more, shall we say, "democratic" than those from Avery. I say this because of their selection methodology vs ours (elites vs all-comers). Also, it seems that Caltech is seriously over-favored (relative to the preference of current college applicants) in their survey. I say this only based on Caltech's current acceptance rate (17%) and yield (34.4%). Whereas Avery put Caltech above MIT back in 1999, the modern (official) statistics hint at a different picture. Now, MIT still has an extremely impressive 66% yield (and 12% acceptance rate). Their 66% yield trumps Caltech's 34%, especially if you consider geographic rivals such as the entire Ivy league (for MIT) vs, largely, Stanford and Berkeley (for Caltech). (I'm using yield here because it gives a broad-strokes hint of revealed preference when no better data is available. I don't have data from 1999, and Avery don't have data from 2010, so there is no direct way to do this comparison.) I'm not sure that there is much to learn from trying to compare preference for Dartmouth vs MIT, frankly. MIT (perhaps along with Caltech) holds a special place in educational mythology. For quantitative people, it's sacred. For some others, it's probably not even on the radar. I don't know how much direct overlap they have in real life, but on our site we don't even have enough direct cross-admits to make a statement about preference: http://college.mychances.net/college/tools/college-cross-admit-comparison.php?compare=Dartmouth&with=MIT http://college.mychances.net/college/tools/college-cross-adm... Edit: As an aside, Caltech is not on this list because it didn't have enough applicants to give us a fair sense of its ranking. Given that only 4,000 people a year apply there, this wasn't that surprising to me. I think we'll have enough data to rank it this year. Edit 2: If you have any methodological suggestions, I'd absolutely be eager to hear your thoughts.
- jsyedidia 16y ago
  It still seems to me like something is very strange about your MIT ranking; there is probably a bug of some sort. Your own head-to-head MIT vs. Other College pages seem to give results that are inconsistent with your MIT rating of 1688. Assuming a 50% result means insufficient data, here are your results for your top 15; MIT systematically wins significantly more than expected according to your rating. 1. Harvard (Elo 2009) wins 78% (would be expected to win 90+%) 2. Yale -- insufficient data 3. Stanford (Elo 1834) wins 56% (would be expected to win 70+%) 4. Princeton (Elo 1788) wins 44% (would be expected to win 60+%) 5. Dartmouth -- insufficient data 6. Penn (Elo 1757) wins 57% (about right) 7. Notre Dame -- insufficient data 8. Columbia (Elo 1742) wins 33% (would be expected to win 55+%) 9. Georgetown (Elo 1715) wins 40% (would be expected to win 50+%) 10. Berkeley (Elo 1711) wins 20% (would be expected to win 50+%) 11. Brown (Elo 1691) wins 40% (would be expected to win 50+%) 12. MIT 13. Amherst (Elo 1687) wins 40% (would be expected to win 50%) 14. Duke (Elo 1682) wins 33% (would be expected to win nearly 50%) 15. Cornell (Elo 1670) wins 24% (would be expected to win nearly 50%)
  - carbocation 16y ago
    You are right that, given the comparison to our own top 15, MIT's direct matchup numbers seem disproportionate to its Elo points. It is possible that this is bug driven. It is also possible that over the past year we have accumulated enough data that MIT is now clearly more favored by our users than previously (since the head to head matchups are realtime and the Elo points are precomputed, these two approaches are currently using quite different data). If true, this would be reflected by an increased Elo score for MIT when 2010 gets computed. There is also the admittedly odd possibility that MIT does better vs better colleges, though I find this possibility fairly unlikely. I'll look for bugs, as you suggest, since I'm going to be re-running the algorithm again in the coming weeks. Edit: After thinking more about my beliefs about MIT not really being comparable with your average college, I'll add another possibility: perhaps MIT is simply engaged in fewer matches against weaker opponents, collecting fewer "free" points than HYP?
    - robrenaud 16y ago

carbocation 16y ago

Out of line - really? I was initially floored at how similar they were, given the vast differences in survey methodology and time between the two surveys. We even see the same blips (exaggerated preference for Notre Dame, for example) that they see. The Avery data is from 1999. They collected theirs by sending out surveys to dozens of "elite" high schools and asking them to hand the surveys to their top 10% (this is from memory, probably not 100% accurate but close). My data is collected from the 50,000 students who have used my site to track their ~250,000 applications to college -- the bulk of which are from 2009 & 2010. It's not surprising to me that my results look a bit more, shall we say, "democratic" than those from Avery. I say this because of their selection methodology vs ours (elites vs all-comers). Also, it seems that Caltech is seriously over-favored (relative to the preference of current college applicants) in their survey. I say this only based on Caltech's current acceptance rate (17%) and yield (34.4%). Whereas Avery put Caltech above MIT back in 1999, the modern (official) statistics hint at a different picture. Now, MIT still has an extremely impressive 66% yield (and 12% acceptance rate). Their 66% yield trumps Caltech's 34%, especially if you consider geographic rivals such as the entire Ivy league (for MIT) vs, largely, Stanford and Berkeley (for Caltech). (I'm using yield here because it gives a broad-strokes hint of revealed preference when no better data is available. I don't have data from 1999, and Avery don't have data from 2010, so there is no direct way to do this comparison.) I'm not sure that there is much to learn from trying to compare preference for Dartmouth vs MIT, frankly. MIT (perhaps along with Caltech) holds a special place in educational mythology. For quantitative people, it's sacred. For some others, it's probably not even on the radar. I don't know how much direct overlap they have in real life, but on our site we don't even have enough direct cross-admits to make a statement about preference: http://college.mychances.net/college/tools/college-cross-admit-comparison.php?compare=Dartmouth&with=MIT http://college.mychances.net/college/tools/college-cross-adm... Edit: As an aside, Caltech is not on this list because it didn't have enough applicants to give us a fair sense of its ranking. Given that only 4,000 people a year apply there, this wasn't that surprising to me. I think we'll have enough data to rank it this year. Edit 2: If you have any methodological suggestions, I'd absolutely be eager to hear your thoughts.

jsyedidia 16y ago
It still seems to me like something is very strange about your MIT ranking; there is probably a bug of some sort. Your own head-to-head MIT vs. Other College pages seem to give results that are inconsistent with your MIT rating of 1688. Assuming a 50% result means insufficient data, here are your results for your top 15; MIT systematically wins significantly more than expected according to your rating. 1. Harvard (Elo 2009) wins 78% (would be expected to win 90+%) 2. Yale -- insufficient data 3. Stanford (Elo 1834) wins 56% (would be expected to win 70+%) 4. Princeton (Elo 1788) wins 44% (would be expected to win 60+%) 5. Dartmouth -- insufficient data 6. Penn (Elo 1757) wins 57% (about right) 7. Notre Dame -- insufficient data 8. Columbia (Elo 1742) wins 33% (would be expected to win 55+%) 9. Georgetown (Elo 1715) wins 40% (would be expected to win 50+%) 10. Berkeley (Elo 1711) wins 20% (would be expected to win 50+%) 11. Brown (Elo 1691) wins 40% (would be expected to win 50+%) 12. MIT 13. Amherst (Elo 1687) wins 40% (would be expected to win 50%) 14. Duke (Elo 1682) wins 33% (would be expected to win nearly 50%) 15. Cornell (Elo 1670) wins 24% (would be expected to win nearly 50%)
- carbocation 16y ago
  You are right that, given the comparison to our own top 15, MIT's direct matchup numbers seem disproportionate to its Elo points. It is possible that this is bug driven. It is also possible that over the past year we have accumulated enough data that MIT is now clearly more favored by our users than previously (since the head to head matchups are realtime and the Elo points are precomputed, these two approaches are currently using quite different data). If true, this would be reflected by an increased Elo score for MIT when 2010 gets computed. There is also the admittedly odd possibility that MIT does better vs better colleges, though I find this possibility fairly unlikely. I'll look for bugs, as you suggest, since I'm going to be re-running the algorithm again in the coming weeks. Edit: After thinking more about my beliefs about MIT not really being comparable with your average college, I'll add another possibility: perhaps MIT is simply engaged in fewer matches against weaker opponents, collecting fewer "free" points than HYP?
  - robrenaud 16y ago