9 ms·

How Reliable Are University Rankings?

65 points by ijustlurk 6y ago

jdmg94 6y ago
I personally think Universities and colleges are a scam that was necessary before we had access to the bulk of human knowledge on the tip of our fingers. We should go back to an apprentice system, go through elementary and middle school and then find a profession you would want to learn OTJ, this would shift the responsibility of training to companies but the people coming out of there would be immediately productive.
tygrak 6y ago
Rankings are a bit dumb, especially because any non english university is going to get ranked terribly basically on every of these rankings. There's no way that the oldest university in central Europe, started in 1348 should be placed 400-500th. That's just a joke.
MR4D 6y ago
The only ranking that matters to the vast majority of people: Median starting salary / median cost to attend. Sure, there are other considerations such as. If it will get you into a better grad school or something, but for most it’s really this simple.
asaph 6y ago
If you only read one sentence from this article, it should be this one: > In this paper, we take a fresh look at this ranking scheme using the public College dataset; we both formally and experimentally show in multiple ways that this ranking scheme is not reliable and cannot be trusted as authoritative because it is too sensitive to weight changes and can easily be gamed.
eindiran 6y ago
Here is the code used by the paper: https://github.com/alidasdan/university-rankings https://github.com/alidasdan/university-rankings
neilv 6y ago
https://www.bostonmagazine.com/news/2014/08/26/how-northeastern-gamed-the-college-rankings/ https://www.bostonmagazine.com/news/2014/08/26/how-northeast... > Freeland swept into Northeastern with a brand-new mantra: recalibrate the school to climb up the ranks. “There’s no question that the system invites gaming,” Freeland tells me. “We made a systematic effort to influence [the outcome].” He directed university researchers to break the U.S. News code and replicate its formulas.
- augustt 6y ago
  Yeah this kind of gaming is probably widespread. UChicago is known for "artificially" driving down their acceptance rate with lots of decision rounds and marketing. Cornell isn't releasing their numbers at all this year - I'd guess because they don't like them.
  - neilv 6y ago
    I've heard many allegations about established universities gaming their ranking by, e.g., soliciting applications widely -- not so much to have a larger and more diverse pool from which to choose -- but to maintain/lower their acceptance rate metric. Two things I like about the Northeastern story are that the official demonstrated even more ambitious gaming, and it was publicized. No criticism of Northeastern; they have many great people, doing great work. And I've heard many faculty objected to the rankings-climbing emphasis as it was happening.
- nu_throwaway 6y ago
  graduated from there. totally agree with what's in that article. at 18 I was swept away by the "allure" of their rapid ascent and aggressive re-development of campus (which has continued to this day) and went there, despite the insane cost. faculty were generally pleasant to interact with, the education was probably average. job prospects were mediocre - alot of competition in boston and NU is in a lower tier, comparatively. as a measure of quality, based on my experience, that ranking is worthless.
  - 6y ago
spideymans 6y ago
Why is there such a fixation on university rankings in the United States? Postsecondary rankings don't seem to matter as much in other Western countries.
- notfromhere 6y ago
  It’s what drives enrollment and tuition fees are a huge part of revenue. In other countries, there’s more state support and you don’t need to market universities
- Infinitesimus 6y ago
  A mix of trying to get a sense of educational quality and prestige. In some specific programs, you'd want to aim for the schools known to have the more robust curriculum but outside of that, it's mostly ego/prestige. "I got to a top N school, I went to harvard", etc. That badge signals _something_ to your peers, parents, and future employers. Think of all those companies that explicitly filter for "target schools". It works pretty well too, you'll get a call-back to an interview because you have an Ivy on your resume even if you barely passed your classes.
- dudul 6y ago
  Education is a market. To make money, universities need to attract customers. And they do that by being highly ranked. And I disagree with your statement. Many other western countries have their "ivy league" equivalent. Not all schools are equal in the rest of the world either. But I do however agree that the "marketing" aspect is definitely more pronounced in the US.
- monadic2 6y ago
  The professional connections pay for the tuition and much more.
  - chasd00 6y ago
    the importance of professional connections are understated in my opinion. You can make a career out of only knowing the right things and you can make a career out of only knowing the right people but you really excel when you know the right things and know the right people.
finolex1 6y ago
Gaming of rankings aside, the ratings organizations themselves offer various 'services' to colleges looking for a reputation boost. https://www.insidehighered.com/news/2013/05/29/methodology-qs-rankings-comes-under-scrutiny https://www.insidehighered.com/news/2013/05/29/methodology-q... For instance, QS sells a 'star-ranking' system for schools. Coincidentally the schools who paid for a '5-star' ranking, such as the Unviersity of Bristol or the Universiti Malaya are also placed higher on the actual QS ranking than those that did not, including Georgia Tech, the University of Washington, Ecole Polytechnique and UIUC.
- alexpetralia 6y ago
  Related to expenses.. "And there are several perverse incentives in the marketplace that make it hard for colleges to cut costs. The most basic one is that the U.S. News algorithm rewards them for spending a lot of money: Higher faculty salaries and more spending on student services lead directly to better rankings. If you reduce your expenses, your ranking will fall, which means that next year your applicant pool will probably shrink." You'd be surprised at the burgeoning expenses of universities, and often meager profitability, despite record-high tuition rates. It seems, often like startups, a burn rate signals potential growth! https://www.nytimes.com/interactive/2019/09/10/magazine/college-admissions-paul-tough.html https://www.nytimes.com/interactive/2019/09/10/magazine/coll...
  - mattkrause 6y ago
    I'm gonna go out on a limb and say that above-market faculty salaries are not a major issue.
    - alexpetralia 6y ago
      I would argue its headcount growth (not salary per employee), but admittedly I could not immediately find data for this. Edit: Also upon looking through the Trinity College financial statements, a huge portion was categorized as "Other", which included consulting fees, travel, interest expenses and various other expenses.
- 6y ago
downerending 6y ago
Regardless, you should think hard before accepting at the "top" university that will take you. Many times it's better to be a big fish in a small pond than vice versa. Also, in terms of learning, it's really about the access you can get to the ten or so percent of the faculty that actually worth anything. Figure out who they are and go after them. Also, network, network, network. Smart or not, everyone you meet in school might give you a reference/job/funding someday. (source: hard experience)
- blendo 6y ago
  > Many times it's better to be a big fish in a small pond than vice versa. Yes, this is seldom considered. For example, if you plan to graduate with a BS in physics, would you rather be in the top quartile at Cal Poly, or the bottom quartile at Cal Tech?
- ordx 6y ago
  Care to elaborate? Did you have any problems after graduating from the "top" university?
  - downerending 6y ago
    More "during" than "after". Not sure if mine counts as "top", but most would say it's easily in the top 25 for my discipline. It was plenty hard for me, though, and in hindsight, I think it would have been better to be the brilliant kid in a lower-ranked school than an also-ran in a higher-ranked school (relative to the range of schools that would admit me). That's just a guess, but I've seen a lot of others suffer from shooting too high.
- jmchuster 6y ago
  I could consider the primary advantage of a "top" university to be its network and the access it gives you to jobs. Namely, companies that do new grad hiring have a specific list of colleges that they target and will source and review resumes from. Not to say that people from other colleges can't get in, but that you'd have to apply directly and have something standout to get the recruiter to even consider you.
qzw 6y ago
From the abstract: *>we both formally and experimentally show in multiple ways that this ranking scheme is not reliable and cannot be trusted as authoritative because it is too sensitive to weight changes and can easily be gamed. That's a feature, not a bug, as far as USNWR (the magazine doing the ranking) is concerned. A more reliable ranking would be more stable and therefore less interesting for the universities. Why pay attention to it if Harvard, Standford, MIT are always 1, 2, 3? It's much more "interesting" and certainly sells more copies/gets more clicks if the rankings have a certain amount of volatility. The fact that it's easily game-able also makes it more attractive to universities because they can do something proactive if they're dissatisfied with their ranking. All these factors combine to make the ranking itself seem more important and prestigious.
- sytelus 6y ago
  Rankings are irrelevant for top 20 but for the rest it matters a lot. Vast majority of population ends up somewhere outside of top 20. They have to chose between A, B and C and rankings plays its part. Reliable ranking is important because it rewards universities with better enrollment (and vice versa).
- musicale 6y ago
  > Standford ^ ;-) Not to be confused with Stamford https://stamford.uconn.edu/ https://stamford.uconn.edu/
- HenryKissinger 6y ago
  > The fact that it's easily game-able also makes it more attractive to universities because they can do something proactive if they're dissatisfied with their ranking Are you saying that there is nothing lower ranked universities could do to improve their ranking in a better, more objective ranking of universities? Because it smacks of elitism.
  - jedberg 6y ago
    It's very similar to sports teams. The tops teams don't vary all that much from year to year. They might change positions a bit, but it's rare for a new team to show up who hasn't shown up before. They essentially have to get lucky and get someone who is super talented into a long term contract, who then attracts additional talent. Or they have to invent a whole new way of doing things, like how the Oakland A's invented a whole new way of recruiting using math (which incidentally no longer helps them because all the top teams adopted it and are back on top again). It's all about network effects.
justlexi93 6y ago
You should never trust any university rankings ever. An education at any level is a multifaceted endeavor and everyone has a different set of requirements to succeed.
skat20phys 6y ago
Sometime not too long ago there was a blog post or article covered on HN, about data showing student scores on some academic achievement measure over time. If I remember, all students generally improved on this measure throughout undergrad, and there was some average tendency for universities with higher average entering standardized test scores to have higher scores, but the effect seemed to be mostly driven by the students. That is, it wasn't that some schools accelerated scores more, it was that they just tended to select for higher-performing students. Also, the signal provided by school, if I remember, was relatively weak. The author was basically arguing that universities in general provide some educational value (or students just learn over time; there was no control of students who didn't go to college) but that the ranking was by and large just a signaling thing. I can't find it now though because all the search terms I can think of just return a bunch of other stuff related to universities and students.
- heyn05tradamu5 6y ago
  Interesting. I wonder how one would measure whether an institution changes student trajectories? Or if that data is available somewhere.
covidacct 6y ago
I'm mostly on board with the critique of university rankings, but I really don't buy this specific critique. It's a strawman. Here's what they did: they took the top research universities and the top liberal arts colleges. They then observed that the rankings can't distinguish between the top institutions within each class, and that you can also flip which of those two classes is preferred. But, in what sense is it even meaningful to compare Amherst College and Harvard [1]? Those are just enormously different types of institutions. It'd be like creating a ranking of "best cars" that includes the top 5 sedans and the top 5 trucks, and then observing that you can jiggle the rankings to get anyone on top. Is a Toyota Corolla better than an F-150? IDK. Stupid question. US News and World Report, for all its problems, does at least get this much right! they break down institutions by "type" and then rank within type. Additionally, just because rankings are noisy and easy to game locally doesn't mean they are inaccurate or easy to game globally. Two institutions within 10-20 slots of one another are probably pretty similar and rankings aren't particularly helpful / are easy to game. But the #100 liberal arts college is probably not as good as Amherst, and the #78 National University is probably not as good as Harvard, and no amount of gaming is going to change that. Rankings are indeed noisy and inaccurate and easy to game. But this particular article is not a compelling demonstration of that fact. [1] For non-US readers: Amherst college (not to be confused with UMass Amherst) is in a class of peculiar institutions that are fairly unique to the USA as described here: https://en.wikipedia.org/wiki/Liberal_arts_college https://en.wikipedia.org/wiki/Liberal_arts_college TL;DR: they're basically the diametric opposite of super-charged research universities like Harvard.
- adas0693 6y ago
  Thank you for your feedback. I am one of the authors of this paper. I don't think the paper gives any hints on us dividing universities into classes and comparing. Would love to know how you reached that conclusion. At the same time, yes, we could have divided universities into different classes: research vs liberal arts, this vs. that state, big vs. small size, etc. These are all trivial groupings but none of these would change the conclusions of this paper. For all practical purposes, we could easily have replaced the university names with labels like U1, U2, ... and still the conclusions would not change. What matters is how a weight-based composite index can be gamed and the paper does show that in multiple ways. Pls review the ILP formulations yourself and run them on the dataset of your choice.
deleted 6y ago
[deleted]
tyingq 6y ago
There is a very current thing happening where individual professors have too much power. The whole "remote teaching" thing is VERY new for some universities and professors. My son recently survived a final where > 90% of the class failed. Mostly due to poor execution of remote learning. Including simple stuff like classes overlapping meeting times, study material uploaded too late, etc. No pressure on the professor or university, though. All on the students. There's a rant letter, from the prof, about effort where effort wasn't a factor. My kid survived solely because he has a very good memory. No accountability. I AM PISSED.
- Aperocky 6y ago
  > where > 90% of the class failed. I can't imagine this happening in a college class where the instructor get to keep his class next year.
  - tyingq 6y ago
    U.S. State college, a fairly well respected one, happening now. Chemical Engineering. They have no idea how to operate "online/remote". And that matters for classes like Fluid Dynamics, Organic Chem, Differential Analysis, etc. Is there some realistic path for me to escalate?
    - deleted 6y ago
      [deleted]
    - covidacct 6y ago
      > Is there some realistic path for me to escalate? Is the instructor an ad junct? Those are cheap and replaceable. They make maybe ~1/10th of what your average FAANG engineer makes because the labor supply is extraordinarily over-saturated. Enough complaints will probably get them fired, if that's what you're looking for, but realize you're probably putting someone with no prospects and no savings out on the street and they will almost certainly be replaced with someone who gives about as many shits as you can expect from someone with a PhD who makes 30k a year. That tuition you're paying isn't flowing to the people teaching the courses. If the instructor is an actual professor, then probably there's nothing you can do. BTW, let your kid at least try their hand at navigating this on their own, even if you provide oversight. Barely competent people doing a poor job is not unique to pandemic times and is definitely not unique to higher ed. Knowing how to use a bureaucracy to route around an incompetent person not doing their job well is a valuable life skill. Plus, everyone involved is going to be more sympathetic to an articulate student than to a "helicopter parent".
thinkloop 6y ago
Any idea what they are measuring against? For example avg. income or accolades of graduates.
dontbenebby 6y ago
I think they're useful-ish, but it's important not to get too focused on specifics. For example I did a stint in policy and I'm reminded of how everyone insisted for a long time that "top" law schools are the "top 14" or T14 for short. This is historically the measure used since Georgetown was in DC, and the children of the elite must be "top tier" regardless of if their school ranked outside top ten on every official metric such as incoming LSATs, publications, etc :D
- barry-cotter 6y ago
  That’s not where T14 comes from. > There exists an informal category known as the "Top Fourteen" or "T14," which refers to the fourteen institutions that regularly claim the top spots in the yearly U.S. News & World Report ranking of American law schools.[8] Furthermore, only these fourteen schools have ever placed within the top ten spots in those rankings.[9] Although "T14" is not a designation used by U.S News itself, the term is "widely known in the legal community."[10] While these schools have seen their position within the top fourteen spots shift frequently, they have generally not placed outside of the top fourteen spots since the inception of the rankings.[11] There have been rare exceptions to this, however, such as UCLA School of Law appearing in the top fourteen instead of Cornell and Northwestern in 1987 and University of Texas School of Law displacing Georgetown in 2018, although the significance of these changes has been debated. https://en.wikipedia.org/wiki/Law_school_rankings_in_the_United_States https://en.wikipedia.org/wiki/Law_school_rankings_in_the_Uni...
  - dontbenebby 6y ago
    >> That’s not where T14 comes from. >There have been rare exceptions to this, however, such as UCLA School of Law appearing in the top fourteen instead of Cornell and Northwestern in 1987 and University of Texas School of Law displacing Georgetown in 2018, although the significance of these changes has been debated. People select measures for a number of reasons. It's perfectly possible the single and only reason T14 is used is for the reasons you lay out. OTOH I have personally had conversations with people who argued that the T14 is a measure of merit when Georgetown was in the T14 then "debated the significance" of the measure when Georgetown dropped out of the top 14 in 2018. It was not clever or endearing.
notechback 6y ago
University rankings have two key issues: 1. They are significantly biased to English language (universities/research output) and bigger countries in general 2. They are totally irrelevant from the student perspective as they mostly focus on research output, which means that you'll get the richest universities and the best professors will be writing papers rather than teaching bachelor level courses. There are some attempts to fix these issues (see eg the EU "Multirank" which allows you to choose your priorities), but it's still hit and miss.
- jseliger 6y ago
  Those are useful issues, to be sure, but there is a lot more incoherence than those two: https://www.newyorker.com/magazine/2011/02/14/the-order-of-things https://www.newyorker.com/magazine/2011/02/14/the-order-of-t...

65 points by ijustlurk 6y ago

I personally think Universities and colleges are a scam that was necessary before we had access to the bulk of human knowledge on the tip of our fingers. We should go back to an apprentice system, go through elementary and middle school and then find a profession you would want to learn OTJ, this would shift the responsibility of training to companies but the people coming out of there would be immediately productive.

tygrak 6y ago

Rankings are a bit dumb, especially because any non english university is going to get ranked terribly basically on every of these rankings. There's no way that the oldest university in central Europe, started in 1348 should be placed 400-500th. That's just a joke.

MR4D 6y ago

The only ranking that matters to the vast majority of people: Median starting salary / median cost to attend. Sure, there are other considerations such as. If it will get you into a better grad school or something, but for most it’s really this simple.

asaph 6y ago

If you only read one sentence from this article, it should be this one: > In this paper, we take a fresh look at this ranking scheme using the public College dataset; we both formally and experimentally show in multiple ways that this ranking scheme is not reliable and cannot be trusted as authoritative because it is too sensitive to weight changes and can easily be gamed.

eindiran 6y ago

Here is the code used by the paper: https://github.com/alidasdan/university-rankings https://github.com/alidasdan/university-rankings

neilv 6y ago

https://www.bostonmagazine.com/news/2014/08/26/how-northeastern-gamed-the-college-rankings/ https://www.bostonmagazine.com/news/2014/08/26/how-northeast... > Freeland swept into Northeastern with a brand-new mantra: recalibrate the school to climb up the ranks. “There’s no question that the system invites gaming,” Freeland tells me. “We made a systematic effort to influence [the outcome].” He directed university researchers to break the U.S. News code and replicate its formulas.

augustt 6y ago
Yeah this kind of gaming is probably widespread. UChicago is known for "artificially" driving down their acceptance rate with lots of decision rounds and marketing. Cornell isn't releasing their numbers at all this year - I'd guess because they don't like them.
- neilv 6y ago
  I've heard many allegations about established universities gaming their ranking by, e.g., soliciting applications widely -- not so much to have a larger and more diverse pool from which to choose -- but to maintain/lower their acceptance rate metric. Two things I like about the Northeastern story are that the official demonstrated even more ambitious gaming, and it was publicized. No criticism of Northeastern; they have many great people, doing great work. And I've heard many faculty objected to the rankings-climbing emphasis as it was happening.
nu_throwaway 6y ago
graduated from there. totally agree with what's in that article. at 18 I was swept away by the "allure" of their rapid ascent and aggressive re-development of campus (which has continued to this day) and went there, despite the insane cost. faculty were generally pleasant to interact with, the education was probably average. job prospects were mediocre - alot of competition in boston and NU is in a lower tier, comparatively. as a measure of quality, based on my experience, that ranking is worthless.
- 6y ago

spideymans 6y ago

Why is there such a fixation on university rankings in the United States? Postsecondary rankings don't seem to matter as much in other Western countries.

notfromhere 6y ago
It’s what drives enrollment and tuition fees are a huge part of revenue. In other countries, there’s more state support and you don’t need to market universities
Infinitesimus 6y ago
A mix of trying to get a sense of educational quality and prestige. In some specific programs, you'd want to aim for the schools known to have the more robust curriculum but outside of that, it's mostly ego/prestige. "I got to a top N school, I went to harvard", etc. That badge signals _something_ to your peers, parents, and future employers. Think of all those companies that explicitly filter for "target schools". It works pretty well too, you'll get a call-back to an interview because you have an Ivy on your resume even if you barely passed your classes.
dudul 6y ago
Education is a market. To make money, universities need to attract customers. And they do that by being highly ranked. And I disagree with your statement. Many other western countries have their "ivy league" equivalent. Not all schools are equal in the rest of the world either. But I do however agree that the "marketing" aspect is definitely more pronounced in the US.
monadic2 6y ago
The professional connections pay for the tuition and much more.
- chasd00 6y ago
  the importance of professional connections are understated in my opinion. You can make a career out of only knowing the right things and you can make a career out of only knowing the right people but you really excel when you know the right things and know the right people.

finolex1 6y ago

Gaming of rankings aside, the ratings organizations themselves offer various 'services' to colleges looking for a reputation boost. https://www.insidehighered.com/news/2013/05/29/methodology-qs-rankings-comes-under-scrutiny https://www.insidehighered.com/news/2013/05/29/methodology-q... For instance, QS sells a 'star-ranking' system for schools. Coincidentally the schools who paid for a '5-star' ranking, such as the Unviersity of Bristol or the Universiti Malaya are also placed higher on the actual QS ranking than those that did not, including Georgia Tech, the University of Washington, Ecole Polytechnique and UIUC.

alexpetralia 6y ago
Related to expenses.. "And there are several perverse incentives in the marketplace that make it hard for colleges to cut costs. The most basic one is that the U.S. News algorithm rewards them for spending a lot of money: Higher faculty salaries and more spending on student services lead directly to better rankings. If you reduce your expenses, your ranking will fall, which means that next year your applicant pool will probably shrink." You'd be surprised at the burgeoning expenses of universities, and often meager profitability, despite record-high tuition rates. It seems, often like startups, a burn rate signals potential growth! https://www.nytimes.com/interactive/2019/09/10/magazine/college-admissions-paul-tough.html https://www.nytimes.com/interactive/2019/09/10/magazine/coll...
- mattkrause 6y ago
  I'm gonna go out on a limb and say that above-market faculty salaries are not a major issue.
  - alexpetralia 6y ago
    I would argue its headcount growth (not salary per employee), but admittedly I could not immediately find data for this. Edit: Also upon looking through the Trinity College financial statements, a huge portion was categorized as "Other", which included consulting fees, travel, interest expenses and various other expenses.
6y ago

downerending 6y ago

Regardless, you should think hard before accepting at the "top" university that will take you. Many times it's better to be a big fish in a small pond than vice versa. Also, in terms of learning, it's really about the access you can get to the ten or so percent of the faculty that actually worth anything. Figure out who they are and go after them. Also, network, network, network. Smart or not, everyone you meet in school might give you a reference/job/funding someday. (source: hard experience)

blendo 6y ago
> Many times it's better to be a big fish in a small pond than vice versa. Yes, this is seldom considered. For example, if you plan to graduate with a BS in physics, would you rather be in the top quartile at Cal Poly, or the bottom quartile at Cal Tech?
ordx 6y ago
Care to elaborate? Did you have any problems after graduating from the "top" university?
- downerending 6y ago
  More "during" than "after". Not sure if mine counts as "top", but most would say it's easily in the top 25 for my discipline. It was plenty hard for me, though, and in hindsight, I think it would have been better to be the brilliant kid in a lower-ranked school than an also-ran in a higher-ranked school (relative to the range of schools that would admit me). That's just a guess, but I've seen a lot of others suffer from shooting too high.
jmchuster 6y ago
I could consider the primary advantage of a "top" university to be its network and the access it gives you to jobs. Namely, companies that do new grad hiring have a specific list of colleges that they target and will source and review resumes from. Not to say that people from other colleges can't get in, but that you'd have to apply directly and have something standout to get the recruiter to even consider you.

qzw 6y ago

From the abstract: *>we both formally and experimentally show in multiple ways that this ranking scheme is not reliable and cannot be trusted as authoritative because it is too sensitive to weight changes and can easily be gamed. That's a feature, not a bug, as far as USNWR (the magazine doing the ranking) is concerned. A more reliable ranking would be more stable and therefore less interesting for the universities. Why pay attention to it if Harvard, Standford, MIT are always 1, 2, 3? It's much more "interesting" and certainly sells more copies/gets more clicks if the rankings have a certain amount of volatility. The fact that it's easily game-able also makes it more attractive to universities because they can do something proactive if they're dissatisfied with their ranking. All these factors combine to make the ranking itself seem more important and prestigious.

sytelus 6y ago
Rankings are irrelevant for top 20 but for the rest it matters a lot. Vast majority of population ends up somewhere outside of top 20. They have to chose between A, B and C and rankings plays its part. Reliable ranking is important because it rewards universities with better enrollment (and vice versa).
musicale 6y ago
> Standford ^ ;-) Not to be confused with Stamford https://stamford.uconn.edu/ https://stamford.uconn.edu/
HenryKissinger 6y ago
> The fact that it's easily game-able also makes it more attractive to universities because they can do something proactive if they're dissatisfied with their ranking Are you saying that there is nothing lower ranked universities could do to improve their ranking in a better, more objective ranking of universities? Because it smacks of elitism.
- jedberg 6y ago
  It's very similar to sports teams. The tops teams don't vary all that much from year to year. They might change positions a bit, but it's rare for a new team to show up who hasn't shown up before. They essentially have to get lucky and get someone who is super talented into a long term contract, who then attracts additional talent. Or they have to invent a whole new way of doing things, like how the Oakland A's invented a whole new way of recruiting using math (which incidentally no longer helps them because all the top teams adopted it and are back on top again). It's all about network effects.

justlexi93 6y ago

You should never trust any university rankings ever. An education at any level is a multifaceted endeavor and everyone has a different set of requirements to succeed.

skat20phys 6y ago

Sometime not too long ago there was a blog post or article covered on HN, about data showing student scores on some academic achievement measure over time. If I remember, all students generally improved on this measure throughout undergrad, and there was some average tendency for universities with higher average entering standardized test scores to have higher scores, but the effect seemed to be mostly driven by the students. That is, it wasn't that some schools accelerated scores more, it was that they just tended to select for higher-performing students. Also, the signal provided by school, if I remember, was relatively weak. The author was basically arguing that universities in general provide some educational value (or students just learn over time; there was no control of students who didn't go to college) but that the ranking was by and large just a signaling thing. I can't find it now though because all the search terms I can think of just return a bunch of other stuff related to universities and students.

heyn05tradamu5 6y ago
Interesting. I wonder how one would measure whether an institution changes student trajectories? Or if that data is available somewhere.

covidacct 6y ago

I'm mostly on board with the critique of university rankings, but I really don't buy this specific critique. It's a strawman. Here's what they did: they took the top research universities and the top liberal arts colleges. They then observed that the rankings can't distinguish between the top institutions within each class, and that you can also flip which of those two classes is preferred. But, in what sense is it even meaningful to compare Amherst College and Harvard [1]? Those are just enormously different types of institutions. It'd be like creating a ranking of "best cars" that includes the top 5 sedans and the top 5 trucks, and then observing that you can jiggle the rankings to get anyone on top. Is a Toyota Corolla better than an F-150? IDK. Stupid question. US News and World Report, for all its problems, does at least get this much right! they break down institutions by "type" and then rank within type. Additionally, just because rankings are noisy and easy to game locally doesn't mean they are inaccurate or easy to game globally. Two institutions within 10-20 slots of one another are probably pretty similar and rankings aren't particularly helpful / are easy to game. But the #100 liberal arts college is probably not as good as Amherst, and the #78 National University is probably not as good as Harvard, and no amount of gaming is going to change that. Rankings are indeed noisy and inaccurate and easy to game. But this particular article is not a compelling demonstration of that fact. [1] For non-US readers: Amherst college (not to be confused with UMass Amherst) is in a class of peculiar institutions that are fairly unique to the USA as described here: https://en.wikipedia.org/wiki/Liberal_arts_college https://en.wikipedia.org/wiki/Liberal_arts_college TL;DR: they're basically the diametric opposite of super-charged research universities like Harvard.

adas0693 6y ago
Thank you for your feedback. I am one of the authors of this paper. I don't think the paper gives any hints on us dividing universities into classes and comparing. Would love to know how you reached that conclusion. At the same time, yes, we could have divided universities into different classes: research vs liberal arts, this vs. that state, big vs. small size, etc. These are all trivial groupings but none of these would change the conclusions of this paper. For all practical purposes, we could easily have replaced the university names with labels like U1, U2, ... and still the conclusions would not change. What matters is how a weight-based composite index can be gamed and the paper does show that in multiple ways. Pls review the ILP formulations yourself and run them on the dataset of your choice.

deleted 6y ago

[deleted]

tyingq 6y ago

There is a very current thing happening where individual professors have too much power. The whole "remote teaching" thing is VERY new for some universities and professors. My son recently survived a final where > 90% of the class failed. Mostly due to poor execution of remote learning. Including simple stuff like classes overlapping meeting times, study material uploaded too late, etc. No pressure on the professor or university, though. All on the students. There's a rant letter, from the prof, about effort where effort wasn't a factor. My kid survived solely because he has a very good memory. No accountability. I AM PISSED.

Aperocky 6y ago
> where > 90% of the class failed. I can't imagine this happening in a college class where the instructor get to keep his class next year.
- tyingq 6y ago
  U.S. State college, a fairly well respected one, happening now. Chemical Engineering. They have no idea how to operate "online/remote". And that matters for classes like Fluid Dynamics, Organic Chem, Differential Analysis, etc. Is there some realistic path for me to escalate?
  - deleted 6y ago
    [deleted]
  - covidacct 6y ago
    > Is there some realistic path for me to escalate? Is the instructor an ad junct? Those are cheap and replaceable. They make maybe ~1/10th of what your average FAANG engineer makes because the labor supply is extraordinarily over-saturated. Enough complaints will probably get them fired, if that's what you're looking for, but realize you're probably putting someone with no prospects and no savings out on the street and they will almost certainly be replaced with someone who gives about as many shits as you can expect from someone with a PhD who makes 30k a year. That tuition you're paying isn't flowing to the people teaching the courses. If the instructor is an actual professor, then probably there's nothing you can do. BTW, let your kid at least try their hand at navigating this on their own, even if you provide oversight. Barely competent people doing a poor job is not unique to pandemic times and is definitely not unique to higher ed. Knowing how to use a bureaucracy to route around an incompetent person not doing their job well is a valuable life skill. Plus, everyone involved is going to be more sympathetic to an articulate student than to a "helicopter parent".

thinkloop 6y ago

Any idea what they are measuring against? For example avg. income or accolades of graduates.

dontbenebby 6y ago

I think they're useful-ish, but it's important not to get too focused on specifics. For example I did a stint in policy and I'm reminded of how everyone insisted for a long time that "top" law schools are the "top 14" or T14 for short. This is historically the measure used since Georgetown was in DC, and the children of the elite must be "top tier" regardless of if their school ranked outside top ten on every official metric such as incoming LSATs, publications, etc :D

barry-cotter 6y ago
That’s not where T14 comes from. > There exists an informal category known as the "Top Fourteen" or "T14," which refers to the fourteen institutions that regularly claim the top spots in the yearly U.S. News & World Report ranking of American law schools.[8] Furthermore, only these fourteen schools have ever placed within the top ten spots in those rankings.[9] Although "T14" is not a designation used by U.S News itself, the term is "widely known in the legal community."[10] While these schools have seen their position within the top fourteen spots shift frequently, they have generally not placed outside of the top fourteen spots since the inception of the rankings.[11] There have been rare exceptions to this, however, such as UCLA School of Law appearing in the top fourteen instead of Cornell and Northwestern in 1987 and University of Texas School of Law displacing Georgetown in 2018, although the significance of these changes has been debated. https://en.wikipedia.org/wiki/Law_school_rankings_in_the_United_States https://en.wikipedia.org/wiki/Law_school_rankings_in_the_Uni...
- dontbenebby 6y ago
  >> That’s not where T14 comes from. >There have been rare exceptions to this, however, such as UCLA School of Law appearing in the top fourteen instead of Cornell and Northwestern in 1987 and University of Texas School of Law displacing Georgetown in 2018, although the significance of these changes has been debated. People select measures for a number of reasons. It's perfectly possible the single and only reason T14 is used is for the reasons you lay out. OTOH I have personally had conversations with people who argued that the T14 is a measure of merit when Georgetown was in the T14 then "debated the significance" of the measure when Georgetown dropped out of the top 14 in 2018. It was not clever or endearing.

notechback 6y ago

University rankings have two key issues: 1. They are significantly biased to English language (universities/research output) and bigger countries in general 2. They are totally irrelevant from the student perspective as they mostly focus on research output, which means that you'll get the richest universities and the best professors will be writing papers rather than teaching bachelor level courses. There are some attempts to fix these issues (see eg the EU "Multirank" which allows you to choose your priorities), but it's still hit and miss.

jseliger 6y ago
Those are useful issues, to be sure, but there is a lot more incoherence than those two: https://www.newyorker.com/magazine/2011/02/14/the-order-of-things https://www.newyorker.com/magazine/2011/02/14/the-order-of-t...