4 ms·

From the abstract: *>we both formally and experimentally show in multiple ways that this ranking scheme is not reliable and cannot be trusted as authoritative

by qzw 7y ago

From the abstract: *>we both formally and experimentally show in multiple ways that this ranking scheme is not reliable and cannot be trusted as authoritative because it is too sensitive to weight changes and can easily be gamed. That's a feature, not a bug, as far as USNWR (the magazine doing the ranking) is concerned. A more reliable ranking would be more stable and therefore less interesting for the universities. Why pay attention to it if Harvard, Standford, MIT are always 1, 2, 3? It's much more "interesting" and certainly sells more copies/gets more clicks if the rankings have a certain amount of volatility. The fact that it's easily game-able also makes it more attractive to universities because they can do something proactive if they're dissatisfied with their ranking. All these factors combine to make the ranking itself seem more important and prestigious.

sytelus 7y ago
Rankings are irrelevant for top 20 but for the rest it matters a lot. Vast majority of population ends up somewhere outside of top 20. They have to chose between A, B and C and rankings plays its part. Reliable ranking is important because it rewards universities with better enrollment (and vice versa).
musicale 7y ago
> Standford ^ ;-) Not to be confused with Stamford https://stamford.uconn.edu/ https://stamford.uconn.edu/
HenryKissinger 7y ago
> The fact that it's easily game-able also makes it more attractive to universities because they can do something proactive if they're dissatisfied with their ranking Are you saying that there is nothing lower ranked universities could do to improve their ranking in a better, more objective ranking of universities? Because it smacks of elitism.
- jedberg 7y ago
  It's very similar to sports teams. The tops teams don't vary all that much from year to year. They might change positions a bit, but it's rare for a new team to show up who hasn't shown up before. They essentially have to get lucky and get someone who is super talented into a long term contract, who then attracts additional talent. Or they have to invent a whole new way of doing things, like how the Oakland A's invented a whole new way of recruiting using math (which incidentally no longer helps them because all the top teams adopted it and are back on top again). It's all about network effects.
  - cpsempek 7y ago
    Unsure of the exact metric you're referring to, and I'm about to be just as informal, but sports teams swing much more wildly and often than universities. In the NBA, most recently GSW, going from 5 straight finals with 3 wins to last in western conference. See also CHI before and after MJ era. LAL most recently after Kobe's retirement and pre-Lebron. In the NFL, SF post-Young pre-Harbaugh. I agree, there are often dynasties and teams can sit in the tops ranks for quite a while, however it is not at all uncommon to see once top ranked teams vying for lottery draft picks. Maybe the over-all mix does not churn that much, but I can't recall the last time a top 10 school dropped out of the top 100 without a scandal involved.
    - jedberg 7y ago
      I would say sports just moves faster than academics. The average sports career is only a few years with a few exceptions, the average academic career is much longer.
      - ghaff 7y ago
        That's true and, furthermore, a big part of why a university is well-ranked in year N boils down to it being well-ranked in year N-1 and N-2. With a sports team, if you have some critical injuries or otherwise lose important personnel from the prior year or two, no one cares that you won the championship a couple years back if you have a mediocre record this year. There's just a lot more inertia in university rankings both on objective factors and on things that basically depend on being a good university because they're a good university.
- qzw 7y ago
  Well, in a hypothetical ideal ranking system that is stable and not easily gamed, a lower ranked university would need to actually improve the quality of its education (and whatever other characteristics the ranking system measures) in order to move up the ranks. But that would likely to take multiple years or decades of committed effort and investment. It’s sadly rare to see such long term action in our world of instant gratification, whether we’re talking about education, business, government, entertainment, relationships, individual behavior...
- ashtonkem 7y ago
  Nothing easy or quick, no.

by qzw 7y ago

sytelus 7y ago

Rankings are irrelevant for top 20 but for the rest it matters a lot. Vast majority of population ends up somewhere outside of top 20. They have to chose between A, B and C and rankings plays its part. Reliable ranking is important because it rewards universities with better enrollment (and vice versa).

musicale 7y ago

> Standford ^ ;-) Not to be confused with Stamford https://stamford.uconn.edu/ https://stamford.uconn.edu/

HenryKissinger 7y ago

> The fact that it's easily game-able also makes it more attractive to universities because they can do something proactive if they're dissatisfied with their ranking Are you saying that there is nothing lower ranked universities could do to improve their ranking in a better, more objective ranking of universities? Because it smacks of elitism.

jedberg 7y ago
It's very similar to sports teams. The tops teams don't vary all that much from year to year. They might change positions a bit, but it's rare for a new team to show up who hasn't shown up before. They essentially have to get lucky and get someone who is super talented into a long term contract, who then attracts additional talent. Or they have to invent a whole new way of doing things, like how the Oakland A's invented a whole new way of recruiting using math (which incidentally no longer helps them because all the top teams adopted it and are back on top again). It's all about network effects.
- cpsempek 7y ago
  Unsure of the exact metric you're referring to, and I'm about to be just as informal, but sports teams swing much more wildly and often than universities. In the NBA, most recently GSW, going from 5 straight finals with 3 wins to last in western conference. See also CHI before and after MJ era. LAL most recently after Kobe's retirement and pre-Lebron. In the NFL, SF post-Young pre-Harbaugh. I agree, there are often dynasties and teams can sit in the tops ranks for quite a while, however it is not at all uncommon to see once top ranked teams vying for lottery draft picks. Maybe the over-all mix does not churn that much, but I can't recall the last time a top 10 school dropped out of the top 100 without a scandal involved.
  - jedberg 7y ago
    I would say sports just moves faster than academics. The average sports career is only a few years with a few exceptions, the average academic career is much longer.
    - ghaff 7y ago
      That's true and, furthermore, a big part of why a university is well-ranked in year N boils down to it being well-ranked in year N-1 and N-2. With a sports team, if you have some critical injuries or otherwise lose important personnel from the prior year or two, no one cares that you won the championship a couple years back if you have a mediocre record this year. There's just a lot more inertia in university rankings both on objective factors and on things that basically depend on being a good university because they're a good university.
qzw 7y ago
Well, in a hypothetical ideal ranking system that is stable and not easily gamed, a lower ranked university would need to actually improve the quality of its education (and whatever other characteristics the ranking system measures) in order to move up the ranks. But that would likely to take multiple years or decades of committed effort and investment. It’s sadly rare to see such long term action in our world of instant gratification, whether we’re talking about education, business, government, entertainment, relationships, individual behavior...
ashtonkem 7y ago
Nothing easy or quick, no.