7 ms·

An Investigation of the Facts Behind Columbia's U.S. News Ranking

266 points by jxding 5y ago

relaxing 5y ago
I couldn’t find his data set available, so his assertion about terminal degrees is curious to me. He seems to consider Ph.D. the only terminal degree, and specifically calls out the Arts school for having faculty with Master’s. But in most fields in the Arts, an MFA (though usually not an MA) is terminal.
robinjhuang 5y ago
This is an amazing takedown by one of Columbia’s own professors. Wow
- readthenotes1 5y ago
  [flagged]
  - ryan93 5y ago
    Columbia should not be ranked 2nd in US news college rankings
    - gargarplex 5y ago
      Maybe rankings should be organized through preferences of applicants admitted to several schools.
      - ryan93 5y ago
        i was answering his question
  - bee_rider 5y ago
    For a TLDR, skip to the conclusion section. Generally for a paper if you don't actually care to see the author back up their arguments (which is not a great way of fully ingesting a paper, but lots of things don't need to be fully ingested) you can just read the intro and the conclusion. > §8: CONCLUSION > No one should try to reform or rehabilitate the ranking. It is irredeemable. In Colin Diver’s memorable formulation, “Trying to rank institutions of higher education is a little like trying to rank religions or philosophies. The entire enterprise is flawed, not only in detail but also in conception.” Got to appreciate an essayist who'll enter the conclusion section guns a'blazing.
  - MandieD 5y ago
    For an academic paper, it was really easy to read, and I say that as someone who has noticed that my own attention span has become alarmingly short.
- lofatdairy 5y ago
  I can't speak for the math department, but I've noticed that there are general contentions between Columbia's faculty and administration that seem somewhat unique to the school. Professor Martin Chalfie has noted how cramped and run-down his lab is in an interview, which seems like soft-spoken criticism at the way Columbia has failed to use its enormous endowment towards acquiring more research and teaching spaces. Perhaps someone at Harvard or MIT can chime in with whether such contentions exist outside of Columbia, but I'm not too surprised to see a professor criticize the school for somewhat blatant wrongdoing.
  - ericd 5y ago
    I can tell you that the student body at MIT generally had a pretty dim view of the admins when I was there, always trying to kill the things that make MIT an amazing place, while building stupidly overpriced vanity buildings. Not to mention the incredibly shameful way they handled the Aaron Swartz case.
    - sokoloff 5y ago
      I think there were a mix of cases when I was there, with admins sometimes supporting the wonderful weird and other times not. Often when they did, it was because the weird was only a background level of annoying/risky and when they didn’t, it was often that the level of cacophony had exceeded that admin’s tolerance for it.
      - Throwawayaerlei 5y ago
        It got worse: while the admissions office and museum were using MIT hacking culture to attract applicants and advance its brand, the school changed both the type of officers hired for the campus police which is necessary due to its inner city location and started arresting students for attempting to pull off harmless hacks. I also heard some reports they got generally more aggressive towards the student population. When I was there they hired older, very experienced police officers who were looking for a good, not normally intense final job for their career. So they were generally understanding and fit into the culture. Aaron Swartz was an entirely different thing. Instead of hammering JSTOR from his own institution of Harvard to the point it blocked the whole campus, he disguised himself demonstrating mens rea, a guilty mind, let himself into a machine room and left behind the laptop to do his act of civil disobedience or whatever. And then proved chronically depressed people have no business committing such crimes. From beginning to end he demonstrated ill intent towards MIT.
deleted 5y ago
[deleted]
rr808 5y ago
I hope he has tenure. :)
- b215826 5y ago
  He is a full professor and this article, although critical of Columbia's push to improve its ranking, is dry and lacks hot-button topics.
  - zucker42 5y ago
    The article comes as close as possible to accusing Columbia of fraud without outright saying it. I hardly think that's dry.
    - barnbuilder 5y ago
      Not only are they apparently lying to US News, they appear to be running a scheme of enrolling an unusually high amount of transfer students (who are not counted in the US News graduation rates) for the purpose of soaking them for tuition revenue. Per the article, transfers get much less financial aid than non-transfers at Columbia and also have much lower graduation rates, which is the opposite of what is seen at other schools with large transfer populations like in the UC system. Prof. Thaddeus is very courageous for bringing these facts to light, and/or has a deep love for his university that he feels is being corrupted by their pursuit of a higher ranking.
tandymodel100 5y ago
colinmhayes 5y ago
I took a class with Northwestern's last president Morton Schapiro. He admitted that the school lies about standardized test numbers in one of the lectures. Can't say it was too surprising. It seems like every school in the top 50 is claiming that at least 25% of their students get 34+ on the act.
- alisonkisk 5y ago
data4lyfe 5y ago
I think U.S. News Ranking only really has merit because it's been around for a long time and has also done a good job of transferring its brand into good SEO.
wolverine876 5y ago
General rankings have limited use. The value of data depends on the interests of the reader, which for college 'rankings' vary widely. For example, if you are interested only in the status of your college degree among the public, then the ranking itself over time and the credibility of US News are probably most valuable, regardless of how they are calculated. But if you are interested in the status of your degree within academia then the 'peer assessment survey' seems most important. Your interests and thus the value of different data will vary by what you want to do (study? work? donate? evaluate research? hire researchers or teachers?), what field(s) you are interested in and at what level, how you learn/work, etc. For many people, the 'ranking' might rely almost completely on one department or just a few specific faculty members. The US News & World Report's ranking system apparently depends (at least in part) on class size (8% of ranking), proportion of faculty with terminal degrees (3% of ranking), proportion of faculty who are full-time (1% of ranking), student-faculty ratio (1%), financial resources per student (10%), retention and graduation rates (35%), student debt (5%), and "'peer assessment survey' [20%] in which college presidents, provosts, and admissions deans are asked to rate other institutions." All that data can be valuable but says little directly about the quality of education received by students or the quality of research, though the latter might be outside USN's scope. Sometimes, objectivity is a distraction because the objective data is too limited to inform us. For example, objective data about a play - number of words, etc. - doesn't tell you much about it. In those cases, expert judgment is an excellent tool and I think USN's 'peer assessment survey' is the right approach for many purposes, but USN surveys people with narrow knowlede - administrators, including admissions officers. I might look at Times Higher Education, which performs a 'reputation survey' that asks (IIRC) thousands of published, tenured faculty about the quality of other schools' departments in their field of expertise. These are people with expertise and significant access, though of course their understanding of the student's perspective will be limited. (Columbia rates around 10th, IIRC.)
- dehrmann 5y ago
  > retention and graduation rates (35%) That's a weird metric for judging education quality (which is at least what college rankings purport to do). I can see cases where either high or low values are markers of either good or bad education quality.
lowpro 5y ago
For the TLDR people: “ In each previous section, what was at issue was a discrepancy between two figures, both obtained from data provided by Columbia. Regarding class sizes, the information provided to U.S. News conflicts with the information in the Directory of Classes. Regarding terminal degrees, the information provided to U.S. News conflicts with the information in the Columbia College Bulletin. Regarding full-time faculty, the information provided to U.S. News conflicts with the information provided to the Department of Education. And so on.” The big conclusions are that Columbia seems to be providing inaccurate data, and that one of the outcomes of chasing rankings are that transfer students end up as second class students, at least at Columbia. I think it’s a data driven case of how elite universities can perpetuate a system of reduced social mobility. For Columbia, the objectively more poor transfer students support the more wealthy non-transfer students, and the graduation rate shows that disparity.
- gnicholas 5y ago
  > one of the outcomes of chasing rankings are that transfer students end up as second class students, at least at Columbia. Interesting — when I was in law school, transfer students benefitted from the fact that they avoided our harsh first-year grading curves (20% A's, 60% B's, and 20% C's). The curve for second- and third-year classes was much more lenient, and it was relatively easy to get a 3.5 average during those years. As a result, the transfer students disproportionately ended up with the highest GPAs, which seemed unfair to those of us who were there all 3 years.
pmyteh 5y ago
Leaving aside the suggestions of fraud, this is also a good demonstration of Goodhart's Law: "A measure that becomes a target becomes a bad measure". There are good reasons to believe that, for example, selectivity of admissions or graduation rates are some kind of quality signal. But as soon as it becomes strategically important to 'score well' on those metrics it becomes well worth universities doing things that will improve the metric without improving the quality - such as rejecting qualified applicants or graduating those who ought to have failed. And now the quality signal is much less useful if you're simply trying to use it to understand what's going on.
btheshoe 5y ago
I go to washu - which does decently well on the rankings. There's a tradition every fall for the incoming freshman class to read a selected book, and my year they chose Cathy oNiel's weapons of math destruction, which goes over instances of data being used for bad outcomes, including a chapter devoted to how gaming the university ranking system had become common practice. I found it a very interesting choice on the part of the uni.
- krisoft 5y ago
  It is important to remember that universities, like most organisations, are not monolithic entities. It is almost certain that the person who selects the book for the common reading program is not the same person who is responsible to optimise the university’s ranking.
deleted 5y ago
[deleted]
7402 5y ago
"As someone who knew a little math, what really drove me bonkers about the college guide was: (a) the logical absurdity of adding together completely unrelated statistics to produce a single measure of merit — the key point being that you can produce an astonishing range of different results depending on the relative weight each component factor is assigned. And there is simply no logical, a priori basis for establishing such a weighting objectively. Do SAT scores count 30% of the total score? 32.2%? 18.78234%? (How about zero?) It's the classic apples + oranges – bananas/kumquats = fruit salad approach to statistics, and is completely meaningless." https://budiansky.blogspot.com/2012/02/us-news-root-of-all-evil.html https://budiansky.blogspot.com/2012/02/us-news-root-of-all-e...
setgree 5y ago
NYT coverage: https://www.nytimes.com/2022/03/17/us/columbia-university-rank.html https://www.nytimes.com/2022/03/17/us/columbia-university-ra...

266 points by jxding 5y ago

I couldn’t find his data set available, so his assertion about terminal degrees is curious to me. He seems to consider Ph.D. the only terminal degree, and specifically calls out the Arts school for having faculty with Master’s. But in most fields in the Arts, an MFA (though usually not an MA) is terminal.

robinjhuang 5y ago

This is an amazing takedown by one of Columbia’s own professors. Wow

readthenotes1 5y ago
[flagged]
- ryan93 5y ago
  Columbia should not be ranked 2nd in US news college rankings
  - gargarplex 5y ago
    Maybe rankings should be organized through preferences of applicants admitted to several schools.
    - ryan93 5y ago
      i was answering his question
- bee_rider 5y ago
  For a TLDR, skip to the conclusion section. Generally for a paper if you don't actually care to see the author back up their arguments (which is not a great way of fully ingesting a paper, but lots of things don't need to be fully ingested) you can just read the intro and the conclusion. > §8: CONCLUSION > No one should try to reform or rehabilitate the ranking. It is irredeemable. In Colin Diver’s memorable formulation, “Trying to rank institutions of higher education is a little like trying to rank religions or philosophies. The entire enterprise is flawed, not only in detail but also in conception.” Got to appreciate an essayist who'll enter the conclusion section guns a'blazing.
- MandieD 5y ago
  For an academic paper, it was really easy to read, and I say that as someone who has noticed that my own attention span has become alarmingly short.
lofatdairy 5y ago
I can't speak for the math department, but I've noticed that there are general contentions between Columbia's faculty and administration that seem somewhat unique to the school. Professor Martin Chalfie has noted how cramped and run-down his lab is in an interview, which seems like soft-spoken criticism at the way Columbia has failed to use its enormous endowment towards acquiring more research and teaching spaces. Perhaps someone at Harvard or MIT can chime in with whether such contentions exist outside of Columbia, but I'm not too surprised to see a professor criticize the school for somewhat blatant wrongdoing.
- ericd 5y ago
  I can tell you that the student body at MIT generally had a pretty dim view of the admins when I was there, always trying to kill the things that make MIT an amazing place, while building stupidly overpriced vanity buildings. Not to mention the incredibly shameful way they handled the Aaron Swartz case.
  - sokoloff 5y ago
    I think there were a mix of cases when I was there, with admins sometimes supporting the wonderful weird and other times not. Often when they did, it was because the weird was only a background level of annoying/risky and when they didn’t, it was often that the level of cacophony had exceeded that admin’s tolerance for it.
    - Throwawayaerlei 5y ago
      It got worse: while the admissions office and museum were using MIT hacking culture to attract applicants and advance its brand, the school changed both the type of officers hired for the campus police which is necessary due to its inner city location and started arresting students for attempting to pull off harmless hacks. I also heard some reports they got generally more aggressive towards the student population. When I was there they hired older, very experienced police officers who were looking for a good, not normally intense final job for their career. So they were generally understanding and fit into the culture. Aaron Swartz was an entirely different thing. Instead of hammering JSTOR from his own institution of Harvard to the point it blocked the whole campus, he disguised himself demonstrating mens rea, a guilty mind, let himself into a machine room and left behind the laptop to do his act of civil disobedience or whatever. And then proved chronically depressed people have no business committing such crimes. From beginning to end he demonstrated ill intent towards MIT.

deleted 5y ago

[deleted]

rr808 5y ago

I hope he has tenure. :)

b215826 5y ago
He is a full professor and this article, although critical of Columbia's push to improve its ranking, is dry and lacks hot-button topics.
- zucker42 5y ago
  The article comes as close as possible to accusing Columbia of fraud without outright saying it. I hardly think that's dry.
  - barnbuilder 5y ago
    Not only are they apparently lying to US News, they appear to be running a scheme of enrolling an unusually high amount of transfer students (who are not counted in the US News graduation rates) for the purpose of soaking them for tuition revenue. Per the article, transfers get much less financial aid than non-transfers at Columbia and also have much lower graduation rates, which is the opposite of what is seen at other schools with large transfer populations like in the UC system. Prof. Thaddeus is very courageous for bringing these facts to light, and/or has a deep love for his university that he feels is being corrupted by their pursuit of a higher ranking.

tandymodel100 5y ago

colinmhayes 5y ago

I took a class with Northwestern's last president Morton Schapiro. He admitted that the school lies about standardized test numbers in one of the lectures. Can't say it was too surprising. It seems like every school in the top 50 is claiming that at least 25% of their students get 34+ on the act.

alisonkisk 5y ago

data4lyfe 5y ago

I think U.S. News Ranking only really has merit because it's been around for a long time and has also done a good job of transferring its brand into good SEO.

wolverine876 5y ago

General rankings have limited use. The value of data depends on the interests of the reader, which for college 'rankings' vary widely. For example, if you are interested only in the status of your college degree among the public, then the ranking itself over time and the credibility of US News are probably most valuable, regardless of how they are calculated. But if you are interested in the status of your degree within academia then the 'peer assessment survey' seems most important. Your interests and thus the value of different data will vary by what you want to do (study? work? donate? evaluate research? hire researchers or teachers?), what field(s) you are interested in and at what level, how you learn/work, etc. For many people, the 'ranking' might rely almost completely on one department or just a few specific faculty members. The US News & World Report's ranking system apparently depends (at least in part) on class size (8% of ranking), proportion of faculty with terminal degrees (3% of ranking), proportion of faculty who are full-time (1% of ranking), student-faculty ratio (1%), financial resources per student (10%), retention and graduation rates (35%), student debt (5%), and "'peer assessment survey' [20%] in which college presidents, provosts, and admissions deans are asked to rate other institutions." All that data can be valuable but says little directly about the quality of education received by students or the quality of research, though the latter might be outside USN's scope. Sometimes, objectivity is a distraction because the objective data is too limited to inform us. For example, objective data about a play - number of words, etc. - doesn't tell you much about it. In those cases, expert judgment is an excellent tool and I think USN's 'peer assessment survey' is the right approach for many purposes, but USN surveys people with narrow knowlede - administrators, including admissions officers. I might look at Times Higher Education, which performs a 'reputation survey' that asks (IIRC) thousands of published, tenured faculty about the quality of other schools' departments in their field of expertise. These are people with expertise and significant access, though of course their understanding of the student's perspective will be limited. (Columbia rates around 10th, IIRC.)

dehrmann 5y ago
> retention and graduation rates (35%) That's a weird metric for judging education quality (which is at least what college rankings purport to do). I can see cases where either high or low values are markers of either good or bad education quality.

lowpro 5y ago

For the TLDR people: “ In each previous section, what was at issue was a discrepancy between two figures, both obtained from data provided by Columbia. Regarding class sizes, the information provided to U.S. News conflicts with the information in the Directory of Classes. Regarding terminal degrees, the information provided to U.S. News conflicts with the information in the Columbia College Bulletin. Regarding full-time faculty, the information provided to U.S. News conflicts with the information provided to the Department of Education. And so on.” The big conclusions are that Columbia seems to be providing inaccurate data, and that one of the outcomes of chasing rankings are that transfer students end up as second class students, at least at Columbia. I think it’s a data driven case of how elite universities can perpetuate a system of reduced social mobility. For Columbia, the objectively more poor transfer students support the more wealthy non-transfer students, and the graduation rate shows that disparity.

gnicholas 5y ago
> one of the outcomes of chasing rankings are that transfer students end up as second class students, at least at Columbia. Interesting — when I was in law school, transfer students benefitted from the fact that they avoided our harsh first-year grading curves (20% A's, 60% B's, and 20% C's). The curve for second- and third-year classes was much more lenient, and it was relatively easy to get a 3.5 average during those years. As a result, the transfer students disproportionately ended up with the highest GPAs, which seemed unfair to those of us who were there all 3 years.

pmyteh 5y ago

Leaving aside the suggestions of fraud, this is also a good demonstration of Goodhart's Law: "A measure that becomes a target becomes a bad measure". There are good reasons to believe that, for example, selectivity of admissions or graduation rates are some kind of quality signal. But as soon as it becomes strategically important to 'score well' on those metrics it becomes well worth universities doing things that will improve the metric without improving the quality - such as rejecting qualified applicants or graduating those who ought to have failed. And now the quality signal is much less useful if you're simply trying to use it to understand what's going on.

btheshoe 5y ago

I go to washu - which does decently well on the rankings. There's a tradition every fall for the incoming freshman class to read a selected book, and my year they chose Cathy oNiel's weapons of math destruction, which goes over instances of data being used for bad outcomes, including a chapter devoted to how gaming the university ranking system had become common practice. I found it a very interesting choice on the part of the uni.

krisoft 5y ago
It is important to remember that universities, like most organisations, are not monolithic entities. It is almost certain that the person who selects the book for the common reading program is not the same person who is responsible to optimise the university’s ranking.

deleted 5y ago

[deleted]

7402 5y ago

"As someone who knew a little math, what really drove me bonkers about the college guide was: (a) the logical absurdity of adding together completely unrelated statistics to produce a single measure of merit — the key point being that you can produce an astonishing range of different results depending on the relative weight each component factor is assigned. And there is simply no logical, a priori basis for establishing such a weighting objectively. Do SAT scores count 30% of the total score? 32.2%? 18.78234%? (How about zero?) It's the classic apples + oranges – bananas/kumquats = fruit salad approach to statistics, and is completely meaningless." https://budiansky.blogspot.com/2012/02/us-news-root-of-all-evil.html https://budiansky.blogspot.com/2012/02/us-news-root-of-all-e...

setgree 5y ago

NYT coverage: https://www.nytimes.com/2022/03/17/us/columbia-university-rank.html https://www.nytimes.com/2022/03/17/us/columbia-university-ra...