7 ms·

Ranking of CS departments based on theory papers

41 points by kenjackson 12y ago

nicolasehrhardt 12y ago
"We certainly do not claim that publication record is the best measure, or a complete measure. However, it is the record that everyone sees and compares in CV, for jobs, grants, reports, etc." > That always makes me really sad.
- seanmcdirmid 12y ago
  Signals are hard to come by in acedemic CS.
  - javert 12y ago
    There are no signals of value under socialism. It's a well-known problem in economics. Yes, academic CS is socialism---I am a CS grad student, I know. CS academics are just sucking the public teets instead of trading value for value.
    - seanmcdirmid 12y ago
      I'm an ex-CS grad student; they even gave me a piece of paper called a PhD for my trouble. Academics isn't socialism. There is definitely an aristocracy in there and whether your work is accepted (not just published, but appreciated) or not depends on how you fit into that. The aristocracy serves on grant and program committees; yes they direct money from the state, but that is where the socialism ends and the rat race begins. You'll learn that eventually :)
    - Tloewald 12y ago
      Sounds like as a CS person you've peripherally processed politic science and come up with binary answers to complex ill-posed questions. Academia is not socialism and even communism is not completely lacking in value signals (otherwise it wouldn't be able to deliver excellent medical care, for example). If you're so convinced academia has no value signals it seems rather foolish (or at minimum hypocritical) of you to pursue academic credentials.
      - seanmcdirmid 12y ago
        Just to be clear, just because something isn't optimal doesn't mean its not worth being a part of. You can be critical and not leave at the same time. Actually, the ones who stay are the critical ones who have hopes (founded or not) that the system can improve.
        Tloewald 12y ago
        Having no value signals is a lot worse than being sub-optimal. Everything is sub-optimal.
tptacek 12y ago
It is hard for me to believe that uchicago.edu should outrank illinois.edu. Illinois.edu is a famous top-tier CS program; uchicago.edu is a top-tier university with a relatively small CS program.
- kenjackson 12y ago
  Using papers to rank reduces the impact of reputation, although of course doesn't fully remove it. Also isn't UIUC more of a systems school?
  - tptacek 12y ago
    Still, this seems a bit like Harvard outranking MIT, or Penn outranking CMU.
    - kenjackson 12y ago
      I looked at the Chicago vs UIUC site. Chicago lists 8 theory professors. UIUC lists 7 theory professors. I don't think Chicago Theory > UIUC Theory seems absurd. Certainly not like Penn > CMU.
- kd0amg 12y ago
  That doesn't mean UIUC outclasses Chicago in any given subfield within CS.
  - deleted 12y ago
    [deleted]
- superuser2 12y ago
  UChicago undergrad is ranked #4 in the United States. It's not that surprising.
  - raincom 12y ago
    Undergrand ranking is different from PhD. For instance, yale undergrad >> UCBerkely; however, UCBerkely PhD >>> yale phd program.
akhilcacharya 12y ago
I'm just surprised my CS department made the list. That's nice at least. Even though we are at the bottom...
- jasonjei 12y ago
  Is it NC State? I found the instruction there excellent--at least for theory classes like Automata, Decision And Complexity Theory as well as Data Structures. They offer a real distance education that is run just like an in-person class, with videotaped lectures, projects, and proctored exams. They are one the few universities that offer the full repertoire of bachelor CS classes online, a little known secret. Even though their distance education isn't well marketed, you could complete in theory a BS computer science remotely just through NC State--something that you can't quite yet do with MITx or Udacity--and in addition, their online CS classes do not have an asterisk on your transcript indicating they were taken online, since you are often lumped into a section with the in-person class.
  - kd0amg 12y ago
    The ranking here is about research output, not quality of teaching.
- javert 12y ago
  This list isn't all that meaningful because they have a narrow definition of "CS theory." It's perfectly meaningful of course if you want a ranking based on that narrow definition.
  - akhilcacharya 12y ago
    That's true, but I do think its disappointing we rank so far behind our neighbors (Duke, UNC) in CS in it seems every ranking.
measureit 12y ago
It should be amusing that the two professors that developed this system, Saeed and Mohammad, are both at MIT and University of Maryland, and this ranking bumps UMD up from like #17 in CS to #7. Quite handy...
- shrig94 12y ago
  UMD is a state university--and basically, if you're not UCB, imo state schools suffer from the whole "ivy bias" that society tends to have.
  - me2i81 12y ago
    I'm not sure that's really true in CS. If you're not UCB, Stanford, CMU, or MIT, you're not a "top" CS dept whether public or private. Georgia Tech, U of Washington, UC San Diego, UIUC, UCLA, UT Austin are all credibly somewhere in the top 20.
    - ghc 12y ago
      As someone who spent time at MIT and Georgia Tech, MIT may be a bit better but the difference in quality of labs, resources and faculty is pretty small, with lots of cross pollination between the schools (exe: Thad Starner jumping ship from Media Lab to GT). I don't think you could credibly call GT anything but top tier unless your only resource is Hacker News or you're focusing on one of the few sub-disciplines the College of Computing (yes CS is its own independent college, not merged with EE) doesn't have great coverage of. Now if you combine EE+CS, I might believe you.
      - me2i81 12y ago
        In terms of actual quality, that's probably true. I've worked with people who absolutely won't hire anyone who didn't go to one of the "top 4", and that's the sort of mindset I was addressing. It's not an "ivy only" mindset, it's a "Stanford-Berkeley-MIT-CMU only" (and sometimes not even CMU and Berkeley) mindset.
bottled_poe 12y ago
Quantity is a pretty poor metric. I wonder how closely it corresponds with faculty budget.
- hayesbh 12y ago
  In general I would normally agree with this sentiment, but they're only looking at top tier conference venues. It's very difficult to pass lackluster work through these places, so it really isn't that terrible of a metric. I would be more interested to see what would happen if they only included double-blind conferences, as name recognition in peer review can help make the rich richer. As another commenter said, there are few reliable signals that one could really use for this in general, let alone to get a current ranking. Citations are a fine metric but typically it will take a few years for those to really build up on a paper. I would be very interested to see them expand these rankings beyond theoretical CS and into the other disciplines... particularly robotics, where US News doesn't offer a ranking.
  - mjn 12y ago
    > It's very difficult to pass lackluster work through these places Not really that hard. It's difficult to pass outright bad work through a top-tier conference, but above a quality threshold, it's basically a crapshoot; the reviewing process doesn't reliably separate middling from great papers. The very low acceptance rates (~10-15%) and very high inter-reviewer variance [1] combine to make it pretty random whether a given paper, of sufficiently "ok" quality, gets accepted or not. So the main successful strategy to accumulate a lot of papers in top-tier conferences is simply to submit a lot of decent papers to top-tier conferences. How many such papers a given professor can churn out depends largely on how many students, post-docs, and research scientists they can hire, which depends on funding. [1] NIPS, one of the top-tier machine learning conferences, did a very interesting experimental study of this. http://blog.mrtz.org/2014/12/15/the-nips-experiment.html http://blog.mrtz.org/2014/12/15/the-nips-experiment.html
jessaustin 12y ago
I'm not an academic, but I don't understand the point of ranking both by "1" and "1 + 1/2". 21 schools were ranked identically by each method. 18 were one rank different. At this point we seem to be straining at gnats. Was this really just to mess with Texas A&M?
doktrin 12y ago
Dear MIT : one day, you will receive your deserved comeuppance. Best regards, CMU.
- javert 12y ago
  This is elitism in the bad sense of the word.
  - robmccoll 12y ago
    Meh, reads like harmless humor to me. -no particular allegiance, perhaps Georgia Tech
  - Dewie 12y ago
    What's the good sense?
  - doktrin 12y ago
    You're entitled to that opinion, but it was intended in good fun.
- calinet6 12y ago
  And Berkeley will just be waiting in the wings for the battle to bring you both down. Muahaha.
- akhilcacharya 12y ago
  Dear CMU : We'll get there someday, but I'm not holding my breath. - NCSU
WhitneyLand 12y ago
Is it the case almost none offer a degree online? Regardless of how effective online courses become I worry most top tier schools will always opt out to try and avoid diluting the brand. Scarcity and exclusivity use to be a resource problem that resulted in prestige. Now it seems the prestige must be artificially maintained by limiting the supply to anyone not happy with just a certificate.
- greenpizza13 12y ago
  Georgia Tech, #8, offers an online Master's of CS. I'm enrolled now and it's quite challenging and rewarding.
  - eranation 12y ago
    Same here. Highly recommended. p.s. the tuition for the whole degree is around $7,000. Stanford and Berkley offer an online program for $60K and $50K (the latter is for a data science program).
  - Pyxl101 12y ago
    Would you mind describing a bit about what you learn or study in the program?
mathetic 12y ago
The title needs "in the US" at the end to be more descriptive.
jagger27 12y ago
How do Canadian universities compare?
calinet6 12y ago
Cal >> Stanfurd.
sytelus 12y ago
Huh? So they are just counting papers published in conferences by faculty members as ranking function? That's nuts in 2015. Exactly the reason why this people couldn't have invented PageRank.

41 points by kenjackson 12y ago

"We certainly do not claim that publication record is the best measure, or a complete measure. However, it is the record that everyone sees and compares in CV, for jobs, grants, reports, etc." > That always makes me really sad.

seanmcdirmid 12y ago
Signals are hard to come by in acedemic CS.
- javert 12y ago
  There are no signals of value under socialism. It's a well-known problem in economics. Yes, academic CS is socialism---I am a CS grad student, I know. CS academics are just sucking the public teets instead of trading value for value.
  - seanmcdirmid 12y ago
    I'm an ex-CS grad student; they even gave me a piece of paper called a PhD for my trouble. Academics isn't socialism. There is definitely an aristocracy in there and whether your work is accepted (not just published, but appreciated) or not depends on how you fit into that. The aristocracy serves on grant and program committees; yes they direct money from the state, but that is where the socialism ends and the rat race begins. You'll learn that eventually :)
  - Tloewald 12y ago
    Sounds like as a CS person you've peripherally processed politic science and come up with binary answers to complex ill-posed questions. Academia is not socialism and even communism is not completely lacking in value signals (otherwise it wouldn't be able to deliver excellent medical care, for example). If you're so convinced academia has no value signals it seems rather foolish (or at minimum hypocritical) of you to pursue academic credentials.
    - seanmcdirmid 12y ago
      Just to be clear, just because something isn't optimal doesn't mean its not worth being a part of. You can be critical and not leave at the same time. Actually, the ones who stay are the critical ones who have hopes (founded or not) that the system can improve.
      - Tloewald 12y ago
        Having no value signals is a lot worse than being sub-optimal. Everything is sub-optimal.

tptacek 12y ago

It is hard for me to believe that uchicago.edu should outrank illinois.edu. Illinois.edu is a famous top-tier CS program; uchicago.edu is a top-tier university with a relatively small CS program.

kenjackson 12y ago
Using papers to rank reduces the impact of reputation, although of course doesn't fully remove it. Also isn't UIUC more of a systems school?
- tptacek 12y ago
  Still, this seems a bit like Harvard outranking MIT, or Penn outranking CMU.
  - kenjackson 12y ago
    I looked at the Chicago vs UIUC site. Chicago lists 8 theory professors. UIUC lists 7 theory professors. I don't think Chicago Theory > UIUC Theory seems absurd. Certainly not like Penn > CMU.
kd0amg 12y ago
That doesn't mean UIUC outclasses Chicago in any given subfield within CS.
- deleted 12y ago
  [deleted]
superuser2 12y ago
UChicago undergrad is ranked #4 in the United States. It's not that surprising.
- raincom 12y ago
  Undergrand ranking is different from PhD. For instance, yale undergrad >> UCBerkely; however, UCBerkely PhD >>> yale phd program.

akhilcacharya 12y ago

I'm just surprised my CS department made the list. That's nice at least. Even though we are at the bottom...

jasonjei 12y ago
Is it NC State? I found the instruction there excellent--at least for theory classes like Automata, Decision And Complexity Theory as well as Data Structures. They offer a real distance education that is run just like an in-person class, with videotaped lectures, projects, and proctored exams. They are one the few universities that offer the full repertoire of bachelor CS classes online, a little known secret. Even though their distance education isn't well marketed, you could complete in theory a BS computer science remotely just through NC State--something that you can't quite yet do with MITx or Udacity--and in addition, their online CS classes do not have an asterisk on your transcript indicating they were taken online, since you are often lumped into a section with the in-person class.
- kd0amg 12y ago
  The ranking here is about research output, not quality of teaching.
javert 12y ago
This list isn't all that meaningful because they have a narrow definition of "CS theory." It's perfectly meaningful of course if you want a ranking based on that narrow definition.
- akhilcacharya 12y ago
  That's true, but I do think its disappointing we rank so far behind our neighbors (Duke, UNC) in CS in it seems every ranking.

measureit 12y ago

It should be amusing that the two professors that developed this system, Saeed and Mohammad, are both at MIT and University of Maryland, and this ranking bumps UMD up from like #17 in CS to #7. Quite handy...

shrig94 12y ago
UMD is a state university--and basically, if you're not UCB, imo state schools suffer from the whole "ivy bias" that society tends to have.
- me2i81 12y ago
  I'm not sure that's really true in CS. If you're not UCB, Stanford, CMU, or MIT, you're not a "top" CS dept whether public or private. Georgia Tech, U of Washington, UC San Diego, UIUC, UCLA, UT Austin are all credibly somewhere in the top 20.
  - ghc 12y ago
    As someone who spent time at MIT and Georgia Tech, MIT may be a bit better but the difference in quality of labs, resources and faculty is pretty small, with lots of cross pollination between the schools (exe: Thad Starner jumping ship from Media Lab to GT). I don't think you could credibly call GT anything but top tier unless your only resource is Hacker News or you're focusing on one of the few sub-disciplines the College of Computing (yes CS is its own independent college, not merged with EE) doesn't have great coverage of. Now if you combine EE+CS, I might believe you.
    - me2i81 12y ago
      In terms of actual quality, that's probably true. I've worked with people who absolutely won't hire anyone who didn't go to one of the "top 4", and that's the sort of mindset I was addressing. It's not an "ivy only" mindset, it's a "Stanford-Berkeley-MIT-CMU only" (and sometimes not even CMU and Berkeley) mindset.

bottled_poe 12y ago

Quantity is a pretty poor metric. I wonder how closely it corresponds with faculty budget.

hayesbh 12y ago
In general I would normally agree with this sentiment, but they're only looking at top tier conference venues. It's very difficult to pass lackluster work through these places, so it really isn't that terrible of a metric. I would be more interested to see what would happen if they only included double-blind conferences, as name recognition in peer review can help make the rich richer. As another commenter said, there are few reliable signals that one could really use for this in general, let alone to get a current ranking. Citations are a fine metric but typically it will take a few years for those to really build up on a paper. I would be very interested to see them expand these rankings beyond theoretical CS and into the other disciplines... particularly robotics, where US News doesn't offer a ranking.
- mjn 12y ago
  > It's very difficult to pass lackluster work through these places Not really that hard. It's difficult to pass outright bad work through a top-tier conference, but above a quality threshold, it's basically a crapshoot; the reviewing process doesn't reliably separate middling from great papers. The very low acceptance rates (~10-15%) and very high inter-reviewer variance [1] combine to make it pretty random whether a given paper, of sufficiently "ok" quality, gets accepted or not. So the main successful strategy to accumulate a lot of papers in top-tier conferences is simply to submit a lot of decent papers to top-tier conferences. How many such papers a given professor can churn out depends largely on how many students, post-docs, and research scientists they can hire, which depends on funding. [1] NIPS, one of the top-tier machine learning conferences, did a very interesting experimental study of this. http://blog.mrtz.org/2014/12/15/the-nips-experiment.html http://blog.mrtz.org/2014/12/15/the-nips-experiment.html

jessaustin 12y ago

I'm not an academic, but I don't understand the point of ranking both by "1" and "1 + 1/2". 21 schools were ranked identically by each method. 18 were one rank different. At this point we seem to be straining at gnats. Was this really just to mess with Texas A&M?

doktrin 12y ago

Dear MIT : one day, you will receive your deserved comeuppance. Best regards, CMU.

javert 12y ago
This is elitism in the bad sense of the word.
- robmccoll 12y ago
  Meh, reads like harmless humor to me. -no particular allegiance, perhaps Georgia Tech
- Dewie 12y ago
  What's the good sense?
- doktrin 12y ago
  You're entitled to that opinion, but it was intended in good fun.
calinet6 12y ago
And Berkeley will just be waiting in the wings for the battle to bring you both down. Muahaha.
akhilcacharya 12y ago
Dear CMU : We'll get there someday, but I'm not holding my breath. - NCSU

WhitneyLand 12y ago

Is it the case almost none offer a degree online? Regardless of how effective online courses become I worry most top tier schools will always opt out to try and avoid diluting the brand. Scarcity and exclusivity use to be a resource problem that resulted in prestige. Now it seems the prestige must be artificially maintained by limiting the supply to anyone not happy with just a certificate.

greenpizza13 12y ago
Georgia Tech, #8, offers an online Master's of CS. I'm enrolled now and it's quite challenging and rewarding.
- eranation 12y ago
  Same here. Highly recommended. p.s. the tuition for the whole degree is around $7,000. Stanford and Berkley offer an online program for $60K and $50K (the latter is for a data science program).
- Pyxl101 12y ago
  Would you mind describing a bit about what you learn or study in the program?

mathetic 12y ago

The title needs "in the US" at the end to be more descriptive.

jagger27 12y ago

How do Canadian universities compare?

calinet6 12y ago

Cal >> Stanfurd.

sytelus 12y ago

Huh? So they are just counting papers published in conferences by faculty members as ranking function? That's nuts in 2015. Exactly the reason why this people couldn't have invented PageRank.