4 ms·

I'd be interested in reading something about what they're doing to keep the process working in that case. I'm involved in various academic conferences, and acc

by mjn 14y ago

I'd be interested in reading something about what they're doing to keep the process working in that case. I'm involved in various academic conferences, and acceptance rate is a big sticking point, which gets more problematic as the number gets lower. It's very easy to put on a symposium with a 60-70% acceptance rate: you just filter out the stuff that is clearly not good, and is not going to contribute. It's not much harder to go to 40-50%: you raise your bar for how well thought out something needs to be, how well it needs to ground itself in the existing research, and how good the prose itself needs to be. But once you start going south of that, things quickly get problematic. Now you start rejecting decent stuff, for reasons that depend on random assignment of reviewers: some reviewers are harsher than others, and if you don't correct for it, that impacts things. You have not easily comparable factors: is borderline contribution in area A better or worse than borderline contribution in area B? How precisely do you weight every possible factor? Is interestingness, rigor, or clearness of communication more important? You also, as you get towards 10-20% acceptance rates, easily end up slipping into a mode where papers get accepted more because of "can't find a reason to reject it" rather than "would actually want to accept it". Reviewers become very critical (since most papers need to be rejected), and look for any reason they can find to reject a paper. So what survives is not the best papers, but the papers nobody could find a good enough reason to reject. What that usually means is either 1) airtight incremental work; or 2) blockbuster media-friendly work. More generally, I think it gets exponentially harder to make any kind of review process meaningful as you accept a smaller and smaller percentage of applications. I believe I can honestly set up a screening system for accepting the top 30% of an applicant pool with at least acceptable error. But the top 5%? It starts looking like rolling dice. If Stanford does it better, I'd love to know how!

aneth4 14y ago
Two things: - They are just as happy with a random selection of 7% from the top 15-20% of applications - There is zero "error". They select who they select, and are not just looking for exactly the top applicants - in fact that has little meaning. What does top mean? Is someone with incredible musical talent but slightly weaker than another applicant in math better or worse? They want a solid diverse group of amazing students. This is not angel investing, where they are afraid of missing out on the one big hit.
- just2n 14y ago
  Unless there is a 100% pass rate, there is definitely "error." Picking someone who you couldn't find a problem with but who is incapable of passing your courses counts as picking the wrong person.
  - barry-cotter 14y ago
    The only elite college in the US with a failure rate higher than 2% is Cal Tech. Everyone else provides ample opportunities to switch to less rigorous majors with lower standards. Given that they're admitting 18 year olds with no previous experience of living away from home they're not realisitically going to approach elite professional school levels of graduation unless they make it really easy to pass.
    - duaneb 14y ago
      Lower standards or a better fit? I would find some subjects, like studio art, prohibitively difficult.
    - ohazi 14y ago
      I went to Caltech for undergrad. I know people who got "burnt out" and switched to non-science majors, but the core curriculum would still require everyone (even an English major) to take quantum mechanics, differential equations, etc. They've made some changes to the core curriculum requirements in the last few years, but I think this is still the case.
  - mtpearce 14y ago
    The six year completion rate of the 2006 cohort was 95% (http://ucomm.stanford.edu/cds/2012 http://ucomm.stanford.edu/cds/2012). This rate has been consistent, both over time and at other top schools. Of the 5% that don't complete, it's rare that the student is incapable of completing (aaron swartz, tiger woods, etc). The admissions department will be the first to tell you that they pass on _a lot_ of qualified people. Even though most people believe that they were an 'admissions mistake', I still don't think that 'error' is the right term.
  - 14y ago
daniel-cussen 14y ago
I remember noticing this in high school as I prepared for Stanford's review process. It's not so much the pursuit of greatness as the pursuit of non-failure.
dvt 14y ago
100% agreed. As a matter of fact, just about all schools on the top USNWR rankings are a crap shoot (http://colleges.usnews.rankingsandreviews.com/best-colleges/rankings/national-universities/int+college_rankings_lists/spp+25 http://colleges.usnews.rankingsandreviews.com/best-colleges/...) You can get in one and not get in 20, get in 20 and not get in the one you really wanted to get into. I go to one of those schools and I personally know people that got into Berkeley and wanted to go to UCLA or got into Caltech when they wanted to go to Stanford, or didn't get into the state schools and got into Stanford. A friend of mine wanted to go to MIT for undergrad, but instead got into Cornell (and later went to MIT for grad school.) The undergraduate admission process at elite schools really makes zero sense. All (most?) elite schools use a "holistic" method of evaluating applications. Which is basically a magical black box that spits out yes's and no's.
seanmcdirmid 14y ago
Lets not conflate paper acceptance with college acceptance, the former is based on one piece of work that can be subjectively measured, at least, the latter is more of a crapshoot. I'm a big fan of low acceptance rates in conferences. There are way too many mediocre papers that get into conferences I care about even when the acceptance rate is low. Also, the surge in papers submitted from China has pushed everyone's acceptance rates down, and quality isn't necessarily getting better. Disclaimer: I'm submitting from china these days. In computer science, if you look at low accept conferences like OSDI/SOSP (systems), you'll see some safe (uncontroversial) papers getting accepted but most of them are just good and interesting. Now, compare that to CHI where the accept rate is still competitive but the quantity is huge. The quality there seems much less consistent. When I review for conferences these days, I'll get a stack of papers where one or two are obvious good, one is controversial that we argue over, and 17 get binned after 5 minutes (the rest if the time you spend on the paper is just to give the authors useful feedback).
- mjn 14y ago
  Yeah, I agree there's a big difference. What reminded me of it is that making finer distinctions get more error-prone as you try to take "the best of the best of the best" and add on more "of the bests". CHI is one of the conferences where I think the low acceptance rate tends towards: 1) a lot of randomness in the reviews; and 2) a tendency towards safe "I did a user study and here are p-values" type of work. I agree there are some times when it's easy to accept only 10% or so: when there is a lot of obvious crap submitted. I'm currently organizing a conference with about a 30% acceptance rate, and it's somewhat painful, though. It's not one of the main catch-all conferences like OSDI, CHI, AAAI, SIGGRAPH, etc. which everyone submits to. Instead mostly people in the area submit to it, and a large proportion of the submissions are legitimately good. I would say 50% solid submissions from regular members of the community, 25% not-great submissions from people legitimately in the area, and 25% either crap or clearly off-topic. So it'd be easy to choose which 50% to accept, but when you're accepting 30%, now some good work is being rejected, just because you have to hit the arbitrary acceptance-rate target to count as "selective" in certain U.S. universities' bibliometric systems. If we went down to a 20% acceptance rate (as some people are pushing, not b/c of quality but solely b/c it makes the conference look better bibliometrically), we'd have to fairly arbitrarily reject some more good papers. We half-joke that the way to fix it next year is to spam the CfP to clearly unsuitable places, in order to elicit a flood of crap submissions that we can then easily reject, allowing us to hit a 30% (or 20%, or lower) acceptance rate without having to reject any good papers.
asafira 14y ago
I've talked with professors that do physics graduate school admissions decisions. All in all, they told me that it is not too difficult to get 700-800 applications down to 100-150. Unfortunately, they need to get it down to more like 60-70, and they all agree it's probably not very much better than random from there =/.

by mjn 14y ago

aneth4 14y ago

Two things: - They are just as happy with a random selection of 7% from the top 15-20% of applications - There is zero "error". They select who they select, and are not just looking for exactly the top applicants - in fact that has little meaning. What does top mean? Is someone with incredible musical talent but slightly weaker than another applicant in math better or worse? They want a solid diverse group of amazing students. This is not angel investing, where they are afraid of missing out on the one big hit.

just2n 14y ago
Unless there is a 100% pass rate, there is definitely "error." Picking someone who you couldn't find a problem with but who is incapable of passing your courses counts as picking the wrong person.
- barry-cotter 14y ago
  The only elite college in the US with a failure rate higher than 2% is Cal Tech. Everyone else provides ample opportunities to switch to less rigorous majors with lower standards. Given that they're admitting 18 year olds with no previous experience of living away from home they're not realisitically going to approach elite professional school levels of graduation unless they make it really easy to pass.
  - duaneb 14y ago
    Lower standards or a better fit? I would find some subjects, like studio art, prohibitively difficult.
  - ohazi 14y ago
    I went to Caltech for undergrad. I know people who got "burnt out" and switched to non-science majors, but the core curriculum would still require everyone (even an English major) to take quantum mechanics, differential equations, etc. They've made some changes to the core curriculum requirements in the last few years, but I think this is still the case.
- mtpearce 14y ago
  The six year completion rate of the 2006 cohort was 95% (http://ucomm.stanford.edu/cds/2012 http://ucomm.stanford.edu/cds/2012). This rate has been consistent, both over time and at other top schools. Of the 5% that don't complete, it's rare that the student is incapable of completing (aaron swartz, tiger woods, etc). The admissions department will be the first to tell you that they pass on _a lot_ of qualified people. Even though most people believe that they were an 'admissions mistake', I still don't think that 'error' is the right term.
- 14y ago

daniel-cussen 14y ago

I remember noticing this in high school as I prepared for Stanford's review process. It's not so much the pursuit of greatness as the pursuit of non-failure.

dvt 14y ago

100% agreed. As a matter of fact, just about all schools on the top USNWR rankings are a crap shoot (http://colleges.usnews.rankingsandreviews.com/best-colleges/rankings/national-universities/int+college_rankings_lists/spp+25 http://colleges.usnews.rankingsandreviews.com/best-colleges/...) You can get in one and not get in 20, get in 20 and not get in the one you really wanted to get into. I go to one of those schools and I personally know people that got into Berkeley and wanted to go to UCLA or got into Caltech when they wanted to go to Stanford, or didn't get into the state schools and got into Stanford. A friend of mine wanted to go to MIT for undergrad, but instead got into Cornell (and later went to MIT for grad school.) The undergraduate admission process at elite schools really makes zero sense. All (most?) elite schools use a "holistic" method of evaluating applications. Which is basically a magical black box that spits out yes's and no's.

seanmcdirmid 14y ago

Lets not conflate paper acceptance with college acceptance, the former is based on one piece of work that can be subjectively measured, at least, the latter is more of a crapshoot. I'm a big fan of low acceptance rates in conferences. There are way too many mediocre papers that get into conferences I care about even when the acceptance rate is low. Also, the surge in papers submitted from China has pushed everyone's acceptance rates down, and quality isn't necessarily getting better. Disclaimer: I'm submitting from china these days. In computer science, if you look at low accept conferences like OSDI/SOSP (systems), you'll see some safe (uncontroversial) papers getting accepted but most of them are just good and interesting. Now, compare that to CHI where the accept rate is still competitive but the quantity is huge. The quality there seems much less consistent. When I review for conferences these days, I'll get a stack of papers where one or two are obvious good, one is controversial that we argue over, and 17 get binned after 5 minutes (the rest if the time you spend on the paper is just to give the authors useful feedback).

mjn 14y ago
Yeah, I agree there's a big difference. What reminded me of it is that making finer distinctions get more error-prone as you try to take "the best of the best of the best" and add on more "of the bests". CHI is one of the conferences where I think the low acceptance rate tends towards: 1) a lot of randomness in the reviews; and 2) a tendency towards safe "I did a user study and here are p-values" type of work. I agree there are some times when it's easy to accept only 10% or so: when there is a lot of obvious crap submitted. I'm currently organizing a conference with about a 30% acceptance rate, and it's somewhat painful, though. It's not one of the main catch-all conferences like OSDI, CHI, AAAI, SIGGRAPH, etc. which everyone submits to. Instead mostly people in the area submit to it, and a large proportion of the submissions are legitimately good. I would say 50% solid submissions from regular members of the community, 25% not-great submissions from people legitimately in the area, and 25% either crap or clearly off-topic. So it'd be easy to choose which 50% to accept, but when you're accepting 30%, now some good work is being rejected, just because you have to hit the arbitrary acceptance-rate target to count as "selective" in certain U.S. universities' bibliometric systems. If we went down to a 20% acceptance rate (as some people are pushing, not b/c of quality but solely b/c it makes the conference look better bibliometrically), we'd have to fairly arbitrarily reject some more good papers. We half-joke that the way to fix it next year is to spam the CfP to clearly unsuitable places, in order to elicit a flood of crap submissions that we can then easily reject, allowing us to hit a 30% (or 20%, or lower) acceptance rate without having to reject any good papers.

asafira 14y ago

I've talked with professors that do physics graduate school admissions decisions. All in all, they told me that it is not too difficult to get 700-800 applications down to 100-150. Unfortunately, they need to get it down to more like 60-70, and they all agree it's probably not very much better than random from there =/.