4 ms·

Hey Sam, semi technical question here. NIPS, ran an experiments a few years ago where for 10% of the papers they had two completely independent teams of review

by Fede_V 11y ago

Hey Sam, semi technical question here. NIPS, ran an experiments a few years ago where for 10% of the papers they had two completely independent teams of reviewers assess papers and score them - the agreement between the teams was very low: http://blog.mrtz.org/2014/12/15/the-nips-experiment.html http://blog.mrtz.org/2014/12/15/the-nips-experiment.html (barely better than chance, actually). Would you consider doing something similar for YC? Obviously, you already have multiple people looking at websites/applications, but I'd be really curious to see what the agreement is like - and you have some absolutely exception AI/data scientists people at openAI who could crunch all the numbers.

sama 11y ago
We have multiple people review each application (except the very good/very bad ones) and we look at the cases where the assessments are very different. Some of the best companies have the most disagreement, which is why we have implemented our "strong yes" policy--if any single partner wants to fund a company, we fund it. It's much more important to our returns that we fund the next Airbnb than if we mistakenly fund one more bad company.
- jackpirate 11y ago
  This is a really uninformative answer. Every peer reviewed conference/journal also does this. The NIPS study was special because they did more. They ran a controlled experiment to determine the exact variance in acceptance rates and then published the results. The GP was asking you to repeat the NIPS experiment.
  - andkon 11y ago
    If any partner can unilaterally say yes to any application, it makes the NIPS study irrelevant to their problem.
  - namenotrequired 11y ago
    They're in different business. NIPS needs to minimise the amount of really bad papers they accept. YC needs to maximise the amount of really good companies they fund.
- komali2 11y ago
  Who makes the first top-level judgement of "this is very good" or "this is very bad?"

by Fede_V 11y ago

sama 11y ago

We have multiple people review each application (except the very good/very bad ones) and we look at the cases where the assessments are very different. Some of the best companies have the most disagreement, which is why we have implemented our "strong yes" policy--if any single partner wants to fund a company, we fund it. It's much more important to our returns that we fund the next Airbnb than if we mistakenly fund one more bad company.

jackpirate 11y ago
This is a really uninformative answer. Every peer reviewed conference/journal also does this. The NIPS study was special because they did more. They ran a controlled experiment to determine the exact variance in acceptance rates and then published the results. The GP was asking you to repeat the NIPS experiment.
- andkon 11y ago
  If any partner can unilaterally say yes to any application, it makes the NIPS study irrelevant to their problem.
- namenotrequired 11y ago
  They're in different business. NIPS needs to minimise the amount of really bad papers they accept. YC needs to maximise the amount of really good companies they fund.
komali2 11y ago
Who makes the first top-level judgement of "this is very good" or "this is very bad?"