6 ms·

I would absolutely love to see a double-blind randomized experiment with controls for YC applications. In one batch, randomly and secretly assign CONTROL or EXP

I would absolutely love to see a double-blind randomized experiment with controls for YC applications. In one batch, randomly and secretly assign CONTROL or EXPERIMENT to each application (in whatever proportion you are comfortable with). Of the CONTROL batch, do regular reviews. For the EXPERIMENT batch, randomly accept the same percentage of groups as the control. Then have people outside of the reviewers try to label each group correctly at different stages and see how they do. These people could be: YC alums, YC staff not on the interview loop, VCs making investments, outside experienced CEOs, successful startup founders. For full double-blindness you could even do interviews with the EXPERIMENT group, then use a program to automatically either use the interview result or not. This might be aggravating for reviewers, though, seeing their work wasted by random choices.

jbooth 13y ago
It'd be unfair to great startups in the experiment batch -- not that it's pg's job to make life fair for people, but it'd put a big dent in YC's reputation. Why spend time flying out there if there's a chance your interview doesn't even matter?
- nwhitehead 13y ago
  You could also accept groups with good interviews in the experiment batch, but not include them in the experiment.
  - jbooth 13y ago
    But then they're effectively excluding the best-interviewing groups from the random experiment.
    - jayd16 13y ago
      Can't you pick the best interviews, then hold your experiment with random groups as long as you mix in a the data from the interview groups in such a way to build a representative sample?
- hotpockets 13y ago
  You can still do a pretty similar experiment. For instance, startups 1-50 get into yc like normal. Startups 51-100, together with 50 randomly selected startups get grouped together and tagged as "the bubble group". Meaning that they were on the bubble of getting into yc. Pretty high honors. However, they don't get any access, consultation, etc. to yc itself. Then after a few months you unblind which were highly selected and which were random, to see who does better.
- baddox 13y ago
  We would only be able to conclude that it's unfair for "great start ups" if the experiment showed that random acceptances performed worse than the interview process.
fragsworth 13y ago
How can you really do any experiments on this kind of shit, when the real winning investments exist in front of a zipf curve? You get one or two huge winners out of hundreds, and they are so huge that they are bigger than all the others combined. The investors care most about getting these rare winners, and if YC does any kind of controlled study on who becomes these winners, I don't believe there will be any statistical significance to speak of.
- fleitz 13y ago
  If there is no statistical significance then it means that there is no difference in picking randomly and doing interviews for the rare winners. Since the winners are what they are after they can omit the interview process.
  - dllthomas 13y ago
    That's not what "no statistical significance" means.
    - fleitz 13y ago
      Sorry, statistically speaking the difference in results of interviewing and not interviewing are insignificant. I have a feeling that eventually there will be a Vangaurd of startups that will massively outperform traditional VC/incubators due to reduced management fees similar to index funds vs. regular funds.
      - vasilipupkin 13y ago
        On the one hand, it's probably true that experts overweight their own ability to pick winners. My guess is, PG is likely overestimating the ability of YC staff to actually identify good patterns simply because it's easy to draw conclusions from just a few datapoints but it's hard to draw statistically significant conclusions that hold up out of sample. However, I doubt that a startup index fund will outperform the best VCs and incubators simply because good deal flow is key, whereas in publicly traded markets, everyone has access to all deals. Having said that, I have no doubt that a VC index fund would outperform many of the VCs that are not that good
brudgers 13y ago
If it were just luck then the companies rejected by YC should statistically perform as well as those selected for interviews which should perform as well as those offered spots in a given batch.
- dangoldin 13y ago
  Unless being a part of YC adds value which I suspect it does. Impossible to really test this unless YC would provide the same level of support and resources to other companies.
- jxcole 13y ago
  Sorry, not an expert and I have never been an entrepreneur so forgive me if I am misunderstanding, but doesn't YC give them money? In order for this statement to be true either (1) having startup seed money has no bearing on success or (2) YC gives the same money to people it rejects as to people it accepts.
  - argonaut 13y ago
    But then you're acknowledging that startup success is not solely dependent on luck...
    - baddox 13y ago
      I think you're interpreting the idea of luck incorrectly. Luck in this context is simply the phenomenon of being one of the start ups with high success. To say that success depends solely on luck means that the success of any startup is simply rand(). That doesn't mean all start ups are equally successful.
    - indrax 13y ago
      Giving selected startups money confounds two factors: the selection criteria and the extra funding. This means that any differences in the success rate can't simply be assigned to the selectors choosing wisely.
  - brudgers 13y ago
powera 13y ago
So pg's years of experience getting better than random results is irrelevant and you'll only believe him if he does a long experiment (that would cost him lots of money if it's not just luck) to prove something he already believes?
- scarmig 13y ago
  It's unclear whether he's getting better than random results. You're assuming the consequent. It's well-known that the most successful startup in a group of startups is likely to earn more in profit than all the rest combined. We're here on HN talking about PG because YC has been the most successful of all incubators. However, there's always going to be one incubator that significantly outperforms all the others as a result of the distribution of startup success. I don't know enough about how well PG's investments have done to make informed judgments, but only a very select few people are likely to have that info. I suspect those people have also done their due diligence to figure out if YC is a value-add compared to other incubators (it should be possible to do this without a convoluted double blind experiment), but anyone who's not them really can't say for sure that that's the case.
- baddox 13y ago
  Generally, and especially with science, the way things work is that you prove something before believing it.
  - temujin 13y ago
    This is oversimplified. In many cases, it is the conjunction of a reasonably strong prior belief and the absence of preexisting "proof" that motivates a scientist (or mathematician) to try to prove something in the first place. The key is to perform proper Bayesian updates in the face of evidence in either direction; if you do, as long as your prior wasn't totally insane it doesn't really matter where you started.
  - epsylon 13y ago
    Thankfully for us, business isn't science.
- 13y ago

jbooth 13y ago

It'd be unfair to great startups in the experiment batch -- not that it's pg's job to make life fair for people, but it'd put a big dent in YC's reputation. Why spend time flying out there if there's a chance your interview doesn't even matter?

nwhitehead 13y ago
You could also accept groups with good interviews in the experiment batch, but not include them in the experiment.
- jbooth 13y ago
  But then they're effectively excluding the best-interviewing groups from the random experiment.
  - jayd16 13y ago
    Can't you pick the best interviews, then hold your experiment with random groups as long as you mix in a the data from the interview groups in such a way to build a representative sample?
hotpockets 13y ago
You can still do a pretty similar experiment. For instance, startups 1-50 get into yc like normal. Startups 51-100, together with 50 randomly selected startups get grouped together and tagged as "the bubble group". Meaning that they were on the bubble of getting into yc. Pretty high honors. However, they don't get any access, consultation, etc. to yc itself. Then after a few months you unblind which were highly selected and which were random, to see who does better.
baddox 13y ago
We would only be able to conclude that it's unfair for "great start ups" if the experiment showed that random acceptances performed worse than the interview process.

fragsworth 13y ago

How can you really do any experiments on this kind of shit, when the real winning investments exist in front of a zipf curve? You get one or two huge winners out of hundreds, and they are so huge that they are bigger than all the others combined. The investors care most about getting these rare winners, and if YC does any kind of controlled study on who becomes these winners, I don't believe there will be any statistical significance to speak of.

fleitz 13y ago
If there is no statistical significance then it means that there is no difference in picking randomly and doing interviews for the rare winners. Since the winners are what they are after they can omit the interview process.
- dllthomas 13y ago
  That's not what "no statistical significance" means.
  - fleitz 13y ago
    Sorry, statistically speaking the difference in results of interviewing and not interviewing are insignificant. I have a feeling that eventually there will be a Vangaurd of startups that will massively outperform traditional VC/incubators due to reduced management fees similar to index funds vs. regular funds.
    - vasilipupkin 13y ago
      On the one hand, it's probably true that experts overweight their own ability to pick winners. My guess is, PG is likely overestimating the ability of YC staff to actually identify good patterns simply because it's easy to draw conclusions from just a few datapoints but it's hard to draw statistically significant conclusions that hold up out of sample. However, I doubt that a startup index fund will outperform the best VCs and incubators simply because good deal flow is key, whereas in publicly traded markets, everyone has access to all deals. Having said that, I have no doubt that a VC index fund would outperform many of the VCs that are not that good

brudgers 13y ago

If it were just luck then the companies rejected by YC should statistically perform as well as those selected for interviews which should perform as well as those offered spots in a given batch.

dangoldin 13y ago
Unless being a part of YC adds value which I suspect it does. Impossible to really test this unless YC would provide the same level of support and resources to other companies.
jxcole 13y ago
Sorry, not an expert and I have never been an entrepreneur so forgive me if I am misunderstanding, but doesn't YC give them money? In order for this statement to be true either (1) having startup seed money has no bearing on success or (2) YC gives the same money to people it rejects as to people it accepts.
- argonaut 13y ago
  But then you're acknowledging that startup success is not solely dependent on luck...
  - baddox 13y ago
    I think you're interpreting the idea of luck incorrectly. Luck in this context is simply the phenomenon of being one of the start ups with high success. To say that success depends solely on luck means that the success of any startup is simply rand(). That doesn't mean all start ups are equally successful.
  - indrax 13y ago
    Giving selected startups money confounds two factors: the selection criteria and the extra funding. This means that any differences in the success rate can't simply be assigned to the selectors choosing wisely.
- brudgers 13y ago

powera 13y ago

So pg's years of experience getting better than random results is irrelevant and you'll only believe him if he does a long experiment (that would cost him lots of money if it's not just luck) to prove something he already believes?

scarmig 13y ago
It's unclear whether he's getting better than random results. You're assuming the consequent. It's well-known that the most successful startup in a group of startups is likely to earn more in profit than all the rest combined. We're here on HN talking about PG because YC has been the most successful of all incubators. However, there's always going to be one incubator that significantly outperforms all the others as a result of the distribution of startup success. I don't know enough about how well PG's investments have done to make informed judgments, but only a very select few people are likely to have that info. I suspect those people have also done their due diligence to figure out if YC is a value-add compared to other incubators (it should be possible to do this without a convoluted double blind experiment), but anyone who's not them really can't say for sure that that's the case.
baddox 13y ago
Generally, and especially with science, the way things work is that you prove something before believing it.
- temujin 13y ago
  This is oversimplified. In many cases, it is the conjunction of a reasonably strong prior belief and the absence of preexisting "proof" that motivates a scientist (or mathematician) to try to prove something in the first place. The key is to perform proper Bayesian updates in the face of evidence in either direction; if you do, as long as your prior wasn't totally insane it doesn't really matter where you started.
- epsylon 13y ago
  Thankfully for us, business isn't science.
13y ago