6 ms·

The Man Who Knows Whether Any Startup Will Live or Die

47 points by InternetGiant 12y ago

dia80 12y ago
The data mining bias / model risk here is huge. Let's see how he gets on truly out of sample.
JonoBB 12y ago
Retroactively fitting a model to prove history is not that difficult. Accurately predicting the future may be a bit trickier. Just ask technical stock traders.
- darkmighty 12y ago
  Well it depends on proper cross-validation, sample size and of course predictability (i.e. 4 * P(Suc.|data) * P(Fail.|data)<<1). But online prediction is of course what will tell the long term reliability of any model.
- math 12y ago
  public markets will be much more information efficient than this.
sickpig 12y ago
from the article: He admits the models will never be perfect, but thinks that even a model that’s only right about 50 percent of the time could help investors and entrepreneurs avoid particularly bad ideas dunno why but I can't stop thinking that tossing a coin could achieve the same goal :)
- joshyeager 12y ago
  That only works if startups fail 50% of the time. If they fail more frequently, a coin flip would be wrong.
  - sickpig 12y ago
    I stand corrected, you're right. I'm supposed to know better.
    - jgeralnik 12y ago
      Nope, you're still right. Assume startups fail 90% of the time. 50% of the time your coin comes up tails, and you claim the startup will fail. 50% of the time your coin comes up heads and you claim the startup will succeed. You guess correctly 0.5 (the chance your coin comes up tails) * 0.9 (the chance the startup fails) + 0.5 (heads) * 0.1 (success) = 0.5 of the time.
  - kylebrown 12y ago
    Its not clear from the article which factor the 50% figure applies. I doubt its the overall odds ratio, but rather the likelihood given some priors. For example, given a prior that 90% of startups fail, what is the likelihood that this particular startup will succeed? If the prior is already factored in, then the model predicts no better than a coin flip. The point made in the article is that investing at the odds of a coin flip would be better than investing with incorrect risk assumptions (ie. buying into "particularly bad ideas").
- crdb 12y ago
  1% * (+500,000,000) + 5% * (+5,000,000) + 94% * (-100,000) >> 50% * (+100,000) + 50% * (-100,000)
colanderman 12y ago
> According to the U.S. Bureau of Labor Statistics, about half of all businesses fail within five years. > He [...] thinks that even a model that’s only right about 50 percent of the time could help investors and entrepreneurs avoid particularly bad ideas ...does he have a bridge to sell me too? What am I missing? (One can simply predict "always succeeds" and will be right half the time.)
- adwf 12y ago
  Because the current rule of thumb is that ~9/10 startups fail. If you can reduce that to a 50/50 bet, you've made quite an improvement.
  - mojuba 12y ago
    Not really. What the article says is, the model would predict 50% of time whether a company will fail or not, which doesn't make sense, because 50% for a binary prediction (i.e. fail or not) is exactly nothing. So maybe it's just bad or confusing wording in the article, the guy actually meant to say something else.
    - adwf 12y ago
      Yeah I'm going with confusing wording, I think he meant what I said above. Not to mention he could be referring to eliminating false positives, which is slightly different to finding true positives versus true negatives.
    - mox1 12y ago
      I think it's easier to relate to a coin flip if we use an "unfair coin". 90% of the time the coin flip returns tails (aka fail). 10% of the time it returns heads (aka win). For a given coin flip, their algorithm can predict the results 50% of the time. At this point I don't remember the calculations off the top of my head, but it involves a Binomial distribution.
      - RogerL 12y ago
pp19dd 12y ago
"The Machine That Won the War."
Simp 12y ago
>> Using this process, he discovered some surprising things—most notably that a company’s team is only about 12 percent predictive of a company’s success. “You need to find a good team that won’t ruin the company, but hiring ‘rock stars’ isn’t that great,” he explains. The market the company is entering is far more important than who’s running the company. Makes you wonder why we're paying these people so much money.
gtirloni 12y ago
> even a model that’s only right about 50 percent of the time could help investors That's equivalent to a coin flip. I didn't get the point he was trying to make. 66% looks good though.
zupa-hu 12y ago
Too bad that the very best startups are always the outliars (aka without historical data).
- bluedevil2k 12y ago
  Do have any data to back that up? It seems a blanket statement to call "the best" start-ups outliers. What defines "the best" startup?
  - zupa-hu 12y ago
    As the topic was about investments, in the context the best means whichever makes more money I guess. You are right, I have no data to back that up.
michaelochurch 12y ago
Ignoring the already-discussed issue of overfitting, I'm going to discuss something that surprises many, but shouldn't. Using this process, he discovered some surprising things—most notably that a company’s team is only about 12 percent predictive of a company’s success. I'm surprised that it's even that high. In the real world (e.g. outside of the VC-funded world) the team is important, because they're going to grow the company from scratch, and that requires getting more things right than wrong over 5+ years. On the other hand, if you're Snapchat, you have several investor-level people working to protect the company from its founder-quality problems and its own worst impulses, and the company will IPO or be bought before its cultural rot reaches a critical point. The Valley's founder-quality problem creates a lot of awful corporate cultures, and it has dragged the status of engineers way down, and it's generally been bad for the world... but it doesn't kill businesses because the investors are able to keep enough of them on track to produce successes. The influence of VC "rocket fuel" and guidance is why a company can have terrible founders and still succeed.
- mojuba 12y ago
  There is a factor at play here, I think, and it's the age. Many investors are simply older and are more experienced individuals. I'm not even sure if their VC status matters here any more than their age. (In a rather comical example, two youngsters want to put together another chat app with cool new smiley icons, then two VCs step in and ask: how are you going to monetize on smileys?) Investors should definitely be viewed as part of the team - one, and two, age should also be taken into account. I wonder if Growth Science considers this as well.
tarkofski 12y ago
"He admits the models will never be perfect, but thinks that even a model that’s only right about 50 percent of the time could help investors and entrepreneurs avoid particularly bad ideas that, to the untrained eye, look like excellent opportunities." He basically admits his model is no better than a monkey
- falcolas 12y ago
  > He basically admits his model is no better than a monkey Well, technically, a monkey with a coin to flip. "Heads this startup will succeed, tails it will fail"
  - gerhardi 12y ago
    Now coin flip wouldn't be anything near 50% accurate, unless there is a world where more than a small fraction of startups survive!
    - deleted 12y ago
      [deleted]
    - cousin_it 12y ago
      A coinflip is always 50% accurate, no matter how improbable the event you're trying to predict. If I flip a coin to predict whether Cthulhu will rise tomorrow, I have a 50% chance of getting it right.
    - Hasu 12y ago
      What? A coin flip would still be 50% accurate. If 10% of startups succeed, and we say that heads is "succeed" and tails is "fail": 5% of startups will be predicted to succeed and will succeed (correct prediction) 5% of startups will be predicted to fail and will succeed (false prediction) 45% of startups will be predicted to succeed and will fail(false prediction) 45% of startups will be predicted to fail and will fail (true prediction) 50% true predictions, 50% false predictions. A coin flip is always 50% accurate.
- jeffwass 12y ago
darkFunction 12y ago
I don't understand what the inputs to the model are.
normloman 12y ago
I wish this guy the best in improving his algorithm. If it really worked, it could do a lot of good. But the economy is so complex, I doubt he'll ever make the model more accurate than a coin toss. I predict the model will just make people overconfident in their investment decisions.
- TeMPOraL 12y ago
  Investing is anti-inductive; if his algorithm actually starts to be used in investment decisions, people will keep gaming it until it will no longer be a useful signal.
  - kylebrown 12y ago
    Reflexive is the term used by George Soros: http://en.wikipedia.org/wiki/Reflexivity_(social_theory)#In_economics http://en.wikipedia.org/wiki/Reflexivity_(social_theory)#In_...
krampian 12y ago
title: "The Man Who Knows Whether Any Startup Will Live or Die" text: ""He admits the models will never be perfect, but thinks that even a model that’s only right about 50 percent of the time could help investors and entrepreneurs..." Which is not surprising... if the title was really true, this man would likely be richer than Warren Buffett at this point.
- rimantas 12y ago
  Wouldn't a coin toss be right 50% of the time?
mojuba 12y ago
I wonder if they use Bayesian logic, because they should. I also suspect that even though the article does not disclose a lot, the major factor at play in their model is the market sizes. I'm pretty sure their online version would heavily rely on the industry/segment you select. In other words it's a business-plan-looking-good approach which in today's rapidly changing world becomes less and less relevant. So I'll remain skeptical about it.
- Someone1234 12y ago
  > I wonder if they use Bayesian logic, because they should. Looking at his qualifications, I think it is safe to say he too took statistics 101.
claypoolb 12y ago
The output of his model is contrarian to what every accelerator and VC firm in the Valley says about investing. They believe the team is the most important factor of success. Thurston says its 12%. We love pushing the envelope!
ballpoint 12y ago
I'm not sure having better predictors of what businesses are good is necessarily a good idea. Part of the attraction of silicon valley is that it takes some of the risk out of trying new things, even if they might be bad ideas. This culture of trying things leads us to find the occasional really good idea. If we sit around all day plugging our ideas into models to see if, statistically speaking, the will succeed, we won't find the really novel ideas that look bad but are actually good.
Fede_V 12y ago
I would be incredibly interested to know how the model was crossvalidated. It is utterly trivial to cherry pick features post-facto to correctly predict winners, however people who are unfamiliar with machine learning might not know this.
skmurphy 12y ago
I wonder if he has applied the algorithim to his firm to improve his chances of survival
- agarden 12y ago
  Yes. From the article: "According to Thurston’s own model, Growth Science’s own chance of survival following its current business model is about 69 percent. Adding the automated service would actually improve its chances, he says."
thurstont 12y ago
Hi Y'all, Thomas here (guy in article). Just want to start by saying (1) this is a very intelligent thread, and (2) I didn't write the article, was just interviewed for it. You never know what's going to be written, no matter what you say. Here's how the models really play out. We compare our accuracy against the 10 year survivorship benchmark of 25% (not the 5 year). When you look at small businesses, VC-backed, and corporate ventures (ex. new products coming out of companies), the 10 year survival rate is around 25%, plus or minus 10% depending on the industry. Our models have made thousands of predictions for around nine years now - all the predictions were live, real-time and forward looking (no back-testing included here). From those predictions, around 3,400 have matured to date. That is, only around 3,400 of the results have happened - the businesses have either become big successes (ex. Uber) or failed. In our research, we have to actually wait for businesses to live or die to test our accuracy. From the roughly 3,400 predictions that have matured, we were right 66% of the time when predicting survivors, and 88% of the time when predicting failures. When we scratched beneath the surface, we were really around 66% accurate in both cases (just most businesses fail, which is why gloomy predictions were 22% more accurate - just a function of dumb luck since most things die). So we consider our algorithms to be 66% accurate, which is much more accurate than anything we're aware of in human history (remember, the baseline we're compared against is 25%). If you do a statistical analysis (to make sure our predictions weren't just luck), the models maintained a statistically significant correlation with 99% confidence. There was less than 1 chance in over 500,000 that the results were a function of luck (definitely not a coin toss). We've used these models in venture, and our performance puts us in the top 5% of all VC funds for our vintage years, so we've monetized these models effectively with real dollars and made considerable gains. I hope this gives folks a better sense for how it works. There's been a very emotional backlash to the Wired article today (not accusing this thread, just thinking of some others) and it's weird because it's just basic scientific research. Pretty drab stuff on most days, but apparently offensive to some people. Not sure why. We're using statistics to improve venture and startup odds, just like stats have been used to improve just about every other field humans have ever taken seriously. Seems obvious that stats are similarly useful in the startup world, and my dream has always been to help more businesses use stats to succeed. Anyway, definitely a lot more controversy and emotion than I would have expected. Otherwise pretty basic science, not claiming perfection, just striving for improvement, using data as best we can, etc. I hope at least some folks see this for what it is - nothing out of the ordinary in any other domain of science. Why should entrepreneurship be any different?
bbody 12y ago
Misleading title but an interesting article nonetheless. The comments here really show how much statistics are misunderstood by the public, even by a more technical-minded crowd.

47 points by InternetGiant 12y ago

dia80 12y ago

The data mining bias / model risk here is huge. Let's see how he gets on truly out of sample.

Retroactively fitting a model to prove history is not that difficult. Accurately predicting the future may be a bit trickier. Just ask technical stock traders.

darkmighty 12y ago
Well it depends on proper cross-validation, sample size and of course predictability (i.e. 4 * P(Suc.|data) * P(Fail.|data)<<1). But online prediction is of course what will tell the long term reliability of any model.
math 12y ago
public markets will be much more information efficient than this.

sickpig 12y ago

from the article: He admits the models will never be perfect, but thinks that even a model that’s only right about 50 percent of the time could help investors and entrepreneurs avoid particularly bad ideas dunno why but I can't stop thinking that tossing a coin could achieve the same goal :)

joshyeager 12y ago
That only works if startups fail 50% of the time. If they fail more frequently, a coin flip would be wrong.
- sickpig 12y ago
  I stand corrected, you're right. I'm supposed to know better.
  - jgeralnik 12y ago
    Nope, you're still right. Assume startups fail 90% of the time. 50% of the time your coin comes up tails, and you claim the startup will fail. 50% of the time your coin comes up heads and you claim the startup will succeed. You guess correctly 0.5 (the chance your coin comes up tails) * 0.9 (the chance the startup fails) + 0.5 (heads) * 0.1 (success) = 0.5 of the time.
- kylebrown 12y ago
  Its not clear from the article which factor the 50% figure applies. I doubt its the overall odds ratio, but rather the likelihood given some priors. For example, given a prior that 90% of startups fail, what is the likelihood that this particular startup will succeed? If the prior is already factored in, then the model predicts no better than a coin flip. The point made in the article is that investing at the odds of a coin flip would be better than investing with incorrect risk assumptions (ie. buying into "particularly bad ideas").
crdb 12y ago
1% * (+500,000,000) + 5% * (+5,000,000) + 94% * (-100,000) >> 50% * (+100,000) + 50% * (-100,000)

colanderman 12y ago

> According to the U.S. Bureau of Labor Statistics, about half of all businesses fail within five years. > He [...] thinks that even a model that’s only right about 50 percent of the time could help investors and entrepreneurs avoid particularly bad ideas ...does he have a bridge to sell me too? What am I missing? (One can simply predict "always succeeds" and will be right half the time.)

adwf 12y ago
Because the current rule of thumb is that ~9/10 startups fail. If you can reduce that to a 50/50 bet, you've made quite an improvement.
- mojuba 12y ago
  Not really. What the article says is, the model would predict 50% of time whether a company will fail or not, which doesn't make sense, because 50% for a binary prediction (i.e. fail or not) is exactly nothing. So maybe it's just bad or confusing wording in the article, the guy actually meant to say something else.
  - adwf 12y ago
    Yeah I'm going with confusing wording, I think he meant what I said above. Not to mention he could be referring to eliminating false positives, which is slightly different to finding true positives versus true negatives.
  - mox1 12y ago
    I think it's easier to relate to a coin flip if we use an "unfair coin". 90% of the time the coin flip returns tails (aka fail). 10% of the time it returns heads (aka win). For a given coin flip, their algorithm can predict the results 50% of the time. At this point I don't remember the calculations off the top of my head, but it involves a Binomial distribution.
    - RogerL 12y ago

pp19dd 12y ago

"The Machine That Won the War."

Simp 12y ago

>> Using this process, he discovered some surprising things—most notably that a company’s team is only about 12 percent predictive of a company’s success. “You need to find a good team that won’t ruin the company, but hiring ‘rock stars’ isn’t that great,” he explains. The market the company is entering is far more important than who’s running the company. Makes you wonder why we're paying these people so much money.

gtirloni 12y ago

> even a model that’s only right about 50 percent of the time could help investors That's equivalent to a coin flip. I didn't get the point he was trying to make. 66% looks good though.

zupa-hu 12y ago

Too bad that the very best startups are always the outliars (aka without historical data).

bluedevil2k 12y ago
Do have any data to back that up? It seems a blanket statement to call "the best" start-ups outliers. What defines "the best" startup?
- zupa-hu 12y ago
  As the topic was about investments, in the context the best means whichever makes more money I guess. You are right, I have no data to back that up.

michaelochurch 12y ago

Ignoring the already-discussed issue of overfitting, I'm going to discuss something that surprises many, but shouldn't. Using this process, he discovered some surprising things—most notably that a company’s team is only about 12 percent predictive of a company’s success. I'm surprised that it's even that high. In the real world (e.g. outside of the VC-funded world) the team is important, because they're going to grow the company from scratch, and that requires getting more things right than wrong over 5+ years. On the other hand, if you're Snapchat, you have several investor-level people working to protect the company from its founder-quality problems and its own worst impulses, and the company will IPO or be bought before its cultural rot reaches a critical point. The Valley's founder-quality problem creates a lot of awful corporate cultures, and it has dragged the status of engineers way down, and it's generally been bad for the world... but it doesn't kill businesses because the investors are able to keep enough of them on track to produce successes. The influence of VC "rocket fuel" and guidance is why a company can have terrible founders and still succeed.

mojuba 12y ago
There is a factor at play here, I think, and it's the age. Many investors are simply older and are more experienced individuals. I'm not even sure if their VC status matters here any more than their age. (In a rather comical example, two youngsters want to put together another chat app with cool new smiley icons, then two VCs step in and ask: how are you going to monetize on smileys?) Investors should definitely be viewed as part of the team - one, and two, age should also be taken into account. I wonder if Growth Science considers this as well.

tarkofski 12y ago

"He admits the models will never be perfect, but thinks that even a model that’s only right about 50 percent of the time could help investors and entrepreneurs avoid particularly bad ideas that, to the untrained eye, look like excellent opportunities." He basically admits his model is no better than a monkey

falcolas 12y ago
> He basically admits his model is no better than a monkey Well, technically, a monkey with a coin to flip. "Heads this startup will succeed, tails it will fail"
- gerhardi 12y ago
  Now coin flip wouldn't be anything near 50% accurate, unless there is a world where more than a small fraction of startups survive!
  - deleted 12y ago
    [deleted]
  - cousin_it 12y ago
    A coinflip is always 50% accurate, no matter how improbable the event you're trying to predict. If I flip a coin to predict whether Cthulhu will rise tomorrow, I have a 50% chance of getting it right.
  - Hasu 12y ago
    What? A coin flip would still be 50% accurate. If 10% of startups succeed, and we say that heads is "succeed" and tails is "fail": 5% of startups will be predicted to succeed and will succeed (correct prediction) 5% of startups will be predicted to fail and will succeed (false prediction) 45% of startups will be predicted to succeed and will fail(false prediction) 45% of startups will be predicted to fail and will fail (true prediction) 50% true predictions, 50% false predictions. A coin flip is always 50% accurate.
jeffwass 12y ago

darkFunction 12y ago

I don't understand what the inputs to the model are.

normloman 12y ago

I wish this guy the best in improving his algorithm. If it really worked, it could do a lot of good. But the economy is so complex, I doubt he'll ever make the model more accurate than a coin toss. I predict the model will just make people overconfident in their investment decisions.

TeMPOraL 12y ago
Investing is anti-inductive; if his algorithm actually starts to be used in investment decisions, people will keep gaming it until it will no longer be a useful signal.
- kylebrown 12y ago
  Reflexive is the term used by George Soros: http://en.wikipedia.org/wiki/Reflexivity_(social_theory)#In_economics http://en.wikipedia.org/wiki/Reflexivity_(social_theory)#In_...

krampian 12y ago

title: "The Man Who Knows Whether Any Startup Will Live or Die" text: ""He admits the models will never be perfect, but thinks that even a model that’s only right about 50 percent of the time could help investors and entrepreneurs..." Which is not surprising... if the title was really true, this man would likely be richer than Warren Buffett at this point.

rimantas 12y ago
Wouldn't a coin toss be right 50% of the time?

mojuba 12y ago

I wonder if they use Bayesian logic, because they should. I also suspect that even though the article does not disclose a lot, the major factor at play in their model is the market sizes. I'm pretty sure their online version would heavily rely on the industry/segment you select. In other words it's a business-plan-looking-good approach which in today's rapidly changing world becomes less and less relevant. So I'll remain skeptical about it.

Someone1234 12y ago
> I wonder if they use Bayesian logic, because they should. Looking at his qualifications, I think it is safe to say he too took statistics 101.

claypoolb 12y ago

The output of his model is contrarian to what every accelerator and VC firm in the Valley says about investing. They believe the team is the most important factor of success. Thurston says its 12%. We love pushing the envelope!

ballpoint 12y ago

I'm not sure having better predictors of what businesses are good is necessarily a good idea. Part of the attraction of silicon valley is that it takes some of the risk out of trying new things, even if they might be bad ideas. This culture of trying things leads us to find the occasional really good idea. If we sit around all day plugging our ideas into models to see if, statistically speaking, the will succeed, we won't find the really novel ideas that look bad but are actually good.

Fede_V 12y ago

I would be incredibly interested to know how the model was crossvalidated. It is utterly trivial to cherry pick features post-facto to correctly predict winners, however people who are unfamiliar with machine learning might not know this.

skmurphy 12y ago

I wonder if he has applied the algorithim to his firm to improve his chances of survival

agarden 12y ago
Yes. From the article: "According to Thurston’s own model, Growth Science’s own chance of survival following its current business model is about 69 percent. Adding the automated service would actually improve its chances, he says."

thurstont 12y ago

Hi Y'all, Thomas here (guy in article). Just want to start by saying (1) this is a very intelligent thread, and (2) I didn't write the article, was just interviewed for it. You never know what's going to be written, no matter what you say. Here's how the models really play out. We compare our accuracy against the 10 year survivorship benchmark of 25% (not the 5 year). When you look at small businesses, VC-backed, and corporate ventures (ex. new products coming out of companies), the 10 year survival rate is around 25%, plus or minus 10% depending on the industry. Our models have made thousands of predictions for around nine years now - all the predictions were live, real-time and forward looking (no back-testing included here). From those predictions, around 3,400 have matured to date. That is, only around 3,400 of the results have happened - the businesses have either become big successes (ex. Uber) or failed. In our research, we have to actually wait for businesses to live or die to test our accuracy. From the roughly 3,400 predictions that have matured, we were right 66% of the time when predicting survivors, and 88% of the time when predicting failures. When we scratched beneath the surface, we were really around 66% accurate in both cases (just most businesses fail, which is why gloomy predictions were 22% more accurate - just a function of dumb luck since most things die). So we consider our algorithms to be 66% accurate, which is much more accurate than anything we're aware of in human history (remember, the baseline we're compared against is 25%). If you do a statistical analysis (to make sure our predictions weren't just luck), the models maintained a statistically significant correlation with 99% confidence. There was less than 1 chance in over 500,000 that the results were a function of luck (definitely not a coin toss). We've used these models in venture, and our performance puts us in the top 5% of all VC funds for our vintage years, so we've monetized these models effectively with real dollars and made considerable gains. I hope this gives folks a better sense for how it works. There's been a very emotional backlash to the Wired article today (not accusing this thread, just thinking of some others) and it's weird because it's just basic scientific research. Pretty drab stuff on most days, but apparently offensive to some people. Not sure why. We're using statistics to improve venture and startup odds, just like stats have been used to improve just about every other field humans have ever taken seriously. Seems obvious that stats are similarly useful in the startup world, and my dream has always been to help more businesses use stats to succeed. Anyway, definitely a lot more controversy and emotion than I would have expected. Otherwise pretty basic science, not claiming perfection, just striving for improvement, using data as best we can, etc. I hope at least some folks see this for what it is - nothing out of the ordinary in any other domain of science. Why should entrepreneurship be any different?

bbody 12y ago

Misleading title but an interesting article nonetheless. The comments here really show how much statistics are misunderstood by the public, even by a more technical-minded crowd.