6 ms·
Business Advice Plagued by Survivor Bias
- yarapavan 17y agoThe article quotes Peter Norvig: When a published paper proclaims "statistically, this could only happen by chance one in twenty times," it is quite possible that similar experiments have been performed twenty times, but have not been published.
- ggchappell 17y agoIt occurred to me that this means there is a real advantage to academia's "publish or perish" phenomenon. It means that when an experiment's results are not what was hoped, those results are still likely to get published. The experimenter would like to have his name on some ground-breaking results, of course, but, failing that, he still must have publications. So he publishes anyway. Not so true in industry.
- jerf 17y agoAre you sure about that? Journals don't (in general) publish negative results. The result of the pressure to publish is not that negative results get published; it's that the scientist has an incentive to create positive results. Not necessarily through data falsification, but through choosing experiments very likely to generate "results", which, unfortunately, also implies that the information content of such experiments is necessarily lower than it could be if the scientist had more freedom to perform an experiment that had a higher likelihood of failure.
- jibiki 17y agoGoogling "journal of negative results" gives a few hits. I think it's probably possible to get negative results published, but I doubt that they look good on the resume.
- zasz 17y agoThis article makes everything pg says about the reasons behind Viaweb's success sound rather suspect, doesn't it?
- jswinghammer 17y agoWasn't it always suspect?
- biohacker42 17y agoIt depends of what you consider suspect. I always saw PG's account of the Viaweb days in the light of the .COM 1.0 bubble.
- rjurney 17y agoIts funny - but for so many startup success stories, bubbles are the pink elephant in the room. When you try to draw lessons from success during downturns, the lessons are very different, and yet most thought leaders come from bubbles. This deeply effects the quality of the advice out there - most of it comes from people who are to some extent unknowing products of bubbles - but who attribute all of their success to actions they took that have universal applicability. When it often ain't so. For this reason, for me, 'startup credibility' goes up tremendously when it comes from a serial entrepreneur, or from someone that has actually built a cash-starved company to success during a downturn. "What can I do with very little?" and "What can I do if it is possible to raise $1 million?" are fundamentally different questions - and the same person isn't necessarily going to be skilled in both situations unless they've demonstrated that ability. Whereas this comment - it has the legitimacy of coming from a complete failure :D Finally... if your community leaders are state employees that can't say anything negative in public... you really need to be aware of that before you take their optimism at face value. Take to time to query them in private, to find out what they actually think. It might save you some mistakes and some grief.
- caffeine 17y agoOddly, self-help gurus seem to enjoy the opposite phenomenon (i.e. those who can't, teach self-help seminars). I remember at one point this guy Richard(?) Kiyosaki had a book "Rich Dad, Poor Dad." In it, you basically found out that Kiyosaki was (a) lying his tits off or (b) a terrible investor. You found out that he'd been bankrupt, and had run several companies into the ground. That he invested in mining companies in Russia whose value would swing 50% up and down from day-to-day - but that this didn't bother him... Wacko. Yet, somehow, the book was a total phenomenon and made the guy huge amounts of money.
- calcnerd256 17y agoTeleology is funny.
- smartbear 17y agoDid I cause you to say that? :-P
- mediaman 17y agoJason, this was a great piece and more should do themselves a service by reading it. It's one reason to pursue 'smaller' opportunities with lower variance of return expectations but higher median returns -- for example many niche businesses -- than industries with gigantic return expectations. The latter attracts far more entrants, and while somebody gets rich, the median player does not. But this isn't clear to new entrants because of survivorship bias. Somewhat related to this is the data availability bias. Big opportunities tend to have much better data sets, and so are easier to study and receive more attention. Many niche industries are closely private and are smaller, so do not have much publicly available data, making study difficult and attracting less attention. Therefore like moths to light bigger industries receive more study and more entrants even though their median expected economics are worse.
- mhartl 17y agoIt's one reason to pursue 'smaller' opportunities with lower variance of return expectations but higher median returns -- for example many niche businesses -- than industries with gigantic return expectations. This neatly encapsulates much of my current business strategy.
- 10ren 17y agoWarren Buffett talks about this in stock investing: you're more likely to find a bargain in the under-researched companies, because there's greater variance between the stock price and the stock value (price inefficiencies). He's publicly claimed to be able to double a million dollars in a year with this approach (it won't work for his $40+ billion, because there aren't enough under-researched stocks to buy that are also bargains).
- gatormax 17y agoI wish it was still on the front page. Understanding bias is, in my book, key, key, key to an effective understanding (and navigation) of reality.
- bigwill 17y agoVery Nassim Taleb. These are great books NT's written that discuss our susceptibility to survivorship bias and other rational failings we have: Fooled by Randomness and The Black Swan.
- smartbear 17y agoYes! You're right I'm hardly the first people to talk about such things. The Black Swan is a classic although oddly enough right now on the front page of HN there's a critique of Black Swan that pokes holes in it. :-) But yes it's true there's lots of good material on these subjects for those who want to explore further. Hopefully my piece whetted your appetite -- it's not the final word!
- donaldc 17y agoThis is a good point. At the same time, I believe there are actual differences between businesses that affect their chance of success. So the trick is to separate the 'accidental' features of successful businesses that don't increase their chances of success from the 'essential' features that do. I wish the article went into how one might do this...
- mediaman 17y agoOne important way referenced in the article is to evaluate failures as well as successes for the likelihood of a trait: for example Jim Collins describes certain features that successful companies have, but does not evaluate the pool of losers to see if they also featured the same trait. Another example is when evaluating various corporate traits against stock market performances, many people will choose a list of companies that exist today and then study their history to identify correlations of various traits. This omits those that no longer exist today, and it is a common problem among, for example, beginning hedge fund equity analysts. Instead, one should study the cohort from a random sample as existed at the beginning of the study, not at the end, and then study their performances from that point forward.
- smartbear 17y agoAgreed with mediaman here. Select both successes AND failures. Where they seem similar, those things may not teach us much. Where there is a pattern shown ONLY IN ONE SIDE (more or less) there might be a lesson. I have a fear that there's isn't a lot you can do to guarantee success, but I admit this is just my feeling, not based on any facts or stats.
- qu1j0t3 17y agoThe winners write the histories.
- swombat 17y agoDo you read business blogs where the author has failed three times without success? Am I the only one who regularly reads blog posts by an entrepreneur analysing how he's failed and what he would do differently next time? They pop up on HN quite often.
- BrandonWatson 17y agoAre you talking about www.thefailingpoint.com? That's me, and I am happy to share my experiences. They have been in the massive up and down times. Success and failure. I find it therapeutic to write, and since my old engineering team is in Y!Combinator this summer, I got the religion about sharing what I know. They asked me for a compendium of all the stupid mistakes I had made so that they wouldn't make them. =) I joked it was too much work, but it turns out that it makes for some good essays, and I really hope to turn it into a published work. For now, I am posting all the essays online for all to read for free. Chapter 1 and 2 are almost fully posted. Please send feedback.
- gommm 17y agoYes and I love reading those... They are usually much more interesting and insightful
- smartbear 17y agoI admit an amount of hyperbole when I implied that you NEVER read such blogs, and of course it's good that you do! In general I think it's fair to say that the more popular blogs and books about startups talk about successes rather than failures, so I'll defend my general point. I did take some license in how I stated it, and thanks for pointing out that it's too harsh and that there ARE good sources of info like that.
- seanstickle 17y agoTalk about survivor bias: http://www.dailymail.co.uk/news/article-1165768/The-man-survived-TWO-nuclear-bombs-Lucky-Yamaguchi-tells-lived-Hiroshima--fled-home-Nagasaki.html http://www.dailymail.co.uk/news/article-1165768/The-man-surv...
- neilk 17y agoThe takeaway here should be: if the book doesn't systematically examine the differences between winners and losers, drop it and walk away. Sentence by sentence, it's making you dumber. So forget In Search of Excellence. Try books like How To Be a Star at Work, which is based on a pretty rigorous study of employee productivity.
- schammy 17y agoThis was a great read. I guess I have underestimated the intelligence of bears!
- movix 17y ago"Planes with bullets in the cockpit or fuel tanks didn't make it home" Such a brilliant piece of lateral thinking - thanks for sharing.
- ojbyrne 17y agoJumping on books marketed at airports to harried executives isn't really valid. There's plenty of actual business research done by professionals (and they've heard of "survivorship bias, along with all the other threats to validity). It's like looking at all the "For Dummies" books in your local book store's computer section and deciding that CS research is poor. And Harvard Business School Case Studies are meant for teaching, and their function is analogous to science textbooks that leave out all the dead ends and fruitless research for pedagogical reasons.