4 ms·

Imagine a drug that beat a placebo in 38/46 trials. It would be approved. Everyone on HN would love an A/B test with such clear results. Call the drug "Berkshi

Imagine a drug that beat a placebo in 38/46 trials. It would be approved. Everyone on HN would love an A/B test with such clear results. Call the drug "Berkshire Hathaway performance" and the placebo "S&P 500 performance" and suddenly it's a coincidence.

murbard2 12y ago
Yes, it's called a prior, and there's nothing wrong with that.
dllthomas 12y ago
Those 46 trials weren't anything like independent.
- derf_ 12y ago
  Buffett addressed this criticism in 1984, when he presented the public records of all of the students of Graham and Dodd it was possible to track: <http://www4.gsb.columbia.edu/null?&exclusive=filemgr.download&file_id=522> http://www4.gsb.columbia.edu/null?&exclusive=filemgr.downloa... The relevant bit: "So these are nine records of “coin-flippers” from Graham-and-Doddsville. I haven’t selected them with hindsight from among thousands. It’s not like I am reciting to you the names of a bunch of lottery winners — people I had never heard of before they won the lottery. I selected these men years ago based upon their framework for investment decision-making. I knew what they had been taught and additionally I had some personal knowledge of their intellect, character, and temperament. ... While they differ greatly in style... all exploit the difference between the market price of a business and its intrinsic value."
  - tempestn 12y ago
    First, there was considerably less access to information in 1984 than now, so the market almost certainly was less efficient. Second, thanks to Fama and French, we now know that there is indeed a value premium. However, the majority view is that it is a risk premium: still useful because it exposes one to different risk than pure market beta, but not a free lunch. (Also it is most likely more efficient to capture the premium using low-cost "tilted" (value) funds, rather than by picking individual stocks and unnecessarily taking on unsystematic risk.)
  - dllthomas 12y ago
    That seems to address hindsight bias, but not independence.
mikeash 12y ago
I frequently see articles on the front page of HN discussing publication bias and how positive outcomes for drug trials are often purely coincidence. Your characterization of the community's perceptions seems to be imaginary.

murbard2 12y ago

Yes, it's called a prior, and there's nothing wrong with that.

dllthomas 12y ago

Those 46 trials weren't anything like independent.

derf_ 12y ago
Buffett addressed this criticism in 1984, when he presented the public records of all of the students of Graham and Dodd it was possible to track: <http://www4.gsb.columbia.edu/null?&exclusive=filemgr.download&file_id=522> http://www4.gsb.columbia.edu/null?&exclusive=filemgr.downloa... The relevant bit: "So these are nine records of “coin-flippers” from Graham-and-Doddsville. I haven’t selected them with hindsight from among thousands. It’s not like I am reciting to you the names of a bunch of lottery winners — people I had never heard of before they won the lottery. I selected these men years ago based upon their framework for investment decision-making. I knew what they had been taught and additionally I had some personal knowledge of their intellect, character, and temperament. ... While they differ greatly in style... all exploit the difference between the market price of a business and its intrinsic value."
- tempestn 12y ago
  First, there was considerably less access to information in 1984 than now, so the market almost certainly was less efficient. Second, thanks to Fama and French, we now know that there is indeed a value premium. However, the majority view is that it is a risk premium: still useful because it exposes one to different risk than pure market beta, but not a free lunch. (Also it is most likely more efficient to capture the premium using low-cost "tilted" (value) funds, rather than by picking individual stocks and unnecessarily taking on unsystematic risk.)
- dllthomas 12y ago
  That seems to address hindsight bias, but not independence.

mikeash 12y ago

I frequently see articles on the front page of HN discussing publication bias and how positive outcomes for drug trials are often purely coincidence. Your characterization of the community's perceptions seems to be imaginary.