4 ms·

A Statistical Test Gets Its Closeup

20 points by clay 16y ago

deleted 16y ago
[deleted]
crasshopper 16y ago
For the non-Bayesians, here's a paper that explains in detail one of the strongest criticisms of null hypothesis testing, by bayesian John Kruschke: http://www.indiana.edu/~kruschke/articles/Kruschke2011PoPS.pdf http://www.indiana.edu/~kruschke/articles/Kruschke2011PoPS.p... To summarize, an experimenter's internal mental state (intentions) can affect the t statistic. That is: not changing the experiment at all, but just thinking something different -- gives a different level of significance. In Kruschke's words: voo-doo.
ramanujan 16y ago
Great quote in the WSJ comment section: http://blogs.wsj.com/numbersguy/a-statistical-test-gets-its-closeup-1050/tab/comments/ http://blogs.wsj.com/numbersguy/a-statistical-test-gets-its-... The agency will bend their statistical significance requirements when hypothesizing that a treatment has risks, but almost never does when considering the efficacy of a treatment. That is the long and short of it. The FDA makes up the rules as they go along. As for the particular issue at stake, the Supreme Court's ruling in the Zicam case is horrifying: http://online.wsj.com/article/SB10001424052748703712504576235683249040812.html?mod=googlenews_wsj http://online.wsj.com/article/SB1000142405274870371250457623... It is indeed true that statistical significance is not the be-all and end-all. It is also true that you need some method of determining whether a result could be due to chance or not. This is particularly true when you have trial lawyers and regulators who smell blood. Dow Corning was sued and had its reputation dragged through the mud because people thought breast implants were causing some kind of disease. Turns out that wasn't true at all: http://query.nytimes.com/gst/fullpage.html?sec=health&res=9B03E6D9103BF932A15755C0A96F958260&n=Top%2fReference%2fTimes%20Topics%2fOrganizations%2fI%2fInstitute%20of%20Medicine http://query.nytimes.com/gst/fullpage.html?sec=health&re... Now that statistical significance has been jettisoned as a defense, any group of anecdotal injuries of dubious provenance is in theory grounds for a class action lawsuit. And trial lawyers specialize in turning that particular kind of theory into practice. There's a reason that "mesothelioma" was the #1 Adword for a while -- ambulance chasers bid on it like crazy. The Supreme Court has just made it a whole lot easier for them.

20 points by clay 16y ago

deleted 16y ago

[deleted]

For the non-Bayesians, here's a paper that explains in detail one of the strongest criticisms of null hypothesis testing, by bayesian John Kruschke: http://www.indiana.edu/~kruschke/articles/Kruschke2011PoPS.pdf http://www.indiana.edu/~kruschke/articles/Kruschke2011PoPS.p... To summarize, an experimenter's internal mental state (intentions) can affect the t statistic. That is: not changing the experiment at all, but just thinking something different -- gives a different level of significance. In Kruschke's words: voo-doo.

ramanujan 16y ago

Great quote in the WSJ comment section: http://blogs.wsj.com/numbersguy/a-statistical-test-gets-its-closeup-1050/tab/comments/ http://blogs.wsj.com/numbersguy/a-statistical-test-gets-its-... The agency will bend their statistical significance requirements when hypothesizing that a treatment has risks, but almost never does when considering the efficacy of a treatment. That is the long and short of it. The FDA makes up the rules as they go along. As for the particular issue at stake, the Supreme Court's ruling in the Zicam case is horrifying: http://online.wsj.com/article/SB10001424052748703712504576235683249040812.html?mod=googlenews_wsj http://online.wsj.com/article/SB1000142405274870371250457623... It is indeed true that statistical significance is not the be-all and end-all. It is also true that you need some method of determining whether a result could be due to chance or not. This is particularly true when you have trial lawyers and regulators who smell blood. Dow Corning was sued and had its reputation dragged through the mud because people thought breast implants were causing some kind of disease. Turns out that wasn't true at all: http://query.nytimes.com/gst/fullpage.html?sec=health&res=9B03E6D9103BF932A15755C0A96F958260&n=Top%2fReference%2fTimes%20Topics%2fOrganizations%2fI%2fInstitute%20of%20Medicine http://query.nytimes.com/gst/fullpage.html?sec=health&re... Now that statistical significance has been jettisoned as a defense, any group of anecdotal injuries of dubious provenance is in theory grounds for a class action lawsuit. And trial lawyers specialize in turning that particular kind of theory into practice. There's a reason that "mesothelioma" was the #1 Adword for a while -- ambulance chasers bid on it like crazy. The Supreme Court has just made it a whole lot easier for them.