5 ms·

Yeah, the 95% confidence interval is completely arbitrary to begin with. Is it really better than, say, 94% -- which will most assuredly not get you published?

Yeah, the 95% confidence interval is completely arbitrary to begin with. Is it really better than, say, 94% -- which will most assuredly not get you published? In many fields, I think that a p-value indicating 95% confidence in the results is fine, you are never going to get 99.999% or even 99% due to the errors inherent in the subject of study, which doesn't invalidate the results out of hand. It's the other host of errors, up to and including blatantly cherry-picking or fabricating results, where the focus should be. There's been a series of recent posts encouraging scientists (and, moreover, journals) to publish negative results, which is probably a good starting point to promote greater transparency and honesty in the paper publishing game.

apsec112 13y ago
This is part of the problem. A p-value of 0.05 absolutely does not (!!!) indicate 95% confidence in the results! What a p-value means is, if the hypothesis in question is false, then we have an x% chance of seeing the observed data. It tells you absolutely nothing about, if you observe such-and-such data, then how strongly should we believe the hypothesis. For example, you could have an extremely unlikely hypothesis (eg. "dice are controlled by alien telepathy"), test it, and still come out with p < 0.05 because of something much more plausible ("the dice were badly made and are biased"). This is known as the base rate fallacy; an extraordinary claim requires extraordinary proof. Moreover, to prove an extraordinary claim, you must show it is more likely not just compared to chance, but compared to every less extraordinary alternative, including "the experimenters are faking the data". It's also entirely possible to get a result at p < 0.05 that makes the hypothesis you're testing less likely. Suppose you want to know how far away the nearest star is. The value in the textbook is 13.4 light-years, and you think it's really 12,000. You take some measurements, and get values of 19.6, 17.4, 20.1, 20.4 and 18.5. Now, this is a significant result at p < 0.05 - if the star is 13.4 light-years away, then getting these numbers has a probability of less than 5%. However, these results completely rule out the hypothesis you're testing. The numbers you get are pretty unlikely if the real number is 13.4, but extraordinarily unlikely if the real number is 12,000, so this experiment makes the 13.4 number more credible. This kind of thing is why lots of researchers still believe in psychic powers - they keep testing for psychic powers, and keep getting results at p < 0.05, but don't notice their results make psychic powers less plausible. (Not joking - see http://commonsenseatheism.com/wp-content/uploads/2010/11/Wagenmakers-Why-Psychologists-Must-Change-the-Way-They-Analyze-Their-Data.pdf http://commonsenseatheism.com/wp-content/uploads/2010/11/Wag... for a detailed explanation.) This is an elementary mistake that every freshman statistics course warns against, and yet it's absolutely pervasive.
- eli_gottlieb 13y ago
  Right, you want this superhero cape with Bayes' Theorem printed on it? It's got priiiiooooooors.
- jmpeax 13y ago
  > What a p-value means is, if the hypothesis in question is false, then we have an x% chance of seeing the observed data. The hypothesis in question is the null hypothesis which must be assumed true not false for the p-value to mean a 100*p% chance of seeing the observed data. I can see why elementary mistakes are so pervasive.
hnha 13y ago
It's not that arbitrary really. 95% equals 2 sigma.

apsec112 13y ago

This is part of the problem. A p-value of 0.05 absolutely does not (!!!) indicate 95% confidence in the results! What a p-value means is, if the hypothesis in question is false, then we have an x% chance of seeing the observed data. It tells you absolutely nothing about, if you observe such-and-such data, then how strongly should we believe the hypothesis. For example, you could have an extremely unlikely hypothesis (eg. "dice are controlled by alien telepathy"), test it, and still come out with p < 0.05 because of something much more plausible ("the dice were badly made and are biased"). This is known as the base rate fallacy; an extraordinary claim requires extraordinary proof. Moreover, to prove an extraordinary claim, you must show it is more likely not just compared to chance, but compared to every less extraordinary alternative, including "the experimenters are faking the data". It's also entirely possible to get a result at p < 0.05 that makes the hypothesis you're testing less likely. Suppose you want to know how far away the nearest star is. The value in the textbook is 13.4 light-years, and you think it's really 12,000. You take some measurements, and get values of 19.6, 17.4, 20.1, 20.4 and 18.5. Now, this is a significant result at p < 0.05 - if the star is 13.4 light-years away, then getting these numbers has a probability of less than 5%. However, these results completely rule out the hypothesis you're testing. The numbers you get are pretty unlikely if the real number is 13.4, but extraordinarily unlikely if the real number is 12,000, so this experiment makes the 13.4 number more credible. This kind of thing is why lots of researchers still believe in psychic powers - they keep testing for psychic powers, and keep getting results at p < 0.05, but don't notice their results make psychic powers less plausible. (Not joking - see http://commonsenseatheism.com/wp-content/uploads/2010/11/Wagenmakers-Why-Psychologists-Must-Change-the-Way-They-Analyze-Their-Data.pdf http://commonsenseatheism.com/wp-content/uploads/2010/11/Wag... for a detailed explanation.) This is an elementary mistake that every freshman statistics course warns against, and yet it's absolutely pervasive.

eli_gottlieb 13y ago
Right, you want this superhero cape with Bayes' Theorem printed on it? It's got priiiiooooooors.
jmpeax 13y ago
> What a p-value means is, if the hypothesis in question is false, then we have an x% chance of seeing the observed data. The hypothesis in question is the null hypothesis which must be assumed true not false for the p-value to mean a 100*p% chance of seeing the observed data. I can see why elementary mistakes are so pervasive.

hnha 13y ago

It's not that arbitrary really. 95% equals 2 sigma.