4 ms·
One very important thing to know about the 0.05 threshold, and which I did not found in this thread, is that the ideal p-value for a problem is a function of th
by nestorD 7y ago
One very important thing to know about the 0.05 threshold, and which I did not found in this thread, is that the ideal p-value for a problem is a function of the number of samples (and the effect size but it as a lesser impact).
0.05 is way to stringent if you have 10 samples and way too lenient if you have 1 million samples.
But, by force of convention, everyone is using 0.05 (a value suggested by Fischer when basically all datasets were small) independently of their sample size and in a world were we are sometimes reaching dataset size that would have been inconceivable when the threshold was suggested.
Here is a good article to start to think about how one would select a p-value for an experiment :
https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0032734 https://journals.plos.org/plosone/article?id=10.1371/journal...
- cuchoi 7y agoIt is not a function of only the number of samples. It is also a function of how costly are false positives (type I error) and false negatives (type II error). That is (from my understanding) the paper's main point. But then you have to be able to calculate how costly is a type I and a type II error! That's seems a relatively straightforward question for a business (for example in A/B testing), but how do you measure that cost in academia? I think this would only introduce confusion and another variable for p-hacking.
- nestorD 7y agoYou can also use the apriori probability of a positiv, instead of a cost, which can be roughly deduced from existing litterature. The strenght here is that you get rid of an arbitrary decision (p-value) and instead use quantities that can be measured and critiqued by a skeptical reviewer. But, in my experience with small data, the impact of the size of the dataset dwarfed the impact of the cost/probability.