2 ms·
It is not a function of only the number of samples. It is also a function of how costly are false positives (type I error) and false negatives (type II error).
by cuchoi 7y ago
It is not a function of only the number of samples. It is also a function of how costly are false positives (type I error) and false negatives (type II error). That is (from my understanding) the paper's main point.
But then you have to be able to calculate how costly is a type I and a type II error! That's seems a relatively straightforward question for a business (for example in A/B testing), but how do you measure that cost in academia?
I think this would only introduce confusion and another variable for p-hacking.
- nestorD 7y agoYou can also use the apriori probability of a positiv, instead of a cost, which can be roughly deduced from existing litterature. The strenght here is that you get rid of an arbitrary decision (p-value) and instead use quantities that can be measured and critiqued by a skeptical reviewer. But, in my experience with small data, the impact of the size of the dataset dwarfed the impact of the cost/probability.