3 ms·
Cutoffs aren't the problem. They are inevitable; the logic is virtually identical to the question "Why can you drink at 21 and vote at 18?" Of course the Respon
by jerf 7y ago
Cutoffs aren't the problem. They are inevitable; the logic is virtually identical to the question "Why can you drink at 21 and vote at 18?" Of course the Responsibility Fairy doesn't visit you that night and make you suddenly able to handle it when you weren't the day before, it's just that at scale, you don't have much choice but to operate that way because everything else is just too expensive.
The problem with p-values is that the shape of the resulting cutoff just isn't what we're looking for. We want something along the lines of "What is the probability that this hypothesis is true, and how true is it?", and "What is the probability that this result could have happened even if the hypothesis is false?" is only an approximation at the best of times, and at worst, downright misleading. That's true even before we consider some of the other issues that my English kind of elides over, but the math contains; one of my problems with significance testing as it is commonly done is that there are actually ranges of hypotheses, and it really overprivileges "the" "null" hypothesis; I could write a decently-sized HN post just criticizing those two quoted words.
There is no rigid process that can produce the answer we really want, but I think we can provide a selection of better default tools. An example of one that has already been deployed to some extent is "power analysis", which is not a direct answer but lets people crafting studies analyze how big their studies will have to be before running them. We can build more tools like that.
- buboard 7y ago> "What is the probability that this hypothesis is true, and how true is it?", and "What is the probability that this result could have happened even if the hypothesis is false? The first is impossible to calculate by definition. The second can be derived from p value. Virtually all journals require rigorous reporting of p values along with averages, and the justification of the statistical test used.
- jerf 7y agoYou've mangled my sentence in your quote. The rest matters. It may not be a masterpiece of the English language, but I already said the first is not directly possible and I wrote the second as a description of p-values quite on purpose. "Virtually all journals require rigorous reporting of p values along with averages, and the justification of the statistical test used." That's begging the question. The entire topic of conversation is whether or not the standards of justification are adequate.
- buboard 7y agoah yes i agree with those. As you said, the selection of hypothesis is the actual problem, and statistical power doesnt solve that one either. There are some scientists thinking over this though. This is an idea from a neuroscience lab: https://www.researchmaps.org/ https://www.researchmaps.org/ The idea was to create causal directed graphs for biology from the literature , which would be used identify what experiments are missing and thus inform future science.