4 ms·

I'm interested in your perspective. What would be a better replacement for p?

patientplatypus 7y ago
Ultimately, I think what we are trying to do is find out if a result is "surprising", as in it doesn't do what we would expect or it would do what we would expect, which is based on an emotional point of view from the perspective of the tester. Much like in art, in the sciences there is much left up to the muse, although in the sciences the dedication to rigor leads more often than not to a disregard to emotion or just assume that all utterances are by definition rational. I would probably keep the p-value, or something like it, for the rigorous analytical side, but I would make the assumptions to be tested much more rigorous. I would want to know not only why the test should be surprising, but I would want to know if our level of surprise is going up and down over time. It may be, for example, that as we accumulate more knowledge over time we are less and less likely to be surprised (or maybe more!) and therefore we should expect the p-values that elicit a surprise or not to change. But overall, I think there is not enough emotional soul searching over what it means for a mathematician to be surprised. And since that is the primary axiom over which everything else follows, and a lot of statistical papers don't really think this through well, there is a ton of faulty analysis out there. Garbage in garbage out. EDIT: To give another clearer example - Suppose we want to test if a miracle drug makes people immortal, to give an obviously ridiculous premise. Then in this case maybe a p-value of .001 is most appropriate because it would overturn huge amounts of medical knowledge. In this case though we need a judgement call backed up by evidence, in the sense of evidence to convince another such as in a law trial, to make a convincing case that this is the "correct" p-value to use. I think there is a desire in science to simply say, "Through sheer logic I have found the answer and therefore charisma/politics/judgement have no bearing on my analysis. And therefore I cannot be disputed on such things". This is an emotional viewpoint that seems in contradiction to the evidence, and it seems a common enough viewpoint that I don't believe it's purely a straw-man argument. But then if the politics do exist how do we get around them? After all, if we start making broad based judgement calls on what p-values should exist then how do we make sure papers are not disputed for begging the question? I would say that the best solution would be to force statisticians to get a third party to give them an appropriate p-value. Then the science has quickly, as many things, become a social problem - how do we organize such a system of statisticians giving each other appropriate p-values such that it is accurate but not corrupt? Unfortunately, this sort of thing is generally considered a "hard" problem.

Ultimately, I think what we are trying to do is find out if a result is "surprising", as in it doesn't do what we would expect or it would do what we would expect, which is based on an emotional point of view from the perspective of the tester. Much like in art, in the sciences there is much left up to the muse, although in the sciences the dedication to rigor leads more often than not to a disregard to emotion or just assume that all utterances are by definition rational. I would probably keep the p-value, or something like it, for the rigorous analytical side, but I would make the assumptions to be tested much more rigorous. I would want to know not only why the test should be surprising, but I would want to know if our level of surprise is going up and down over time. It may be, for example, that as we accumulate more knowledge over time we are less and less likely to be surprised (or maybe more!) and therefore we should expect the p-values that elicit a surprise or not to change. But overall, I think there is not enough emotional soul searching over what it means for a mathematician to be surprised. And since that is the primary axiom over which everything else follows, and a lot of statistical papers don't really think this through well, there is a ton of faulty analysis out there. Garbage in garbage out. EDIT: To give another clearer example - Suppose we want to test if a miracle drug makes people immortal, to give an obviously ridiculous premise. Then in this case maybe a p-value of .001 is most appropriate because it would overturn huge amounts of medical knowledge. In this case though we need a judgement call backed up by evidence, in the sense of evidence to convince another such as in a law trial, to make a convincing case that this is the "correct" p-value to use. I think there is a desire in science to simply say, "Through sheer logic I have found the answer and therefore charisma/politics/judgement have no bearing on my analysis. And therefore I cannot be disputed on such things". This is an emotional viewpoint that seems in contradiction to the evidence, and it seems a common enough viewpoint that I don't believe it's purely a straw-man argument. But then if the politics do exist how do we get around them? After all, if we start making broad based judgement calls on what p-values should exist then how do we make sure papers are not disputed for begging the question? I would say that the best solution would be to force statisticians to get a third party to give them an appropriate p-value. Then the science has quickly, as many things, become a social problem - how do we organize such a system of statisticians giving each other appropriate p-values such that it is accurate but not corrupt? Unfortunately, this sort of thing is generally considered a "hard" problem.