5 ms·
I don't follow your argument. You've got two premises: 1) You are saying that people are committing the transposing the conditional fallacy: p(H0|data) != p(da
by nonbel 9y ago
I don't follow your argument. You've got two premises:
1) You are saying that people are committing the transposing the conditional fallacy: p(H0|data) != p(data|H0):
- OK
2) You say to use Bayes theorem to get the value we want:
- OK, but actually a better formulation is
p(H_0|data) = p(H_0)*p(data|H_0)/[p(H_0)*p(data|H_0) + p(H_1)*p(data|H_1) + ... + p(H_n)*p(data|H_n)]
You probably don't need to add up all the way to hypothesis n since the terms eventually become negligible and can be dropped from the denominator. The point is that you have to compare how likely the result would be under other hypotheses, not just H_0.
3) You propose lowering the threshold for "significance"
- How does this follow from the premises? Lets say you get a very low value for p(H_0)p(data|H_0), this can still be much higher than p(H_1)p(data|H_1), etc so it is still the best choice. Ie, you can get a low p-value given H_0 but if there is no better model out there you should still keep H_0.
- edwinksl 9y agoMinor point, but you are missing p(H_0)*p(data|H_0) in the denominator in bullet point 2.
- nonbel 9y agoThanks, fixed.
- tprice7 9y agoI’m assuming the question we are trying to answer is not “which H_n is most probable”, but rather “how safe is it to conclude that H_0 is not true”. For example, say we are concerned with whether the difference between two groups is less than a certain amount or greater than a certain amount.
- nonbel 9y ago>“how safe is it to conclude that H_0 is not true” I would just take "very safe" as a principle, there is even the truism "all models are wrong". >"the difference between two groups is less than a certain amount or greater than a certain amount" You are ignoring a lot of the model being tested here (eg, normality, independence of the measurements, etc) and only considering one parameter.
- tprice7 9y ago“how safe is it to conclude that H_0 is not true” What I meant here by H_0 was the hypothesis that the difference between groups is less than some particular threshold. I think if you made the threshold large enough then it would not be safe to conclude that H_0 is not true. "You are ignoring a lot of the model being tested here (eg, normality, independence of the measurements, etc) and only considering one parameter. " I said enough so that if you were arguing in good faith you could fill in the gaps yourself.
- nonbel 9y agoI am arguing in good faith. There may be some cases in physics where there is some theoretical distribution derived related specifically to the problem, and they believe the model to be actually 100% true. Otherwise, the model should be assumed to only be an approximation at best.
- tprice7 9y agoYes, the t-test does assume normality and you can never be sure of perfect normality if that's what you are getting at (although I believe that simulation tests of the robustness of the t-test against deviations from normality generally show that this isn't too much of a practical concern). I wasn't trying to address every potential weakness with the t-test (or p-values in general); I was addressing the one stated in the article.
- nonbel 9y agoI'm saying that unless you have bothered to derive a statistical model from your theory, and you believe that theory may actually be correct, then you know that you will reject your model if enough time/money is spent on testing it.
- tprice7 9y agoOk, I will assume that by "model" here you mean a probability distribution on the parameters relevant to your experiment. In that case I agree with what you just said: knowing exactly the correct model is impossible in a similar way that knowing someone's height to an infinite degree of precision is impossible. But I never said anything to the contrary. The H_0 I gave corresponds to an infinite set of models and not a single one (note that I said the difference is less than a certain threshold, not that the difference is 0 (although it still would be an infinite set in that case, but the probability would be 0)).