6 ms·

'The fundamental problem is that p values don't mean what we "need" them to mean, that is p(null | significant effect).' From Bayes' theorem, this more useful

'The fundamental problem is that p values don't mean what we "need" them to mean, that is p(null | significant effect).' From Bayes' theorem, this more useful probability is given by p * x, where x = p(null) / p(significant effect). Maybe we could just lower the accepted threshold for statistical significance by several orders of magnitude so that, for statistically significant p, p * x is still small even for careful (i.e. big) estimates of x (e.g. maybe a Fermi approximation of the total number of experiments ever performed in the field in question). This doesn't necessarily imply impractically big sample sizes, although obviously this depends on the specifics (I believe the p value for a given value of the t-statistic decays exponentially with sample size).

kkylin 8y ago
This was basically the suggestion here: https://www.nature.com/articles/s41562-017-0189-z.epdf https://www.nature.com/articles/s41562-017-0189-z.epdf Also previous HN discussion: https://news.ycombinator.com/item?id=15192610 https://news.ycombinator.com/item?id=15192610
nonbel 8y ago
I don't follow your argument. You've got two premises: 1) You are saying that people are committing the transposing the conditional fallacy: p(H0|data) != p(data|H0): - OK 2) You say to use Bayes theorem to get the value we want: - OK, but actually a better formulation is p(H_0|data) = p(H_0)*p(data|H_0)/[p(H_0)*p(data|H_0) + p(H_1)*p(data|H_1) + ... + p(H_n)*p(data|H_n)] You probably don't need to add up all the way to hypothesis n since the terms eventually become negligible and can be dropped from the denominator. The point is that you have to compare how likely the result would be under other hypotheses, not just H_0. 3) You propose lowering the threshold for "significance" - How does this follow from the premises? Lets say you get a very low value for p(H_0)p(data|H_0), this can still be much higher than p(H_1)p(data|H_1), etc so it is still the best choice. Ie, you can get a low p-value given H_0 but if there is no better model out there you should still keep H_0.
- edwinksl 8y ago
  Minor point, but you are missing p(H_0)*p(data|H_0) in the denominator in bullet point 2.
  - nonbel 8y ago
    Thanks, fixed.
- tprice7 8y ago
  I’m assuming the question we are trying to answer is not “which H_n is most probable”, but rather “how safe is it to conclude that H_0 is not true”. For example, say we are concerned with whether the difference between two groups is less than a certain amount or greater than a certain amount.
  - nonbel 8y ago
    >“how safe is it to conclude that H_0 is not true” I would just take "very safe" as a principle, there is even the truism "all models are wrong". >"the difference between two groups is less than a certain amount or greater than a certain amount" You are ignoring a lot of the model being tested here (eg, normality, independence of the measurements, etc) and only considering one parameter.
    - tprice7 8y ago
      “how safe is it to conclude that H_0 is not true” What I meant here by H_0 was the hypothesis that the difference between groups is less than some particular threshold. I think if you made the threshold large enough then it would not be safe to conclude that H_0 is not true. "You are ignoring a lot of the model being tested here (eg, normality, independence of the measurements, etc) and only considering one parameter. " I said enough so that if you were arguing in good faith you could fill in the gaps yourself.
      - nonbel 8y ago
nonbel 8y ago
Responding to this but getting rid of the intense nesting: “I'm not sure what you are arguing anymore.” It’s the claim I make in 3, and then the secondary claim that making our upper bound on p(null | T, normal, iid) small for significant p-values (i.e. p(T | null, normal, id)) could be used as a criterion for whether our threshold for statistical significance is small enough. “You seemed to be disagreeing with that” I’m not sure what I said that gave that impression. I didn’t mention anything about the normal / iid assumptions initially not because I thought we weren’t making these assumptions but because I didn’t think these details were essential to my point." Please give some example code or calculation steps for what you are talking about.
- tprice7 8y ago
  You mean for computing the p-value associated with a range of means rather than a single null mean value? That’s the only thing I can think of for which providing code or calculation steps would be applicable (well, there’s also getting a Fermi approximation for the total number of experiments ever performed in a discipline, which I gave as a preliminary suggestion for a cautious estimate of x, but I don’t have time to do that and the main difficulty of that isn’t the math anyways). Anyways I would be happy to provide that if you can confirm that this is what you meant and that you are asking out of genuine curiosity.
  - nonbel 8y ago
    >"our upper bound on p(null | T, normal, iid) small for significant p-values (i.e. p(T | null, normal, id))" This sounds like nonsense to me so I would like to see an example of what you mean.
    - tprice7 8y ago
      Ooops, I forgot to include something in the list of conditions. For every probability I mentioned in the post you just quoted and its grandparent post, we should add to the list of conditions that the number of samples used is the same as the number of samples used in our experiment. Anyways, if it sounds like nonsense then I will try rephrasing. Right now the threshold for statistical significance is p = 0.05, sometimes lower depending on the field. Let’s say we want to lower this threshold, and we need some way of determining how low it should be. I am suggesting that this could be done by deciding on an acceptably safe threshold, say q, for p(null | T, num samples, normal, iid), and an acceptably safe upper bound, say x_0, for p(null | num samples, normal, iid) / p(T | num samples, normal, iid). Then we use q / x_0 as the threshold for statistical significance. We would also be using p-values for a null hypothesis corresponding to a range of means rather than a single value (or a range of differences in means, or something else suitably adapted to the type of test we are doing), because with a single value, having an upper bound on the probability of the null hypothesis is vacuous, as you very eagerly point out. You could argue that this is useless because what we really want to bound is p(null | num samples, T), not p(null | num samples, T, normal, iid). I would respond that, yes this would be the more desirable quantity to know, but we need to make some simplifying assumptions in order to make this problem tractable, and while the normal and iid assumptions probably aren’t true, there are at least situations where we can be confident that they approximate reality reasonably well.

kkylin 8y ago

This was basically the suggestion here: https://www.nature.com/articles/s41562-017-0189-z.epdf https://www.nature.com/articles/s41562-017-0189-z.epdf Also previous HN discussion: https://news.ycombinator.com/item?id=15192610 https://news.ycombinator.com/item?id=15192610

nonbel 8y ago

I don't follow your argument. You've got two premises: 1) You are saying that people are committing the transposing the conditional fallacy: p(H0|data) != p(data|H0): - OK 2) You say to use Bayes theorem to get the value we want: - OK, but actually a better formulation is p(H_0|data) = p(H_0)*p(data|H_0)/[p(H_0)*p(data|H_0) + p(H_1)*p(data|H_1) + ... + p(H_n)*p(data|H_n)] You probably don't need to add up all the way to hypothesis n since the terms eventually become negligible and can be dropped from the denominator. The point is that you have to compare how likely the result would be under other hypotheses, not just H_0. 3) You propose lowering the threshold for "significance" - How does this follow from the premises? Lets say you get a very low value for p(H_0)p(data|H_0), this can still be much higher than p(H_1)p(data|H_1), etc so it is still the best choice. Ie, you can get a low p-value given H_0 but if there is no better model out there you should still keep H_0.

edwinksl 8y ago
Minor point, but you are missing p(H_0)*p(data|H_0) in the denominator in bullet point 2.
- nonbel 8y ago
  Thanks, fixed.
tprice7 8y ago
I’m assuming the question we are trying to answer is not “which H_n is most probable”, but rather “how safe is it to conclude that H_0 is not true”. For example, say we are concerned with whether the difference between two groups is less than a certain amount or greater than a certain amount.
- nonbel 8y ago
  >“how safe is it to conclude that H_0 is not true” I would just take "very safe" as a principle, there is even the truism "all models are wrong". >"the difference between two groups is less than a certain amount or greater than a certain amount" You are ignoring a lot of the model being tested here (eg, normality, independence of the measurements, etc) and only considering one parameter.
  - tprice7 8y ago
    “how safe is it to conclude that H_0 is not true” What I meant here by H_0 was the hypothesis that the difference between groups is less than some particular threshold. I think if you made the threshold large enough then it would not be safe to conclude that H_0 is not true. "You are ignoring a lot of the model being tested here (eg, normality, independence of the measurements, etc) and only considering one parameter. " I said enough so that if you were arguing in good faith you could fill in the gaps yourself.
    - nonbel 8y ago

nonbel 8y ago

Responding to this but getting rid of the intense nesting: “I'm not sure what you are arguing anymore.” It’s the claim I make in 3, and then the secondary claim that making our upper bound on p(null | T, normal, iid) small for significant p-values (i.e. p(T | null, normal, id)) could be used as a criterion for whether our threshold for statistical significance is small enough. “You seemed to be disagreeing with that” I’m not sure what I said that gave that impression. I didn’t mention anything about the normal / iid assumptions initially not because I thought we weren’t making these assumptions but because I didn’t think these details were essential to my point." Please give some example code or calculation steps for what you are talking about.

tprice7 8y ago
You mean for computing the p-value associated with a range of means rather than a single null mean value? That’s the only thing I can think of for which providing code or calculation steps would be applicable (well, there’s also getting a Fermi approximation for the total number of experiments ever performed in a discipline, which I gave as a preliminary suggestion for a cautious estimate of x, but I don’t have time to do that and the main difficulty of that isn’t the math anyways). Anyways I would be happy to provide that if you can confirm that this is what you meant and that you are asking out of genuine curiosity.
- nonbel 8y ago
  >"our upper bound on p(null | T, normal, iid) small for significant p-values (i.e. p(T | null, normal, id))" This sounds like nonsense to me so I would like to see an example of what you mean.
  - tprice7 8y ago
    Ooops, I forgot to include something in the list of conditions. For every probability I mentioned in the post you just quoted and its grandparent post, we should add to the list of conditions that the number of samples used is the same as the number of samples used in our experiment. Anyways, if it sounds like nonsense then I will try rephrasing. Right now the threshold for statistical significance is p = 0.05, sometimes lower depending on the field. Let’s say we want to lower this threshold, and we need some way of determining how low it should be. I am suggesting that this could be done by deciding on an acceptably safe threshold, say q, for p(null | T, num samples, normal, iid), and an acceptably safe upper bound, say x_0, for p(null | num samples, normal, iid) / p(T | num samples, normal, iid). Then we use q / x_0 as the threshold for statistical significance. We would also be using p-values for a null hypothesis corresponding to a range of means rather than a single value (or a range of differences in means, or something else suitably adapted to the type of test we are doing), because with a single value, having an upper bound on the probability of the null hypothesis is vacuous, as you very eagerly point out. You could argue that this is useless because what we really want to bound is p(null | num samples, T), not p(null | num samples, T, normal, iid). I would respond that, yes this would be the more desirable quantity to know, but we need to make some simplifying assumptions in order to make this problem tractable, and while the normal and iid assumptions probably aren’t true, there are at least situations where we can be confident that they approximate reality reasonably well.