4 ms·

Objective Bayesian Hypothesis Testing

79 points by rnburn 2y ago

elmomle 2y ago
When the author says "objective" they are referring to a prior that gives equal weight to values within the null hypothesis and to those without (along with a few other things: symmetric and non-increasing away from the mean). I appreciate this approach, and think there's much to commend it, but think that that's a key thing to be aware of (because any use of "objective" when referring to priors is, shall we say, dubious).
- vcdimension 2y ago
  Yes, it would be nice to know how things change for different weightings of the null and alternative priors.
vcdimension 2y ago
This article is very interesting and informative, however it's a bit ironic that an article about misinterpretations of the meaning of the p-value, misinterprets the misinterpretation; in the first blue box it's clear that Bernstein is interpreting the p-value as the probability of randomly rejecting the null (which is what you do when you get something statistically significant) yet in the text following that they say he's interpreting it as the probability of the null. Bernsteins mistake is that he appears to interpret it as an unconditional probability rather than a conditional one (correct interpretation; p-value = Prob(rejecting the null when the null is true)).
- null08 2y ago
  Yes I had the same issue. But the wording "there is a < 5% probability that an outcome was the result of chance" is in fact problematic since many readers will go on to conclude "hence a >95% probability that the outcome was not the result of chance", so it is easier to misinterpret than the technical definition P( Observation | H_0 ). In courses I will typically use wordings like "If there was truly no association, then the probability of getting an observation like this is <5%".
- kqr 2y ago
  > correct interpretation; p-value = Prob(rejecting the null when the null is true) This is also not quite correct. The p-value is the probability of falsely rejecting the null due to sampling error. It is quiet on all other errors that are frequently committed. The real probability of falsely rejecting the null starts at 15 % thanks to mathematical slip-ups alone: https://two-wrongs.com/the-lying-p-value https://two-wrongs.com/the-lying-p-value
  - nalzok 2y ago
    > by kqr, published 2024-11-19 It's from the future! ;)
underlines 2y ago
A great question that I came across in Hypothesis Driven Development a long time ago: Should you use Frequentist Statistics or Bayesian Statistics? It's relevant when you do A/B or Multivariate Testing. As it was very difficult for someone like me without higher stats or math education, I can highly recommend the following additional sources: - https://www.redjournal.org/article/S0360-3016(21)03256-9/fulltext https://www.redjournal.org/article/S0360-3016(21)03256-9/ful... - https://amplitude.com/blog/frequentist-vs-bayesian-statistics-methods https://amplitude.com/blog/frequentist-vs-bayesian-statistic... - https://indico.cern.ch/event/568904/contributions/2651065/attachments/1487369/2310671/BvsF.pdf https://indico.cern.ch/event/568904/contributions/2651065/at...
- shiandow 2y ago
  The Bayesian approach to A/B testing gives an interesting example of how frequentists and Bayesian approaches can differ. A frequentist approach tries to limit the probability that a test setup will accept a 'false' result, one that could simply arise by chance. A Bayesian approach actually calculates the probability that a test result could occur 'by chance'. You can then stop the test at any point and be sure you only accept <x% of results that could occur by chance, by the power of expectation values you never breach the x% limit no matter how often you 'stop' the test. The interesting thing is that while these would seem to be very similar, there actually isn't anything stopping the Bayesian approach from accepting any test eventually. Giving it 0 statistical power in the frequentist sense. The only thing the Bayesian approach ensures is that for any 'false' test you accept after time T there are many more that will keep running.
  - Vecr 2y ago
    Why would you care about that though? Calculate the odds between your hypotheses, not the probability you'd ever see one.
    - shiandow 2y ago
      The Bayesian stance is that you should not care. The frequentist stance is that a test that has a p-value of 1 is the worst possible. My stance is that you should know why to care about either. Oh and that the thing you're calculating an expected value off should somehow contribute linearly to your profits/costs, averages do strange things to nonlinear functions.
bookofjoe 2y ago
Off topic but topical: Mike Lynch's yacht was named "Bayesian"
- vcdimension 2y ago
  So I guess we'll never know the p-value of that event...
  - clircle 2y ago
    Since “that event” happened, its probability is 1.
    - vcdimension 2y ago
      but what if H0 = Hewlett Packard did not plan to eliminate Mike Lynch and Stephen Chamberlain...

79 points by rnburn 2y ago

When the author says "objective" they are referring to a prior that gives equal weight to values within the null hypothesis and to those without (along with a few other things: symmetric and non-increasing away from the mean). I appreciate this approach, and think there's much to commend it, but think that that's a key thing to be aware of (because any use of "objective" when referring to priors is, shall we say, dubious).

vcdimension 2y ago
Yes, it would be nice to know how things change for different weightings of the null and alternative priors.

vcdimension 2y ago

This article is very interesting and informative, however it's a bit ironic that an article about misinterpretations of the meaning of the p-value, misinterprets the misinterpretation; in the first blue box it's clear that Bernstein is interpreting the p-value as the probability of randomly rejecting the null (which is what you do when you get something statistically significant) yet in the text following that they say he's interpreting it as the probability of the null. Bernsteins mistake is that he appears to interpret it as an unconditional probability rather than a conditional one (correct interpretation; p-value = Prob(rejecting the null when the null is true)).

null08 2y ago
Yes I had the same issue. But the wording "there is a < 5% probability that an outcome was the result of chance" is in fact problematic since many readers will go on to conclude "hence a >95% probability that the outcome was not the result of chance", so it is easier to misinterpret than the technical definition P( Observation | H_0 ). In courses I will typically use wordings like "If there was truly no association, then the probability of getting an observation like this is <5%".
kqr 2y ago
> correct interpretation; p-value = Prob(rejecting the null when the null is true) This is also not quite correct. The p-value is the probability of falsely rejecting the null due to sampling error. It is quiet on all other errors that are frequently committed. The real probability of falsely rejecting the null starts at 15 % thanks to mathematical slip-ups alone: https://two-wrongs.com/the-lying-p-value https://two-wrongs.com/the-lying-p-value
- nalzok 2y ago
  > by kqr, published 2024-11-19 It's from the future! ;)

underlines 2y ago

A great question that I came across in Hypothesis Driven Development a long time ago: Should you use Frequentist Statistics or Bayesian Statistics? It's relevant when you do A/B or Multivariate Testing. As it was very difficult for someone like me without higher stats or math education, I can highly recommend the following additional sources: - https://www.redjournal.org/article/S0360-3016(21)03256-9/fulltext https://www.redjournal.org/article/S0360-3016(21)03256-9/ful... - https://amplitude.com/blog/frequentist-vs-bayesian-statistics-methods https://amplitude.com/blog/frequentist-vs-bayesian-statistic... - https://indico.cern.ch/event/568904/contributions/2651065/attachments/1487369/2310671/BvsF.pdf https://indico.cern.ch/event/568904/contributions/2651065/at...

shiandow 2y ago
The Bayesian approach to A/B testing gives an interesting example of how frequentists and Bayesian approaches can differ. A frequentist approach tries to limit the probability that a test setup will accept a 'false' result, one that could simply arise by chance. A Bayesian approach actually calculates the probability that a test result could occur 'by chance'. You can then stop the test at any point and be sure you only accept <x% of results that could occur by chance, by the power of expectation values you never breach the x% limit no matter how often you 'stop' the test. The interesting thing is that while these would seem to be very similar, there actually isn't anything stopping the Bayesian approach from accepting any test eventually. Giving it 0 statistical power in the frequentist sense. The only thing the Bayesian approach ensures is that for any 'false' test you accept after time T there are many more that will keep running.
- Vecr 2y ago
  Why would you care about that though? Calculate the odds between your hypotheses, not the probability you'd ever see one.
  - shiandow 2y ago
    The Bayesian stance is that you should not care. The frequentist stance is that a test that has a p-value of 1 is the worst possible. My stance is that you should know why to care about either. Oh and that the thing you're calculating an expected value off should somehow contribute linearly to your profits/costs, averages do strange things to nonlinear functions.

bookofjoe 2y ago

Off topic but topical: Mike Lynch's yacht was named "Bayesian"

vcdimension 2y ago
So I guess we'll never know the p-value of that event...
- clircle 2y ago
  Since “that event” happened, its probability is 1.
  - vcdimension 2y ago
    but what if H0 = Hewlett Packard did not plan to eliminate Mike Lynch and Stephen Chamberlain...