3 ms·

“Pick a measure” just meant that you’re predicting the difference will be exactly 0%. P(0) = 1. The difference between a Bayes factor and a likelihood ratio is

“Pick a measure” just meant that you’re predicting the difference will be exactly 0%. P(0) = 1. The difference between a Bayes factor and a likelihood ratio is Bayes factor uses the marginal likelihood. So you need to pick your parameters ahead of time, weighted by priors. With a likelihood ratio, you can use the best parameters given the data. You can do the likelihood ratio in an objective way, because you’re choosing whatever has the least error given the data. Bayes factor you can’t be totally objective. You need to choose ahead. Upside is it reduces overfitting.

kgwgk 3y ago
> “Pick a measure” just meant that you’re predicting the difference will be exactly 0%. The difference of what? If you mean for example the difference between the population means of two groups that doesn’t mean that the observed difference between two sample means is zero. A non-zero observation doesn’t mean that “the model loses”. A non-zero observed difference is not just something that can happen, it’s what is expected. If you mean that the null hypothesis is really “the difference between the observed means is exactly zero” that doesn’t seem very useful and I’ve never seen anyone do that. You don’t need statistics of any kind to reject the model “the observation is zero” when the observation is not zero. Apart from that I agree that different hypothesis testing procedures do different things and their respective merits are debatable. My point was just that if you have a well-defined “null” model you can do different things with it. Using the same exact model.
- travisjungroth 3y ago
  > that doesn’t seem very useful and I’ve never seen anyone do that. Yes. That’s why it’s a straw man. I’m being sort of uncharitable in that description. My point is that’s a starting point. To go from there, you need to choose a model which will have parameters or priors.
  - kgwgk 3y ago
    That’s where the « null hypothesis » comes in. If it fixes the parameters in the model you get a well-defined « null » model with a well-defined probability distribution for the observation and - just like you can take this null hypothesis model and do frequentist calculations with it - you can take this model and calculate a Bayes factor relative to some other model. (To be clear, if the null hypothesis doesn’t fully specify the parameters the preceding paragraph doesnt apply and the situation is more complex.)

kgwgk 3y ago

> “Pick a measure” just meant that you’re predicting the difference will be exactly 0%. The difference of what? If you mean for example the difference between the population means of two groups that doesn’t mean that the observed difference between two sample means is zero. A non-zero observation doesn’t mean that “the model loses”. A non-zero observed difference is not just something that can happen, it’s what is expected. If you mean that the null hypothesis is really “the difference between the observed means is exactly zero” that doesn’t seem very useful and I’ve never seen anyone do that. You don’t need statistics of any kind to reject the model “the observation is zero” when the observation is not zero. Apart from that I agree that different hypothesis testing procedures do different things and their respective merits are debatable. My point was just that if you have a well-defined “null” model you can do different things with it. Using the same exact model.

travisjungroth 3y ago
> that doesn’t seem very useful and I’ve never seen anyone do that. Yes. That’s why it’s a straw man. I’m being sort of uncharitable in that description. My point is that’s a starting point. To go from there, you need to choose a model which will have parameters or priors.
- kgwgk 3y ago
  That’s where the « null hypothesis » comes in. If it fixes the parameters in the model you get a well-defined « null » model with a well-defined probability distribution for the observation and - just like you can take this null hypothesis model and do frequentist calculations with it - you can take this model and calculate a Bayes factor relative to some other model. (To be clear, if the null hypothesis doesn’t fully specify the parameters the preceding paragraph doesnt apply and the situation is more complex.)