5 ms·

Mean and standard deviation up front would mean that you set your null as a normal distribution with a mean of 0 and a standard deviation of 2%, for example. Th

Mean and standard deviation up front would mean that you set your null as a normal distribution with a mean of 0 and a standard deviation of 2%, for example. This is different from saying that you’ll just say it’s normal, and take the mean and standard deviation from the sample. I mean, I could be wrong on this. You could do that if you want. I just think of Bayes factors as a competition and it needs to be “fair”. So it doesn’t make sense to let the null update as data comes in but not the alternative.

kgwgk 3y ago
Ok. As you said "Bayes factors work with comparing models." I don't think that there is a concept of "fairness" that prevents us from including in the comparison a very simple model. "There is no null model. What, 0% effect?" For example. But even if the true effect is zero you may get a non-zero observation because measurements are not perfect. If what you measure is precisely what you want to know there is no need for statistical analysis! Let's say for the sake of this example that you know that your measurement error is distributed normally with unit variance. "Ok, there was a non-zero effect." There was a non-zero _observation_. "That model loses since it put the probability of 0% at 1 and everything else at 0." It didn't. The prediction of the null model is that observations will be normally distributed around 0. That's precisely what the author of the blogpost complains about: that if your observation is 1 the prediction of the null model (close to 0) was more accurate than the prediction of the second model (somewhere between 1 and 10) and the observation favours the former over the latter.
- travisjungroth 3y ago
  As far as I see it, you have three options: Pick a measure. That’s what I mean by “effect is 0%”. It’s a straw man here. Pick a fully specified model. This is a model that, up front, you could ask what is the probability of event E? For a normal distribution, this would require choosing concrete mean and standard deviation. Pick an under-specified model. This would be that it’s normal, but you don’t pick the mean and standard deviation. You pull them from the sample. As I’ve described it here, you can’t get P(E) from that. The expectation from our alternative hypothesis and model is that it’s fully formed before we look at the data. It’s a choice whether you want that to be the case or not with the null model. “Fair” as I’m describing it is that you would pick something.
  - kgwgk 3y ago
    A> Pick a measure. That’s what I mean by “effect is 0%”. It’s a straw man here. I don't understand what you mean by "pick a measure" but maybe the "it’s a straw man here" (that I don't really understand either) indicates that looking at the other two options is enough. B> Pick a fully specified model. This is a model that, up front, you could ask what is the probability of event E? For a normal distribution, this would require choosing concrete mean and standard deviation. Ok. That seems to describe a simple classical null hypothesis like the example I gave in my previous comment. The underlying thing of interest is zero and the sampling distribution for the data is normally distributed around zero. C> Pick an under-specified model. This would be that it’s normal, but you don’t pick the mean and standard deviation. You pull them from the sample. As I’ve described it here, you can’t get P(E) from that. That is not the kind of null hypothesis I gave in my example, I think we can agree on that. > The expectation from our alternative hypothesis and model is that it’s fully formed before we look at the data. I don't understand that sentence. What is "it" that is fully formed before we look at the data? The alternative hypothesis and model? > It’s a choice whether you want that to be the case or not with the null model. What is "that"? Being formed before we look at the data? (In that case I hope that the null model I described would satisfy that.) > “Fair” as I’m describing it is that you would pick something. Pick something of what? I'm completely lost, I'm afraid. I'm just saying that I can have a null model of the form B like in the example "the underlying thing is zero and the data generated by this model has a probability distribution p(x)=exp(-x^2/2)". And I can compare that model it with any other model described by a distribution probability for the underlying thing which, taking into account the measurement error, results in a probability distribution p'(x) for the data generated.
    - travisjungroth 3y ago
      “Pick a measure” just meant that you’re predicting the difference will be exactly 0%. P(0) = 1. The difference between a Bayes factor and a likelihood ratio is Bayes factor uses the marginal likelihood. So you need to pick your parameters ahead of time, weighted by priors. With a likelihood ratio, you can use the best parameters given the data. You can do the likelihood ratio in an objective way, because you’re choosing whatever has the least error given the data. Bayes factor you can’t be totally objective. You need to choose ahead. Upside is it reduces overfitting.

kgwgk 3y ago

Ok. As you said "Bayes factors work with comparing models." I don't think that there is a concept of "fairness" that prevents us from including in the comparison a very simple model. "There is no null model. What, 0% effect?" For example. But even if the true effect is zero you may get a non-zero observation because measurements are not perfect. If what you measure is precisely what you want to know there is no need for statistical analysis! Let's say for the sake of this example that you know that your measurement error is distributed normally with unit variance. "Ok, there was a non-zero effect." There was a non-zero _observation_. "That model loses since it put the probability of 0% at 1 and everything else at 0." It didn't. The prediction of the null model is that observations will be normally distributed around 0. That's precisely what the author of the blogpost complains about: that if your observation is 1 the prediction of the null model (close to 0) was more accurate than the prediction of the second model (somewhere between 1 and 10) and the observation favours the former over the latter.

travisjungroth 3y ago
As far as I see it, you have three options: Pick a measure. That’s what I mean by “effect is 0%”. It’s a straw man here. Pick a fully specified model. This is a model that, up front, you could ask what is the probability of event E? For a normal distribution, this would require choosing concrete mean and standard deviation. Pick an under-specified model. This would be that it’s normal, but you don’t pick the mean and standard deviation. You pull them from the sample. As I’ve described it here, you can’t get P(E) from that. The expectation from our alternative hypothesis and model is that it’s fully formed before we look at the data. It’s a choice whether you want that to be the case or not with the null model. “Fair” as I’m describing it is that you would pick something.
- kgwgk 3y ago
  A> Pick a measure. That’s what I mean by “effect is 0%”. It’s a straw man here. I don't understand what you mean by "pick a measure" but maybe the "it’s a straw man here" (that I don't really understand either) indicates that looking at the other two options is enough. B> Pick a fully specified model. This is a model that, up front, you could ask what is the probability of event E? For a normal distribution, this would require choosing concrete mean and standard deviation. Ok. That seems to describe a simple classical null hypothesis like the example I gave in my previous comment. The underlying thing of interest is zero and the sampling distribution for the data is normally distributed around zero. C> Pick an under-specified model. This would be that it’s normal, but you don’t pick the mean and standard deviation. You pull them from the sample. As I’ve described it here, you can’t get P(E) from that. That is not the kind of null hypothesis I gave in my example, I think we can agree on that. > The expectation from our alternative hypothesis and model is that it’s fully formed before we look at the data. I don't understand that sentence. What is "it" that is fully formed before we look at the data? The alternative hypothesis and model? > It’s a choice whether you want that to be the case or not with the null model. What is "that"? Being formed before we look at the data? (In that case I hope that the null model I described would satisfy that.) > “Fair” as I’m describing it is that you would pick something. Pick something of what? I'm completely lost, I'm afraid. I'm just saying that I can have a null model of the form B like in the example "the underlying thing is zero and the data generated by this model has a probability distribution p(x)=exp(-x^2/2)". And I can compare that model it with any other model described by a distribution probability for the underlying thing which, taking into account the measurement error, results in a probability distribution p'(x) for the data generated.
  - travisjungroth 3y ago
    “Pick a measure” just meant that you’re predicting the difference will be exactly 0%. P(0) = 1. The difference between a Bayes factor and a likelihood ratio is Bayes factor uses the marginal likelihood. So you need to pick your parameters ahead of time, weighted by priors. With a likelihood ratio, you can use the best parameters given the data. You can do the likelihood ratio in an objective way, because you’re choosing whatever has the least error given the data. Bayes factor you can’t be totally objective. You need to choose ahead. Upside is it reduces overfitting.