5 ms·

> Nobody claims that there is the null hypothesis. Google “the null hypothesis”. If you mean null model, then I’m not fighting against anyone. We all agree wh

> Nobody claims that there is the null hypothesis. Google “the null hypothesis”. If you mean null model, then I’m not fighting against anyone. We all agree which null model to use is a choice to be made. Otherwise, I’m not even sure what you’re trying to convince me of at this point. I’ll restate the essence of my first comment more concisely. Bayes factors are a method of model comparison. You take the ratio of marginal likelihoods for two models given the data. Choosing a null model for this purpose requires more assumptions than doing null hypothesis testing with frequentist statistics. Mixing the schools of thought of Bayesian and frequentist makes things more confusing than operating within them individually. Bayes factors have other uses than null hypothesis testing. Maybe you straight up disagree with one of those sentences to the point you could quote it and say “this is wrong because…”. I have this feeling you got to my second paragraph on my first comment and started quoting stuff before reading it all the way through. Or maybe you just had some stuff you really wanted to talk about. Because my whole point was how calculating Bayes factors and the normal mentality of null hypothesis testing don’t play nice together, but can have other benefits. So a line by line comparison doesn’t really make sense.

kgwgk 3y ago
> Choosing a null model for this purpose requires more assumptions than doing null hypothesis testing with frequentist statistics. How so? I could choose the same null model that predicts that the observation is distributed as, say, a standard Gaussian. What additional assumptions are required?
- travisjungroth 3y ago
  Mean and standard deviation up front, not from the sample. At least, that would go against my understanding of Bayes factors and how I’ve calculated them. You can do other stats, t-test for example, without declaring that up front.
  - kgwgk 3y ago
    I don’t understand what you say, sorry. If my choice of null model is p(x)=exp(-x^2/2) and I get some observation xobs I can do frequentist things with it - like calculating the p-value p(|x|>|xobs|) for example - and I can compare it with some alternative model p’(x) using the Bayes factor p(xobs)/p’(xobs). What does “Mean and standard deviation up front, not from the sample.” mean in this context?
    - travisjungroth 3y ago
      Mean and standard deviation up front would mean that you set your null as a normal distribution with a mean of 0 and a standard deviation of 2%, for example. This is different from saying that you’ll just say it’s normal, and take the mean and standard deviation from the sample. I mean, I could be wrong on this. You could do that if you want. I just think of Bayes factors as a competition and it needs to be “fair”. So it doesn’t make sense to let the null update as data comes in but not the alternative.
      - kgwgk 3y ago
        Ok. As you said "Bayes factors work with comparing models." I don't think that there is a concept of "fairness" that prevents us from including in the comparison a very simple model. "There is no null model. What, 0% effect?" For example. But even if the true effect is zero you may get a non-zero observation because measurements are not perfect. If what you measure is precisely what you want to know there is no need for statistical analysis! Let's say for the sake of this example that you know that your measurement error is distributed normally with unit variance. "Ok, there was a non-zero effect." There was a non-zero _observation_. "That model loses since it put the probability of 0% at 1 and everything else at 0." It didn't. The prediction of the null model is that observations will be normally distributed around 0. That's precisely what the author of the blogpost complains about: that if your observation is 1 the prediction of the null model (close to 0) was more accurate than the prediction of the second model (somewhere between 1 and 10) and the observation favours the former over the latter.
        travisjungroth 3y ago
LudwigNagasena 3y ago
> Google “the null hypothesis”. > If you mean null model, then I’m not fighting against anyone. We all agree which null model to use is a choice to be made. I don’t understand what difference you are trying to imply by drawing a distinction between a null model and a null hypothesis. > Otherwise, I’m not even sure what you’re trying to convince me of at this point. I’ll restate the essence of my first comment more concisely. I will try to make it as clear as possible. > Bayes factors are a method of model comparison. Are you implying that hypothesis testing isn’t? That’s just false. And I’ve explained why. > You take the ratio of marginal likelihoods for two models given the data. Choosing a null model for this purpose requires more assumptions than doing null hypothesis testing with frequentist statistics. And in frequentist statistics you just calculate likehood because you can’t integrate over your model probabilities to get marginal likehood because you don’t assume your models to have a probability of being true. That’s the only extra assumption you have in Bayesian statistics. Everything else is the same. If you are saying that there are some other extra assumptions, that’s just false as I’ve explained in my previous comments. There are no extra assumptions for a “null model” beyond putting a prior on it. > Mixing the schools of thought of Bayesian and frequentist makes things more confusing than operating within them individually. Bayes factors have other uses than null hypothesis testing. There is no any confusing “mixing”. It’s just statistical decision theory. In the frequentists approach you calculate the risk of your decision rule for each model and call it a day. In the Bayesian approach you go one step further and average your risks using your priors to get the “total” Bayes risk. Both approaches have uses other than null hypothesis testing. Null hypothesis testing is just a particular case of a decision problem with a 0-1 loss function. The loss is 0 if you have chosen the correct hypothesis and it is 1 if you have encountered type I or type II error.
- travisjungroth 3y ago
  > > Bayes factors are a method of model comparison. > Are you implying that hypothesis testing isn’t? No.
  - LudwigNagasena 3y ago
    Then I don't get the meaning of this: > Bayes factors work with comparing models. There is no null model. What, 0% effect? Ok, there was a non-zero effect. That model loses since it put the probability of 0% at 1 and everything else at 0. And if you do anything else, you’re encoding some amount of belief into the model, some judgment you’ve made. > So, you need to pick two models and compare them. I’m not saying this is right for science. It’s working well for my purposes. One model meaning “as planned”, one model meaning “not as planned”, use the Bayes factor to decide if things are going as planned. But you do need to be explicit about what models you’re comparing. You have to be able to just put some data in and get a probability back, or it’s not going to work. It is the same way with traditional hypothesis testing. You take two models and compare their likehood.

kgwgk 3y ago

> Choosing a null model for this purpose requires more assumptions than doing null hypothesis testing with frequentist statistics. How so? I could choose the same null model that predicts that the observation is distributed as, say, a standard Gaussian. What additional assumptions are required?

travisjungroth 3y ago
Mean and standard deviation up front, not from the sample. At least, that would go against my understanding of Bayes factors and how I’ve calculated them. You can do other stats, t-test for example, without declaring that up front.
- kgwgk 3y ago
  I don’t understand what you say, sorry. If my choice of null model is p(x)=exp(-x^2/2) and I get some observation xobs I can do frequentist things with it - like calculating the p-value p(|x|>|xobs|) for example - and I can compare it with some alternative model p’(x) using the Bayes factor p(xobs)/p’(xobs). What does “Mean and standard deviation up front, not from the sample.” mean in this context?
  - travisjungroth 3y ago
    Mean and standard deviation up front would mean that you set your null as a normal distribution with a mean of 0 and a standard deviation of 2%, for example. This is different from saying that you’ll just say it’s normal, and take the mean and standard deviation from the sample. I mean, I could be wrong on this. You could do that if you want. I just think of Bayes factors as a competition and it needs to be “fair”. So it doesn’t make sense to let the null update as data comes in but not the alternative.
    - kgwgk 3y ago
      Ok. As you said "Bayes factors work with comparing models." I don't think that there is a concept of "fairness" that prevents us from including in the comparison a very simple model. "There is no null model. What, 0% effect?" For example. But even if the true effect is zero you may get a non-zero observation because measurements are not perfect. If what you measure is precisely what you want to know there is no need for statistical analysis! Let's say for the sake of this example that you know that your measurement error is distributed normally with unit variance. "Ok, there was a non-zero effect." There was a non-zero _observation_. "That model loses since it put the probability of 0% at 1 and everything else at 0." It didn't. The prediction of the null model is that observations will be normally distributed around 0. That's precisely what the author of the blogpost complains about: that if your observation is 1 the prediction of the null model (close to 0) was more accurate than the prediction of the second model (somewhere between 1 and 10) and the observation favours the former over the latter.
      - travisjungroth 3y ago

LudwigNagasena 3y ago

> Google “the null hypothesis”. > If you mean null model, then I’m not fighting against anyone. We all agree which null model to use is a choice to be made. I don’t understand what difference you are trying to imply by drawing a distinction between a null model and a null hypothesis. > Otherwise, I’m not even sure what you’re trying to convince me of at this point. I’ll restate the essence of my first comment more concisely. I will try to make it as clear as possible. > Bayes factors are a method of model comparison. Are you implying that hypothesis testing isn’t? That’s just false. And I’ve explained why. > You take the ratio of marginal likelihoods for two models given the data. Choosing a null model for this purpose requires more assumptions than doing null hypothesis testing with frequentist statistics. And in frequentist statistics you just calculate likehood because you can’t integrate over your model probabilities to get marginal likehood because you don’t assume your models to have a probability of being true. That’s the only extra assumption you have in Bayesian statistics. Everything else is the same. If you are saying that there are some other extra assumptions, that’s just false as I’ve explained in my previous comments. There are no extra assumptions for a “null model” beyond putting a prior on it. > Mixing the schools of thought of Bayesian and frequentist makes things more confusing than operating within them individually. Bayes factors have other uses than null hypothesis testing. There is no any confusing “mixing”. It’s just statistical decision theory. In the frequentists approach you calculate the risk of your decision rule for each model and call it a day. In the Bayesian approach you go one step further and average your risks using your priors to get the “total” Bayes risk. Both approaches have uses other than null hypothesis testing. Null hypothesis testing is just a particular case of a decision problem with a 0-1 loss function. The loss is 0 if you have chosen the correct hypothesis and it is 1 if you have encountered type I or type II error.

travisjungroth 3y ago
> > Bayes factors are a method of model comparison. > Are you implying that hypothesis testing isn’t? No.
- LudwigNagasena 3y ago
  Then I don't get the meaning of this: > Bayes factors work with comparing models. There is no null model. What, 0% effect? Ok, there was a non-zero effect. That model loses since it put the probability of 0% at 1 and everything else at 0. And if you do anything else, you’re encoding some amount of belief into the model, some judgment you’ve made. > So, you need to pick two models and compare them. I’m not saying this is right for science. It’s working well for my purposes. One model meaning “as planned”, one model meaning “not as planned”, use the Bayes factor to decide if things are going as planned. But you do need to be explicit about what models you’re comparing. You have to be able to just put some data in and get a probability back, or it’s not going to work. It is the same way with traditional hypothesis testing. You take two models and compare their likehood.