4 ms·

> Bayes factors work with comparing models. So does the traditional Neyman–Pearson hypothesis testing. > There is no null model. Why can’t there be? > What,

> Bayes factors work with comparing models. So does the traditional Neyman–Pearson hypothesis testing. > There is no null model. Why can’t there be? > What, 0% effect? Ok, there was a non-zero effect. That model loses since it put the probability of 0% at 1 and everything else at 0%. Well, if your null hypothesis is deterministic and says 0% effect, getting anything other than 0% absolutely will make you reject the null hypothesis. But most of the time hypotheses are not deterministic. Usually you sample random variables. > And if you do anything else, you’re encoding some amount of belief into the model, some judgment you’ve made. Traditional hypothesis testing is a particular case of minimising risk, ie the expected value of your loss given possible models and your decision rule. You don’t assume any belief on the probability of a specific model to be true, thus I think it is incorrect to claim that you encode a belief. You don’t even claim that it can be measured. Of course, that makes it impossible to quantify the risk over all possible models. Thus, you only deal with Type I and Type II errors, which values presumes that the null or alternative hypothesis is correct. If you have a probability measure for models, you can simply average your risks over it and get what is known as Bayes risk. That would be encoding some belief.

travisjungroth 3y ago
You've sliced up what I've said to the point it doesn't really make sense. This is exactly what I said was confusing. I'm talking about Bayes factors and you're talking about null hypothesis testing. I'll just answer your question, why there can't be a null model. You can have a hypothesis that represents all differences between groups are due to chance. To make this a statistical model, something that can calculate the probability of an event, you have to make assumptions. Maybe it's just about the distribution. Maybe it's independence. But, it's always something. You said it yourself "You don’t assume any belief on the probability of a specific model to be true." To be a statistical model, to calculate the probability of an event, to calculate marginal likelihoods, to calculate Bayes factors, you have to do that. This is largely a philosophical point. You can have a null model. Something you pick to represent "no effect". But there's not the null, this belief free model that's categorically different from a model with priors. If there's a belief-free model that can give a marginal likelihood, then I'm wrong. I'd also very much like to know about it.
- LudwigNagasena 3y ago
  I think there is some confusion going on. Nobody claims that there is the null hypothesis. I think you are fighting windmills. Let’s say you study P. You know that P belongs to the family 𝒫. For example, 𝒫 = {N(μ,σ²): μ∈ℝ, σ²>0}. To be aware of 𝒫 is a prerequisite to do any sort of testing. For example, to test H0: μ=0 vs H1: μ≠0. After all, a typical hypothesis test is just a likehood ratio test. What you don’t have to know or even to assume existence of—if you don’t do Bayesian stuff—is a probability measure Π on 𝒫 (and its appropriate sigma-field). That’s the philosophical difference. But you have to have well-defined 𝒫 either way. The priors is Π. Existence of 𝒫 means that you have a family of models, but it doesn’t force you to assume any priors about those models. I don’t see how not having Π makes P∈𝒫 less of a model. You are still allowed to do conditional reasoning, eg the aforementioned type I and type II errors.
  - travisjungroth 3y ago
    > Nobody claims that there is the null hypothesis. Google “the null hypothesis”. If you mean null model, then I’m not fighting against anyone. We all agree which null model to use is a choice to be made. Otherwise, I’m not even sure what you’re trying to convince me of at this point. I’ll restate the essence of my first comment more concisely. Bayes factors are a method of model comparison. You take the ratio of marginal likelihoods for two models given the data. Choosing a null model for this purpose requires more assumptions than doing null hypothesis testing with frequentist statistics. Mixing the schools of thought of Bayesian and frequentist makes things more confusing than operating within them individually. Bayes factors have other uses than null hypothesis testing. Maybe you straight up disagree with one of those sentences to the point you could quote it and say “this is wrong because…”. I have this feeling you got to my second paragraph on my first comment and started quoting stuff before reading it all the way through. Or maybe you just had some stuff you really wanted to talk about. Because my whole point was how calculating Bayes factors and the normal mentality of null hypothesis testing don’t play nice together, but can have other benefits. So a line by line comparison doesn’t really make sense.
    - kgwgk 3y ago

travisjungroth 3y ago

You've sliced up what I've said to the point it doesn't really make sense. This is exactly what I said was confusing. I'm talking about Bayes factors and you're talking about null hypothesis testing. I'll just answer your question, why there can't be a null model. You can have a hypothesis that represents all differences between groups are due to chance. To make this a statistical model, something that can calculate the probability of an event, you have to make assumptions. Maybe it's just about the distribution. Maybe it's independence. But, it's always something. You said it yourself "You don’t assume any belief on the probability of a specific model to be true." To be a statistical model, to calculate the probability of an event, to calculate marginal likelihoods, to calculate Bayes factors, you have to do that. This is largely a philosophical point. You can have a null model. Something you pick to represent "no effect". But there's not the null, this belief free model that's categorically different from a model with priors. If there's a belief-free model that can give a marginal likelihood, then I'm wrong. I'd also very much like to know about it.

LudwigNagasena 3y ago
I think there is some confusion going on. Nobody claims that there is the null hypothesis. I think you are fighting windmills. Let’s say you study P. You know that P belongs to the family 𝒫. For example, 𝒫 = {N(μ,σ²): μ∈ℝ, σ²>0}. To be aware of 𝒫 is a prerequisite to do any sort of testing. For example, to test H0: μ=0 vs H1: μ≠0. After all, a typical hypothesis test is just a likehood ratio test. What you don’t have to know or even to assume existence of—if you don’t do Bayesian stuff—is a probability measure Π on 𝒫 (and its appropriate sigma-field). That’s the philosophical difference. But you have to have well-defined 𝒫 either way. The priors is Π. Existence of 𝒫 means that you have a family of models, but it doesn’t force you to assume any priors about those models. I don’t see how not having Π makes P∈𝒫 less of a model. You are still allowed to do conditional reasoning, eg the aforementioned type I and type II errors.
- travisjungroth 3y ago
  > Nobody claims that there is the null hypothesis. Google “the null hypothesis”. If you mean null model, then I’m not fighting against anyone. We all agree which null model to use is a choice to be made. Otherwise, I’m not even sure what you’re trying to convince me of at this point. I’ll restate the essence of my first comment more concisely. Bayes factors are a method of model comparison. You take the ratio of marginal likelihoods for two models given the data. Choosing a null model for this purpose requires more assumptions than doing null hypothesis testing with frequentist statistics. Mixing the schools of thought of Bayesian and frequentist makes things more confusing than operating within them individually. Bayes factors have other uses than null hypothesis testing. Maybe you straight up disagree with one of those sentences to the point you could quote it and say “this is wrong because…”. I have this feeling you got to my second paragraph on my first comment and started quoting stuff before reading it all the way through. Or maybe you just had some stuff you really wanted to talk about. Because my whole point was how calculating Bayes factors and the normal mentality of null hypothesis testing don’t play nice together, but can have other benefits. So a line by line comparison doesn’t really make sense.
  - kgwgk 3y ago