5 ms·

I think there is some confusion going on. Nobody claims that there is the null hypothesis. I think you are fighting windmills. Let’s say you study P. You know

I think there is some confusion going on. Nobody claims that there is the null hypothesis. I think you are fighting windmills. Let’s say you study P. You know that P belongs to the family 𝒫. For example, 𝒫 = {N(μ,σ²): μ∈ℝ, σ²>0}. To be aware of 𝒫 is a prerequisite to do any sort of testing. For example, to test H0: μ=0 vs H1: μ≠0. After all, a typical hypothesis test is just a likehood ratio test. What you don’t have to know or even to assume existence of—if you don’t do Bayesian stuff—is a probability measure Π on 𝒫 (and its appropriate sigma-field). That’s the philosophical difference. But you have to have well-defined 𝒫 either way. The priors is Π. Existence of 𝒫 means that you have a family of models, but it doesn’t force you to assume any priors about those models. I don’t see how not having Π makes P∈𝒫 less of a model. You are still allowed to do conditional reasoning, eg the aforementioned type I and type II errors.

travisjungroth 3y ago
> Nobody claims that there is the null hypothesis. Google “the null hypothesis”. If you mean null model, then I’m not fighting against anyone. We all agree which null model to use is a choice to be made. Otherwise, I’m not even sure what you’re trying to convince me of at this point. I’ll restate the essence of my first comment more concisely. Bayes factors are a method of model comparison. You take the ratio of marginal likelihoods for two models given the data. Choosing a null model for this purpose requires more assumptions than doing null hypothesis testing with frequentist statistics. Mixing the schools of thought of Bayesian and frequentist makes things more confusing than operating within them individually. Bayes factors have other uses than null hypothesis testing. Maybe you straight up disagree with one of those sentences to the point you could quote it and say “this is wrong because…”. I have this feeling you got to my second paragraph on my first comment and started quoting stuff before reading it all the way through. Or maybe you just had some stuff you really wanted to talk about. Because my whole point was how calculating Bayes factors and the normal mentality of null hypothesis testing don’t play nice together, but can have other benefits. So a line by line comparison doesn’t really make sense.
- kgwgk 3y ago
  > Choosing a null model for this purpose requires more assumptions than doing null hypothesis testing with frequentist statistics. How so? I could choose the same null model that predicts that the observation is distributed as, say, a standard Gaussian. What additional assumptions are required?
  - travisjungroth 3y ago
    Mean and standard deviation up front, not from the sample. At least, that would go against my understanding of Bayes factors and how I’ve calculated them. You can do other stats, t-test for example, without declaring that up front.
    - kgwgk 3y ago
      I don’t understand what you say, sorry. If my choice of null model is p(x)=exp(-x^2/2) and I get some observation xobs I can do frequentist things with it - like calculating the p-value p(|x|>|xobs|) for example - and I can compare it with some alternative model p’(x) using the Bayes factor p(xobs)/p’(xobs). What does “Mean and standard deviation up front, not from the sample.” mean in this context?
      - travisjungroth 3y ago
        Mean and standard deviation up front would mean that you set your null as a normal distribution with a mean of 0 and a standard deviation of 2%, for example. This is different from saying that you’ll just say it’s normal, and take the mean and standard deviation from the sample. I mean, I could be wrong on this. You could do that if you want. I just think of Bayes factors as a competition and it needs to be “fair”. So it doesn’t make sense to let the null update as data comes in but not the alternative.
        kgwgk 3y ago
        Ok. As you said "Bayes factors work with comparing models." I don't think that there is a concept of "fairness" that prevents us from including in the comparison a very simple model. "There is no null model. What, 0% effect?" For example. But even if the true effect is zero you may get a non-zero observation because measurements are not perfect. If what you measure is precisely what you want to know there is no need for statistical analysis! Let's say for the sake of this example that you know that your measurement error is distributed normally with unit variance. "Ok, there was a non-zero effect." There was a non-zero _observation_. "That model loses since it put the probability of 0% at 1 and everything else at 0." It didn't. The prediction of the null model is that observations will be normally distributed around 0. That's precisely what the author of the blogpost complains about: that if your observation is 1 the prediction of the null model (close to 0) was more accurate than the prediction of the second model (somewhere between 1 and 10) and the observation favours the former over the latter.
amluto 3y ago
I think this approach to non-Bayesian hypothesis testing is dangerous or misleading. > Let’s say you study P. You know that P belongs to the family 𝒫. For example, 𝒫 = {N(μ,σ²): μ∈ℝ, σ²>0}. To be aware of 𝒫 is a prerequisite to do any sort of testing. Okay, sure: you’ve decided on a family of probability distributions. It’s surely an approximation (very few things you would test are actually Gaussian — for one thing, the negative tail often makes no sense). > For example, to test H0: μ=0 vs H1: μ≠0. After all, a typical hypothesis test is just a likehood ratio test. This is indeed the usual formulation. > What you don’t have to know or even to assume existence of—if you don’t do Bayesian stuff—is a probability measure Π on 𝒫 (and its appropriate sigma-field). … But you have to have well-defined 𝒫 either way. If 𝒫 is well defined, then there is some probability that H0 is true. But H0 occupies a lower-dimensional space than 𝒫 — it’s a measure-zero subset. Most probability measures 𝒫 (and all measures that are continuous on their parameters) give zero probability to H0. So (in Bayesian terms) H0 is a priori wrong w.p. 1. And in non-Bayesian terms, you’re calculating the likelihood of your measurements under two competing hypotheses, one of which is correct w.p. 0 even conditioned on one of the two hypotheses being correct. And this results in what I consider to be useless headline results: “This intervention has an effect with significance 0.02” — great, of course it has an effect. What is the effect? Can you say anything intelligent about effect size? Did you even try? “We did not find significant evidence that some intervention causes some undesirable effect” — great, but that’s actually a statement about your trial and conveys essentially no information about whether the effect is there. I can do a study with n=1 and fail to find significant evidence of anything! But I also learn nothing! Why didn’t you either (a) come up with an actual reasonable hypothesis and test that or (b) put some confidence bounds on the size of the undesirable effect. And you can do (b) without a Bayesian prior as long as it choose your hypothesis well. “Our data is inconsistent with the intervention causing the undesired effect in more than 0.001% of cases” with some clarification as to what “inconsistent” means.
- kgwgk 3y ago
  > Most probability measures 𝒫 (and all measures that are continuous on their parameters) give zero probability to H0. And some don't. > So (in Bayesian terms) H0 is a priori wrong w.p. 1. Or not. > And in non-Bayesian terms, you’re calculating the likelihood of your measurements under two competing hypotheses, one of which is correct w.p. 0 even conditioned on one of the two hypotheses being correct. If one of the two (H0 or H1) is correct and you don't know which one then it could be either... Of course is you knew a priori which one is correct you wouldn't be considering the other at all.
randomcarbloke 3y ago
>Nobody claims that there is the null hypothesis. I think you are fighting windmills. it says it right there in the blog post...
- LudwigNagasena 3y ago
  It provides a null hypothesis. You can compare two hypotheses without any of them designating absence of effect. You don't have to have "the null hypothesis" in some philosophical sense to do a t-test and whatnot.

travisjungroth 3y ago

> Nobody claims that there is the null hypothesis. Google “the null hypothesis”. If you mean null model, then I’m not fighting against anyone. We all agree which null model to use is a choice to be made. Otherwise, I’m not even sure what you’re trying to convince me of at this point. I’ll restate the essence of my first comment more concisely. Bayes factors are a method of model comparison. You take the ratio of marginal likelihoods for two models given the data. Choosing a null model for this purpose requires more assumptions than doing null hypothesis testing with frequentist statistics. Mixing the schools of thought of Bayesian and frequentist makes things more confusing than operating within them individually. Bayes factors have other uses than null hypothesis testing. Maybe you straight up disagree with one of those sentences to the point you could quote it and say “this is wrong because…”. I have this feeling you got to my second paragraph on my first comment and started quoting stuff before reading it all the way through. Or maybe you just had some stuff you really wanted to talk about. Because my whole point was how calculating Bayes factors and the normal mentality of null hypothesis testing don’t play nice together, but can have other benefits. So a line by line comparison doesn’t really make sense.

kgwgk 3y ago
> Choosing a null model for this purpose requires more assumptions than doing null hypothesis testing with frequentist statistics. How so? I could choose the same null model that predicts that the observation is distributed as, say, a standard Gaussian. What additional assumptions are required?
- travisjungroth 3y ago
  Mean and standard deviation up front, not from the sample. At least, that would go against my understanding of Bayes factors and how I’ve calculated them. You can do other stats, t-test for example, without declaring that up front.
  - kgwgk 3y ago
    I don’t understand what you say, sorry. If my choice of null model is p(x)=exp(-x^2/2) and I get some observation xobs I can do frequentist things with it - like calculating the p-value p(|x|>|xobs|) for example - and I can compare it with some alternative model p’(x) using the Bayes factor p(xobs)/p’(xobs). What does “Mean and standard deviation up front, not from the sample.” mean in this context?
    - travisjungroth 3y ago
      Mean and standard deviation up front would mean that you set your null as a normal distribution with a mean of 0 and a standard deviation of 2%, for example. This is different from saying that you’ll just say it’s normal, and take the mean and standard deviation from the sample. I mean, I could be wrong on this. You could do that if you want. I just think of Bayes factors as a competition and it needs to be “fair”. So it doesn’t make sense to let the null update as data comes in but not the alternative.
      - kgwgk 3y ago
        Ok. As you said "Bayes factors work with comparing models." I don't think that there is a concept of "fairness" that prevents us from including in the comparison a very simple model. "There is no null model. What, 0% effect?" For example. But even if the true effect is zero you may get a non-zero observation because measurements are not perfect. If what you measure is precisely what you want to know there is no need for statistical analysis! Let's say for the sake of this example that you know that your measurement error is distributed normally with unit variance. "Ok, there was a non-zero effect." There was a non-zero _observation_. "That model loses since it put the probability of 0% at 1 and everything else at 0." It didn't. The prediction of the null model is that observations will be normally distributed around 0. That's precisely what the author of the blogpost complains about: that if your observation is 1 the prediction of the null model (close to 0) was more accurate than the prediction of the second model (somewhere between 1 and 10) and the observation favours the former over the latter.

amluto 3y ago

I think this approach to non-Bayesian hypothesis testing is dangerous or misleading. > Let’s say you study P. You know that P belongs to the family 𝒫. For example, 𝒫 = {N(μ,σ²): μ∈ℝ, σ²>0}. To be aware of 𝒫 is a prerequisite to do any sort of testing. Okay, sure: you’ve decided on a family of probability distributions. It’s surely an approximation (very few things you would test are actually Gaussian — for one thing, the negative tail often makes no sense). > For example, to test H0: μ=0 vs H1: μ≠0. After all, a typical hypothesis test is just a likehood ratio test. This is indeed the usual formulation. > What you don’t have to know or even to assume existence of—if you don’t do Bayesian stuff—is a probability measure Π on 𝒫 (and its appropriate sigma-field). … But you have to have well-defined 𝒫 either way. If 𝒫 is well defined, then there is some probability that H0 is true. But H0 occupies a lower-dimensional space than 𝒫 — it’s a measure-zero subset. Most probability measures 𝒫 (and all measures that are continuous on their parameters) give zero probability to H0. So (in Bayesian terms) H0 is a priori wrong w.p. 1. And in non-Bayesian terms, you’re calculating the likelihood of your measurements under two competing hypotheses, one of which is correct w.p. 0 even conditioned on one of the two hypotheses being correct. And this results in what I consider to be useless headline results: “This intervention has an effect with significance 0.02” — great, of course it has an effect. What is the effect? Can you say anything intelligent about effect size? Did you even try? “We did not find significant evidence that some intervention causes some undesirable effect” — great, but that’s actually a statement about your trial and conveys essentially no information about whether the effect is there. I can do a study with n=1 and fail to find significant evidence of anything! But I also learn nothing! Why didn’t you either (a) come up with an actual reasonable hypothesis and test that or (b) put some confidence bounds on the size of the undesirable effect. And you can do (b) without a Bayesian prior as long as it choose your hypothesis well. “Our data is inconsistent with the intervention causing the undesired effect in more than 0.001% of cases” with some clarification as to what “inconsistent” means.

kgwgk 3y ago
> Most probability measures 𝒫 (and all measures that are continuous on their parameters) give zero probability to H0. And some don't. > So (in Bayesian terms) H0 is a priori wrong w.p. 1. Or not. > And in non-Bayesian terms, you’re calculating the likelihood of your measurements under two competing hypotheses, one of which is correct w.p. 0 even conditioned on one of the two hypotheses being correct. If one of the two (H0 or H1) is correct and you don't know which one then it could be either... Of course is you knew a priori which one is correct you wouldn't be considering the other at all.

randomcarbloke 3y ago

>Nobody claims that there is the null hypothesis. I think you are fighting windmills. it says it right there in the blog post...

LudwigNagasena 3y ago
It provides a null hypothesis. You can compare two hypotheses without any of them designating absence of effect. You don't have to have "the null hypothesis" in some philosophical sense to do a t-test and whatnot.