4 ms·

I think this approach to non-Bayesian hypothesis testing is dangerous or misleading. > Let’s say you study P. You know that P belongs to the family 𝒫. For exa

by amluto 3y ago

I think this approach to non-Bayesian hypothesis testing is dangerous or misleading. > Let’s say you study P. You know that P belongs to the family 𝒫. For example, 𝒫 = {N(μ,σ²): μ∈ℝ, σ²>0}. To be aware of 𝒫 is a prerequisite to do any sort of testing. Okay, sure: you’ve decided on a family of probability distributions. It’s surely an approximation (very few things you would test are actually Gaussian — for one thing, the negative tail often makes no sense). > For example, to test H0: μ=0 vs H1: μ≠0. After all, a typical hypothesis test is just a likehood ratio test. This is indeed the usual formulation. > What you don’t have to know or even to assume existence of—if you don’t do Bayesian stuff—is a probability measure Π on 𝒫 (and its appropriate sigma-field). … But you have to have well-defined 𝒫 either way. If 𝒫 is well defined, then there is some probability that H0 is true. But H0 occupies a lower-dimensional space than 𝒫 — it’s a measure-zero subset. Most probability measures 𝒫 (and all measures that are continuous on their parameters) give zero probability to H0. So (in Bayesian terms) H0 is a priori wrong w.p. 1. And in non-Bayesian terms, you’re calculating the likelihood of your measurements under two competing hypotheses, one of which is correct w.p. 0 even conditioned on one of the two hypotheses being correct. And this results in what I consider to be useless headline results: “This intervention has an effect with significance 0.02” — great, of course it has an effect. What is the effect? Can you say anything intelligent about effect size? Did you even try? “We did not find significant evidence that some intervention causes some undesirable effect” — great, but that’s actually a statement about your trial and conveys essentially no information about whether the effect is there. I can do a study with n=1 and fail to find significant evidence of anything! But I also learn nothing! Why didn’t you either (a) come up with an actual reasonable hypothesis and test that or (b) put some confidence bounds on the size of the undesirable effect. And you can do (b) without a Bayesian prior as long as it choose your hypothesis well. “Our data is inconsistent with the intervention causing the undesired effect in more than 0.001% of cases” with some clarification as to what “inconsistent” means.

kgwgk 3y ago
> Most probability measures 𝒫 (and all measures that are continuous on their parameters) give zero probability to H0. And some don't. > So (in Bayesian terms) H0 is a priori wrong w.p. 1. Or not. > And in non-Bayesian terms, you’re calculating the likelihood of your measurements under two competing hypotheses, one of which is correct w.p. 0 even conditioned on one of the two hypotheses being correct. If one of the two (H0 or H1) is correct and you don't know which one then it could be either... Of course is you knew a priori which one is correct you wouldn't be considering the other at all.
LudwigNagasena 3y ago
What is dangerous or misleading? It is a formulation I learned in my graduate statistics course. It is the one that makes the most sense and requires the least amount of handwaving. > But H0 occupies a lower-dimensional space than 𝒫 — it’s a measure-zero subset. In non-Bayesian framework your hypotheses don't have to be a part of a measurable structure at all. Nevertheless, if you have a measure, it doesn't have to be a zero at every point. I think it is quite intuitive to see. Let's look at two questions: (1) Does X have an effect? (2) How large is that effect? If your prior puts a non-zero probability that the answer to the first question is "No", then priors for the second question will have non-zero at point 0, even though the probability of any other point may be zero. > And this results in what I consider to be useless headline results These headlines don't have anything to do with Bayesian vs non-Bayesian as far as I see. People not doing power analysis is a people problem, not an issue with a statistical framework.
- amluto 3y ago
  I’ve never taken graduate statistics, but I’m generally not impressed with the state of statistics or statistics education. Just because almost every university teaches almost every student this (and most published papers work this way) doesn’t mean it’s good. Intuitive, sure. Wise, not so much. > Let's look at two questions: (1) Does X have an effect? (2) How large is that effect? If your prior puts a non-zero probability that the answer to the first question is "No", then priors for the second question will have non-zero at point 0, even though the probability of any other point may be zero. Even ignoring whether it makes sense to have a nonzero prior for X having no effect, it’s generally not useful to learn the answer. An arbitrarily small effect is, for practical purposes, indistinguishable from no effect — an experiment that is intended to be useful should say something about the size of an effect. If a pill helps depression enough to be worth taking the pill, that’s one thing. If it helps in the sense that you could dose literally everyone in the world and one person would feel very slightly better for a day, that’s not helpful. Similarly, the existence of an effect says nothing about the sign of the effect. So I think hypotheses being tested should be useful. If you want to determine whether something is useful, at least set a threshold for usefulness and test that. Or come up with a quantitative measure. The Bayesian-vs-frequentist debate is IMO somewhat secondary to this except insofar as it seems less common to make worthless but mathematically correct Bayesian tests because thinking about priors at all requires some acknowledgment of whether a prior is remotely plausible. Also, how exactly can you make a well defined experiment that can confirm the null hypothesis without putting something resembling a prior on the non-null hypothesis if the non-null hypothesis contains distributions that are arbitrarily close to null? I’m sure it’s doable, but it seems likely to be pretty messy if you dig in. (There are exceptions. For example, the existence of a neutrino mass is very interesting irrespective of what that mass is. But even then, physics results like this generally put bounds on a value that is hypothesized to be zero instead of merely testing for zeroness, because every experiment has finite power to detect small effects, and the readers of an outcome of an experiment should care more about the detection limits than, say, the number of dollars the experiment cost.)

by amluto 3y ago

kgwgk 3y ago

> Most probability measures 𝒫 (and all measures that are continuous on their parameters) give zero probability to H0. And some don't. > So (in Bayesian terms) H0 is a priori wrong w.p. 1. Or not. > And in non-Bayesian terms, you’re calculating the likelihood of your measurements under two competing hypotheses, one of which is correct w.p. 0 even conditioned on one of the two hypotheses being correct. If one of the two (H0 or H1) is correct and you don't know which one then it could be either... Of course is you knew a priori which one is correct you wouldn't be considering the other at all.

LudwigNagasena 3y ago

What is dangerous or misleading? It is a formulation I learned in my graduate statistics course. It is the one that makes the most sense and requires the least amount of handwaving. > But H0 occupies a lower-dimensional space than 𝒫 — it’s a measure-zero subset. In non-Bayesian framework your hypotheses don't have to be a part of a measurable structure at all. Nevertheless, if you have a measure, it doesn't have to be a zero at every point. I think it is quite intuitive to see. Let's look at two questions: (1) Does X have an effect? (2) How large is that effect? If your prior puts a non-zero probability that the answer to the first question is "No", then priors for the second question will have non-zero at point 0, even though the probability of any other point may be zero. > And this results in what I consider to be useless headline results These headlines don't have anything to do with Bayesian vs non-Bayesian as far as I see. People not doing power analysis is a people problem, not an issue with a statistical framework.

amluto 3y ago
I’ve never taken graduate statistics, but I’m generally not impressed with the state of statistics or statistics education. Just because almost every university teaches almost every student this (and most published papers work this way) doesn’t mean it’s good. Intuitive, sure. Wise, not so much. > Let's look at two questions: (1) Does X have an effect? (2) How large is that effect? If your prior puts a non-zero probability that the answer to the first question is "No", then priors for the second question will have non-zero at point 0, even though the probability of any other point may be zero. Even ignoring whether it makes sense to have a nonzero prior for X having no effect, it’s generally not useful to learn the answer. An arbitrarily small effect is, for practical purposes, indistinguishable from no effect — an experiment that is intended to be useful should say something about the size of an effect. If a pill helps depression enough to be worth taking the pill, that’s one thing. If it helps in the sense that you could dose literally everyone in the world and one person would feel very slightly better for a day, that’s not helpful. Similarly, the existence of an effect says nothing about the sign of the effect. So I think hypotheses being tested should be useful. If you want to determine whether something is useful, at least set a threshold for usefulness and test that. Or come up with a quantitative measure. The Bayesian-vs-frequentist debate is IMO somewhat secondary to this except insofar as it seems less common to make worthless but mathematically correct Bayesian tests because thinking about priors at all requires some acknowledgment of whether a prior is remotely plausible. Also, how exactly can you make a well defined experiment that can confirm the null hypothesis without putting something resembling a prior on the non-null hypothesis if the non-null hypothesis contains distributions that are arbitrarily close to null? I’m sure it’s doable, but it seems likely to be pretty messy if you dig in. (There are exceptions. For example, the existence of a neutrino mass is very interesting irrespective of what that mass is. But even then, physics results like this generally put bounds on a value that is hypothesized to be zero instead of merely testing for zeroness, because every experiment has finite power to detect small effects, and the readers of an outcome of an experiment should care more about the detection limits than, say, the number of dollars the experiment cost.)