3 ms·

What is dangerous or misleading? It is a formulation I learned in my graduate statistics course. It is the one that makes the most sense and requires the least

What is dangerous or misleading? It is a formulation I learned in my graduate statistics course. It is the one that makes the most sense and requires the least amount of handwaving. > But H0 occupies a lower-dimensional space than 𝒫 — it’s a measure-zero subset. In non-Bayesian framework your hypotheses don't have to be a part of a measurable structure at all. Nevertheless, if you have a measure, it doesn't have to be a zero at every point. I think it is quite intuitive to see. Let's look at two questions: (1) Does X have an effect? (2) How large is that effect? If your prior puts a non-zero probability that the answer to the first question is "No", then priors for the second question will have non-zero at point 0, even though the probability of any other point may be zero. > And this results in what I consider to be useless headline results These headlines don't have anything to do with Bayesian vs non-Bayesian as far as I see. People not doing power analysis is a people problem, not an issue with a statistical framework.

amluto 3y ago
I’ve never taken graduate statistics, but I’m generally not impressed with the state of statistics or statistics education. Just because almost every university teaches almost every student this (and most published papers work this way) doesn’t mean it’s good. Intuitive, sure. Wise, not so much. > Let's look at two questions: (1) Does X have an effect? (2) How large is that effect? If your prior puts a non-zero probability that the answer to the first question is "No", then priors for the second question will have non-zero at point 0, even though the probability of any other point may be zero. Even ignoring whether it makes sense to have a nonzero prior for X having no effect, it’s generally not useful to learn the answer. An arbitrarily small effect is, for practical purposes, indistinguishable from no effect — an experiment that is intended to be useful should say something about the size of an effect. If a pill helps depression enough to be worth taking the pill, that’s one thing. If it helps in the sense that you could dose literally everyone in the world and one person would feel very slightly better for a day, that’s not helpful. Similarly, the existence of an effect says nothing about the sign of the effect. So I think hypotheses being tested should be useful. If you want to determine whether something is useful, at least set a threshold for usefulness and test that. Or come up with a quantitative measure. The Bayesian-vs-frequentist debate is IMO somewhat secondary to this except insofar as it seems less common to make worthless but mathematically correct Bayesian tests because thinking about priors at all requires some acknowledgment of whether a prior is remotely plausible. Also, how exactly can you make a well defined experiment that can confirm the null hypothesis without putting something resembling a prior on the non-null hypothesis if the non-null hypothesis contains distributions that are arbitrarily close to null? I’m sure it’s doable, but it seems likely to be pretty messy if you dig in. (There are exceptions. For example, the existence of a neutrino mass is very interesting irrespective of what that mass is. But even then, physics results like this generally put bounds on a value that is hypothesized to be zero instead of merely testing for zeroness, because every experiment has finite power to detect small effects, and the readers of an outcome of an experiment should care more about the detection limits than, say, the number of dollars the experiment cost.)
- LudwigNagasena 3y ago
  > it’s generally not useful to learn the answer The point of priors is to represent beliefs about reality. It would certainly be surprising if you had a zero prior for neutrino to have no charge. Why cannot people have similar beliefs about drugs? > an experiment that is intended to be useful should say something about the size of an effect I agree. Nothing stops people who study depression drugs from doing power analysis and reporting those numbers. > how exactly can you make a well defined experiment that can confirm the null hypothesis I think you refer to the common of idea of rejecting or not rejecting H0. Well, from the point of view of statistical decision theory, the distinction between non-rejecting and accepting just doesn’t exist. You either choose H0 or H1. If you had n=1, of course there is a high chance of Type II error. Do power analysis and design an appropriate experiment given your desired significance level, statistical power and effect size.

amluto 3y ago

I’ve never taken graduate statistics, but I’m generally not impressed with the state of statistics or statistics education. Just because almost every university teaches almost every student this (and most published papers work this way) doesn’t mean it’s good. Intuitive, sure. Wise, not so much. > Let's look at two questions: (1) Does X have an effect? (2) How large is that effect? If your prior puts a non-zero probability that the answer to the first question is "No", then priors for the second question will have non-zero at point 0, even though the probability of any other point may be zero. Even ignoring whether it makes sense to have a nonzero prior for X having no effect, it’s generally not useful to learn the answer. An arbitrarily small effect is, for practical purposes, indistinguishable from no effect — an experiment that is intended to be useful should say something about the size of an effect. If a pill helps depression enough to be worth taking the pill, that’s one thing. If it helps in the sense that you could dose literally everyone in the world and one person would feel very slightly better for a day, that’s not helpful. Similarly, the existence of an effect says nothing about the sign of the effect. So I think hypotheses being tested should be useful. If you want to determine whether something is useful, at least set a threshold for usefulness and test that. Or come up with a quantitative measure. The Bayesian-vs-frequentist debate is IMO somewhat secondary to this except insofar as it seems less common to make worthless but mathematically correct Bayesian tests because thinking about priors at all requires some acknowledgment of whether a prior is remotely plausible. Also, how exactly can you make a well defined experiment that can confirm the null hypothesis without putting something resembling a prior on the non-null hypothesis if the non-null hypothesis contains distributions that are arbitrarily close to null? I’m sure it’s doable, but it seems likely to be pretty messy if you dig in. (There are exceptions. For example, the existence of a neutrino mass is very interesting irrespective of what that mass is. But even then, physics results like this generally put bounds on a value that is hypothesized to be zero instead of merely testing for zeroness, because every experiment has finite power to detect small effects, and the readers of an outcome of an experiment should care more about the detection limits than, say, the number of dollars the experiment cost.)

LudwigNagasena 3y ago
> it’s generally not useful to learn the answer The point of priors is to represent beliefs about reality. It would certainly be surprising if you had a zero prior for neutrino to have no charge. Why cannot people have similar beliefs about drugs? > an experiment that is intended to be useful should say something about the size of an effect I agree. Nothing stops people who study depression drugs from doing power analysis and reporting those numbers. > how exactly can you make a well defined experiment that can confirm the null hypothesis I think you refer to the common of idea of rejecting or not rejecting H0. Well, from the point of view of statistical decision theory, the distinction between non-rejecting and accepting just doesn’t exist. You either choose H0 or H1. If you had n=1, of course there is a high chance of Type II error. Do power analysis and design an appropriate experiment given your desired significance level, statistical power and effect size.