3 ms·

>"Statistics alone cannot tell you which of those two models is correct." Statistics can tell you whether a model is consistent with the data. But you need to

by nonbel 10y ago

>"Statistics alone cannot tell you which of those two models is correct." Statistics can tell you whether a model is consistent with the data. But you need to deduce the null hypothesis from your model rather than use the default "no difference" (of course, sometimes no difference is deduced from a real model, but not often, in that case: great!). In fact, that is the proper use of statistics. I would guess >99.99% of current usage is incorrect (ie pseudoscience) and amounts to a waste of time at best. The usual usage turns scientific reasoning on its head, and has lead to a (literally for most people) unbelievable amount of trouble. This was pointed out most aptly by Paul Meehl long, long ago: http://www.fisme.science.uu.nl/staff/christianb/downloads/meehl1967.pdf http://www.fisme.science.uu.nl/staff/christianb/downloads/me...

lisper 10y ago
> Statistics can tell you whether a model is consistent with the data. Yes, that's true, but it badly misses the point. The power of statistics is to tell you when a model (the null hypothesis) is (most likely) inconsistent with the data so that you can confidently rule it out. Any finite data set is consistent with an infinite number of models, so knowing that a model and the data are consistent tells you absolutely nothing about whether or not that model has any relationship with reality (which, at the risk of stating the obvious, is what science actually cares about). This is the reason that rejecting the null hypothesis is considered a positive result.
- ACow_Adonis 10y ago
  Except such a data set is also inconsistent with an infinite number of models, so ruling one out via rejecting a null also provides practically no information value and moves us no closer to understanding. /devils advocate In practical terms, we're not interested in true models, but useful ones, so the description of a model's consistency with observed data is often the more useful metric in practice than rejecting nulls :/ especially in applications where you can't set up repeated experiments. OK, I realise its more nuanced than that too, but given how many papers and practitioners seem incapable of understanding that evidence against the null it's not explicit evidence for an arbitrary alternative, practically and consequentially I don't think that's how we should be working...
  - lisper 10y ago
    > Except such a data set is also inconsistent with an infinite number of models, so ruling one out via rejecting a null also provides practically no information value and moves us no closer to understanding. No, that's not true, because experiments are not done in a (figurative) vacuum. They are done in the context of an explanatory theory that has already gone through a vigorous filter and shown to be consistent with the all prior experimental data and has better explanatory power than all of its competitors. It is only when more than one theory survives this filter that an experiment is done, and the experiment is designed specifically to distinguish between the surviving theories. So while it is true that an experiment allows you to eliminate an infinite number of theories, it's irrelevant, because by the time the experiment is done nearly all of those theories have already been eliminated anyway.
- nonbel 10y ago
  >"The power of statistics is to tell you when a model (the null hypothesis) is (most likely) inconsistent with the data so that you can confidently rule it out" If you know whether your model is consistent with the data, you know whether it is inconsistent... I think you are talking about some other issue than I am. The point about deducing the null hypothesis from your explanatory model is that the null hypothesis is precise. In that case you will get a strong test of the model, and it will get stronger as more data gets collected. Using a default null and and vague alternative is the exact opposite. (check the Meehl 1967 paper I linked earlier in this thread).
  - lisper 10y ago
    > If you know whether your model is consistent with the data, you know whether it is inconsistent... No, that's not true. And in fact I got it wrong earlier when I agreed with you that statistics can tell you when data is consistent with a model. They can't. At best they can tell you whether the data are not inconsistent. That sounds like the same thing, but it isn't. It's like the distinction between "not guilty" and "actually innocent." At best (or at worst depending on how you look at it) a statistical test can tell you, "This theory cannot be confidently ruled out on the basis of that data under the following assumptions..." > I think you are talking about some other issue than I am. That is quite possible. [UPDATE:] BTW, I read the Meehl paper, and I completely agree with what he says. So you and I may be in "violent agreement" here.

by nonbel 10y ago

lisper 10y ago

> Statistics can tell you whether a model is consistent with the data. Yes, that's true, but it badly misses the point. The power of statistics is to tell you when a model (the null hypothesis) is (most likely) inconsistent with the data so that you can confidently rule it out. Any finite data set is consistent with an infinite number of models, so knowing that a model and the data are consistent tells you absolutely nothing about whether or not that model has any relationship with reality (which, at the risk of stating the obvious, is what science actually cares about). This is the reason that rejecting the null hypothesis is considered a positive result.

ACow_Adonis 10y ago
Except such a data set is also inconsistent with an infinite number of models, so ruling one out via rejecting a null also provides practically no information value and moves us no closer to understanding. /devils advocate In practical terms, we're not interested in true models, but useful ones, so the description of a model's consistency with observed data is often the more useful metric in practice than rejecting nulls :/ especially in applications where you can't set up repeated experiments. OK, I realise its more nuanced than that too, but given how many papers and practitioners seem incapable of understanding that evidence against the null it's not explicit evidence for an arbitrary alternative, practically and consequentially I don't think that's how we should be working...
- lisper 10y ago
  > Except such a data set is also inconsistent with an infinite number of models, so ruling one out via rejecting a null also provides practically no information value and moves us no closer to understanding. No, that's not true, because experiments are not done in a (figurative) vacuum. They are done in the context of an explanatory theory that has already gone through a vigorous filter and shown to be consistent with the all prior experimental data and has better explanatory power than all of its competitors. It is only when more than one theory survives this filter that an experiment is done, and the experiment is designed specifically to distinguish between the surviving theories. So while it is true that an experiment allows you to eliminate an infinite number of theories, it's irrelevant, because by the time the experiment is done nearly all of those theories have already been eliminated anyway.
nonbel 10y ago
>"The power of statistics is to tell you when a model (the null hypothesis) is (most likely) inconsistent with the data so that you can confidently rule it out" If you know whether your model is consistent with the data, you know whether it is inconsistent... I think you are talking about some other issue than I am. The point about deducing the null hypothesis from your explanatory model is that the null hypothesis is precise. In that case you will get a strong test of the model, and it will get stronger as more data gets collected. Using a default null and and vague alternative is the exact opposite. (check the Meehl 1967 paper I linked earlier in this thread).
- lisper 10y ago
  > If you know whether your model is consistent with the data, you know whether it is inconsistent... No, that's not true. And in fact I got it wrong earlier when I agreed with you that statistics can tell you when data is consistent with a model. They can't. At best they can tell you whether the data are not inconsistent. That sounds like the same thing, but it isn't. It's like the distinction between "not guilty" and "actually innocent." At best (or at worst depending on how you look at it) a statistical test can tell you, "This theory cannot be confidently ruled out on the basis of that data under the following assumptions..." > I think you are talking about some other issue than I am. That is quite possible. [UPDATE:] BTW, I read the Meehl paper, and I completely agree with what he says. So you and I may be in "violent agreement" here.