10 ms·

> a statistical test answers one central question: what are the odds that the results you observed could have arisen by chance Well, no, that'd be very interes

> a statistical test answers one central question: what are the odds that the results you observed could have arisen by chance Well, no, that'd be very interesting but unfortunately what a statistical test really says is the probability of the results you observed (or more extreme) given chance. P(data|model) and not P(model|data).

lisper 10y ago
It's actually P(data|null-hypothesis).
- wfunction 10y ago
  The model is the null hypothesis here... I don't think the parent comment was wrong w.r.t. this.
  - lisper 10y ago
    No, there's a difference. The model is something like, "This drug attaches itself selectively to cancer cells and kills them." The null hypothesis is, "This drug has no effect." So you conduct a double-blind study, measure the effect of the drug on cancer cells, collect some data and compute that P(data|null-hypothesis) is 1%. It it not the case that there is a 99% chance that your model is correct and that the drug does in fact attach itself to cancer cells and kill them. Because there are other possible models, e.g.: the drug redirects your Chi and channels it into your chakras which kills the cancer cells. Statistics alone cannot tell you which of those two models is correct.
    - fluxion 10y ago
      I think this is just an argument over semantics, the "null hypothesis" is a perfectly valid model using the definition of the statistics community [0] (i.e. a collection of probability distributions over some sample space). [0] https://en.wikipedia.org/wiki/Statistical_model https://en.wikipedia.org/wiki/Statistical_model
      - lisper 10y ago
        The word "model" is being used here in two mutually incompatible ways. Yes, the null hypothesis is a model, but it is not an explanatory model. A scientific hypothesis has to meet two tests to be considered a valid theory. It has to be consistent with the data, and it has to have explanatory power. The theory that cancer drugs work by aligning a patient's chi with their chakras is rejected not because it is inconsistent with the data (it's not) but because it lacks the explanatory power of alternative theories based on molecular biology. The null hypothesis never has explanatory power. The null hypothesis is always a statement of the form, "The explanatory hypothesis under test is wrong for some unknown reason." This is why rejecting the null hypothesis, i.e. showing that the data are (with high probability) inconsistent with the null hypothesis, is considered a positive result.
    - nonbel 10y ago
      >"Statistics alone cannot tell you which of those two models is correct." Statistics can tell you whether a model is consistent with the data. But you need to deduce the null hypothesis from your model rather than use the default "no difference" (of course, sometimes no difference is deduced from a real model, but not often, in that case: great!). In fact, that is the proper use of statistics. I would guess >99.99% of current usage is incorrect (ie pseudoscience) and amounts to a waste of time at best. The usual usage turns scientific reasoning on its head, and has lead to a (literally for most people) unbelievable amount of trouble. This was pointed out most aptly by Paul Meehl long, long ago: http://www.fisme.science.uu.nl/staff/christianb/downloads/meehl1967.pdf http://www.fisme.science.uu.nl/staff/christianb/downloads/me...
      - lisper 10y ago
        > Statistics can tell you whether a model is consistent with the data. Yes, that's true, but it badly misses the point. The power of statistics is to tell you when a model (the null hypothesis) is (most likely) inconsistent with the data so that you can confidently rule it out. Any finite data set is consistent with an infinite number of models, so knowing that a model and the data are consistent tells you absolutely nothing about whether or not that model has any relationship with reality (which, at the risk of stating the obvious, is what science actually cares about). This is the reason that rejecting the null hypothesis is considered a positive result.
        ACow_Adonis 10y ago
        Except such a data set is also inconsistent with an infinite number of models, so ruling one out via rejecting a null also provides practically no information value and moves us no closer to understanding. /devils advocate In practical terms, we're not interested in true models, but useful ones, so the description of a model's consistency with observed data is often the more useful metric in practice than rejecting nulls :/ especially in applications where you can't set up repeated experiments. OK, I realise its more nuanced than that too, but given how many papers and practitioners seem incapable of understanding that evidence against the null it's not explicit evidence for an arbitrary alternative, practically and consequentially I don't think that's how we should be working...
- qwrusz 10y ago
  How about you guys are both right, sort of. There are both Bayesian and Frequentist approaches in statistics! They represent very different methods to statistics, but they are also quite similar. My apologies, I couldn't find one link that gave a good description of Bayesian vs. Frequentist. Here are a couple links to get started: https://xkcd.com/1132/ https://xkcd.com/1132/ http://jakevdp.github.io/blog/2014/03/11/frequentism-and-bayesianism-a-practical-intro/ http://jakevdp.github.io/blog/2014/03/11/frequentism-and-bay... If anyone comes across a link that describes Bayesian and Frequentist clearly, please share it if you don't mind.
  - lisper 10y ago
    That cartoon is actually a pretty good illustration of why frequentists are wrong.
    - qwrusz 10y ago
      Do you mean probably wrong? Either way I disagree. Both approaches have pros and cons depending on the type of analysis. At the very least, just the presence of competing approaches in the field has pushed statisticians to have more rigour and do way more double checking than they might have, out of fear the other side actively looking to poke holes. It's easy to lie with statistics and the only people who can call statisticians out on their BS is other statisticians...Need more of this.
      - lisper 10y ago
        > Do you mean probably wrong? Nope. > Both approaches have pros and cons What are the pros of the frequentist approach?
        danieltillett 10y ago
        Simple maths that can be done without a computer. Useful in the past - not so useful today.
        rspeer 10y ago
- stdbrouw 10y ago
  Yup, and the null hypothesis is a reduced model. Different words, same thing.
  - lisper 10y ago
    I have no idea what you mean by "reduced" but they are not the same thing. The difference in their explanatory power. A (scientific) hypothesis explains things. A null hypothesis simply says, "This hypothesis is wrong" but doesn't say why. It's a crucial distinction.
deleted 10y ago
[deleted]
kgwgk 10y ago
> what are the odds that the results you observed could have arisen by chance? > the probability of the results you observed (or more extreme) given chance I think he said the same thing as you. He didn't talk about "the odds that the results you observed have arisen by chance" but the odds that the results you have observed (the or more extreme is implicit) could have arisen by chance (i.e. the probability of happening by chance alone, given that only chance was involved).

lisper 10y ago

It's actually P(data|null-hypothesis).

wfunction 10y ago
The model is the null hypothesis here... I don't think the parent comment was wrong w.r.t. this.
- lisper 10y ago
  No, there's a difference. The model is something like, "This drug attaches itself selectively to cancer cells and kills them." The null hypothesis is, "This drug has no effect." So you conduct a double-blind study, measure the effect of the drug on cancer cells, collect some data and compute that P(data|null-hypothesis) is 1%. It it not the case that there is a 99% chance that your model is correct and that the drug does in fact attach itself to cancer cells and kill them. Because there are other possible models, e.g.: the drug redirects your Chi and channels it into your chakras which kills the cancer cells. Statistics alone cannot tell you which of those two models is correct.
  - fluxion 10y ago
    I think this is just an argument over semantics, the "null hypothesis" is a perfectly valid model using the definition of the statistics community [0] (i.e. a collection of probability distributions over some sample space). [0] https://en.wikipedia.org/wiki/Statistical_model https://en.wikipedia.org/wiki/Statistical_model
    - lisper 10y ago
      The word "model" is being used here in two mutually incompatible ways. Yes, the null hypothesis is a model, but it is not an explanatory model. A scientific hypothesis has to meet two tests to be considered a valid theory. It has to be consistent with the data, and it has to have explanatory power. The theory that cancer drugs work by aligning a patient's chi with their chakras is rejected not because it is inconsistent with the data (it's not) but because it lacks the explanatory power of alternative theories based on molecular biology. The null hypothesis never has explanatory power. The null hypothesis is always a statement of the form, "The explanatory hypothesis under test is wrong for some unknown reason." This is why rejecting the null hypothesis, i.e. showing that the data are (with high probability) inconsistent with the null hypothesis, is considered a positive result.
  - nonbel 10y ago
    >"Statistics alone cannot tell you which of those two models is correct." Statistics can tell you whether a model is consistent with the data. But you need to deduce the null hypothesis from your model rather than use the default "no difference" (of course, sometimes no difference is deduced from a real model, but not often, in that case: great!). In fact, that is the proper use of statistics. I would guess >99.99% of current usage is incorrect (ie pseudoscience) and amounts to a waste of time at best. The usual usage turns scientific reasoning on its head, and has lead to a (literally for most people) unbelievable amount of trouble. This was pointed out most aptly by Paul Meehl long, long ago: http://www.fisme.science.uu.nl/staff/christianb/downloads/meehl1967.pdf http://www.fisme.science.uu.nl/staff/christianb/downloads/me...
    - lisper 10y ago
      > Statistics can tell you whether a model is consistent with the data. Yes, that's true, but it badly misses the point. The power of statistics is to tell you when a model (the null hypothesis) is (most likely) inconsistent with the data so that you can confidently rule it out. Any finite data set is consistent with an infinite number of models, so knowing that a model and the data are consistent tells you absolutely nothing about whether or not that model has any relationship with reality (which, at the risk of stating the obvious, is what science actually cares about). This is the reason that rejecting the null hypothesis is considered a positive result.
      - ACow_Adonis 10y ago
        Except such a data set is also inconsistent with an infinite number of models, so ruling one out via rejecting a null also provides practically no information value and moves us no closer to understanding. /devils advocate In practical terms, we're not interested in true models, but useful ones, so the description of a model's consistency with observed data is often the more useful metric in practice than rejecting nulls :/ especially in applications where you can't set up repeated experiments. OK, I realise its more nuanced than that too, but given how many papers and practitioners seem incapable of understanding that evidence against the null it's not explicit evidence for an arbitrary alternative, practically and consequentially I don't think that's how we should be working...
qwrusz 10y ago
How about you guys are both right, sort of. There are both Bayesian and Frequentist approaches in statistics! They represent very different methods to statistics, but they are also quite similar. My apologies, I couldn't find one link that gave a good description of Bayesian vs. Frequentist. Here are a couple links to get started: https://xkcd.com/1132/ https://xkcd.com/1132/ http://jakevdp.github.io/blog/2014/03/11/frequentism-and-bayesianism-a-practical-intro/ http://jakevdp.github.io/blog/2014/03/11/frequentism-and-bay... If anyone comes across a link that describes Bayesian and Frequentist clearly, please share it if you don't mind.
- lisper 10y ago
  That cartoon is actually a pretty good illustration of why frequentists are wrong.
  - qwrusz 10y ago
    Do you mean probably wrong? Either way I disagree. Both approaches have pros and cons depending on the type of analysis. At the very least, just the presence of competing approaches in the field has pushed statisticians to have more rigour and do way more double checking than they might have, out of fear the other side actively looking to poke holes. It's easy to lie with statistics and the only people who can call statisticians out on their BS is other statisticians...Need more of this.
    - lisper 10y ago
      > Do you mean probably wrong? Nope. > Both approaches have pros and cons What are the pros of the frequentist approach?
      - danieltillett 10y ago
        Simple maths that can be done without a computer. Useful in the past - not so useful today.
      - rspeer 10y ago
stdbrouw 10y ago
Yup, and the null hypothesis is a reduced model. Different words, same thing.
- lisper 10y ago
  I have no idea what you mean by "reduced" but they are not the same thing. The difference in their explanatory power. A (scientific) hypothesis explains things. A null hypothesis simply says, "This hypothesis is wrong" but doesn't say why. It's a crucial distinction.

deleted 10y ago

[deleted]

kgwgk 10y ago

> what are the odds that the results you observed could have arisen by chance? > the probability of the results you observed (or more extreme) given chance I think he said the same thing as you. He didn't talk about "the odds that the results you observed have arisen by chance" but the odds that the results you have observed (the or more extreme is implicit) could have arisen by chance (i.e. the probability of happening by chance alone, given that only chance was involved).