4 ms·

>The problem is that the p-value gives the right answer to the wrong question. What we really want to know is not the probability of the observations given a hy

>The problem is that the p-value gives the right answer to the wrong question. What we really want to know is not the probability of the observations given a hypothesis about the existence of a real effect, but rather the probability that there is a real effect – that the hypothesis is true – given the observations. And that is a problem of induction. The problem of induction is real and unavoidable in the general case, but there is no such thing as "the probability that there is a real effect". Either there is a "real effect" or there is not. It might be possible to find a "probability" that you observed a "real effect". >The problem of induction was solved, in principle, by the Reverend Thomas Bayes in the middle of the 18th century. The problem of induction is fundamentally unsolvable (hence, "problem"). The article just states that it was solved and never mentions it again. Is it a widely held view that induction was solved by Bayes? Does anyone know where I can read more detailed claims about how people believe Bayes solved induction?

bayeslives 10y ago
Bayes did not "solve induction". If we define induction as telling which specific model generated this data: that is not possible. Countless models could in theory generate our data. What we need is some restriction. Like a prior. And when was the last time you started a research project without any idea what to expect? And if you did, wouldn't it be wiser to do some literature study, experts interviews etc. before starting experiments? Modeling the state of the art, pre-experiment, seems like a clever move anyway. To name just a few of the NHST/p-value flaws: 1- I'm interested in P(H1 | data) but I get P(data | Ho). Contrary to popular belief P(data|Ho) != P(Ho|data). Let alone that conclusions about P(H1|data) can be drawn. 2- it is vulnerable to wrong interpretations. * No, a 95% confidence interval (a,b) does NOT mean there is a 95% chance that Mu is in (a,b). * No, p=0.04 does not mean they is a 96% chance that H1 is true. 3- the p-value depends on the intentions of the scientist. If you end your experiment after 80 observations, as planned, your p-value is different from that of an experiment that ended unplanned after 80 observations. So the same data have different evidential power, influenced by results you did not see in experiments you did not do. This is very unsatisfactory. 4- the idea of "an effect that exists or does not exist", based on some arbitrary threshold. The reality is, in many cases, uncertainty and variation. In group A I see effects of medicine A, with lots of variation between persons. In group B I see varying effects of medicine B. Then I introduce uncertainty by drawing random samples from A and B. Let's day I used those samples to make an inference: is, on average, medicine A better than medicine B ? Matras like "there is an effect, or there isn't" are not very helpful. Statistics should be about quantifying uncertainty rather than give false yes/no statements. 5. Basing decisions and knowledge on the data only makes t vulnerable to outliers, unlucky samples and so on. And why should you NOT use information, when it's there ?
- vmuhonen 10y ago
  P-value being defined as the probability of observing a result equally or more extreme under a model H0. So if you start with with assumption that H0 is true, there's not much you can say about alternative hypotheses. The American Statistical Association actually put out a statement this year on the issue of p-value. You can find the whole article here http://dx.doi.org/10.1080/00031305.2016.1154108 http://dx.doi.org/10.1080/00031305.2016.1154108 but here are the main points: 1) P-values can indicate how incompatible the data are with a specified statistical model. 2) P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone. 3) Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold. 4) Proper inference requires full reporting and transparency. 5) A p-value, or statistical significance, does not measure the size of an effect or the importance of a result. 6) By itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis. As an additional curiosity, the group of writers was not completely unconflicted coming up those definitions and the article contains a number of supplemental articles by the individual authors to clarify/dispute some of the points made. [edit] fixed formatting
  - bayeslives 10y ago
    Good points. The NHST thing was invented by Neyman & Pearson as a tool for decision making, not for finding the truth.95% confidence means your intervals will be not to far off, in 95% of all samples. Perhaps this is nice fur Quality Asurance in factories. Where I do repeated measurements and where I want a simple YES or NO. But science usually asks: "what can I learn from this specific data? I don't do 100 samples and I'm not interested in being "not to far off moist of the time". I want a best estimation based on this specific sample". NHST does not give that answer. Bayes does.
    - jsprogrammer 10y ago
      Science only asks, "Have I observed something contrary to my theories?" For non-deductive theories, the only real approach is to count the number of times you observed something agreeing with your theories vs. the total number of times you observed something (ie. p-value).

bayeslives 10y ago

Bayes did not "solve induction". If we define induction as telling which specific model generated this data: that is not possible. Countless models could in theory generate our data. What we need is some restriction. Like a prior. And when was the last time you started a research project without any idea what to expect? And if you did, wouldn't it be wiser to do some literature study, experts interviews etc. before starting experiments? Modeling the state of the art, pre-experiment, seems like a clever move anyway. To name just a few of the NHST/p-value flaws: 1- I'm interested in P(H1 | data) but I get P(data | Ho). Contrary to popular belief P(data|Ho) != P(Ho|data). Let alone that conclusions about P(H1|data) can be drawn. 2- it is vulnerable to wrong interpretations. * No, a 95% confidence interval (a,b) does NOT mean there is a 95% chance that Mu is in (a,b). * No, p=0.04 does not mean they is a 96% chance that H1 is true. 3- the p-value depends on the intentions of the scientist. If you end your experiment after 80 observations, as planned, your p-value is different from that of an experiment that ended unplanned after 80 observations. So the same data have different evidential power, influenced by results you did not see in experiments you did not do. This is very unsatisfactory. 4- the idea of "an effect that exists or does not exist", based on some arbitrary threshold. The reality is, in many cases, uncertainty and variation. In group A I see effects of medicine A, with lots of variation between persons. In group B I see varying effects of medicine B. Then I introduce uncertainty by drawing random samples from A and B. Let's day I used those samples to make an inference: is, on average, medicine A better than medicine B ? Matras like "there is an effect, or there isn't" are not very helpful. Statistics should be about quantifying uncertainty rather than give false yes/no statements. 5. Basing decisions and knowledge on the data only makes t vulnerable to outliers, unlucky samples and so on. And why should you NOT use information, when it's there ?

vmuhonen 10y ago
P-value being defined as the probability of observing a result equally or more extreme under a model H0. So if you start with with assumption that H0 is true, there's not much you can say about alternative hypotheses. The American Statistical Association actually put out a statement this year on the issue of p-value. You can find the whole article here http://dx.doi.org/10.1080/00031305.2016.1154108 http://dx.doi.org/10.1080/00031305.2016.1154108 but here are the main points: 1) P-values can indicate how incompatible the data are with a specified statistical model. 2) P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone. 3) Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold. 4) Proper inference requires full reporting and transparency. 5) A p-value, or statistical significance, does not measure the size of an effect or the importance of a result. 6) By itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis. As an additional curiosity, the group of writers was not completely unconflicted coming up those definitions and the article contains a number of supplemental articles by the individual authors to clarify/dispute some of the points made. [edit] fixed formatting
- bayeslives 10y ago
  Good points. The NHST thing was invented by Neyman & Pearson as a tool for decision making, not for finding the truth.95% confidence means your intervals will be not to far off, in 95% of all samples. Perhaps this is nice fur Quality Asurance in factories. Where I do repeated measurements and where I want a simple YES or NO. But science usually asks: "what can I learn from this specific data? I don't do 100 samples and I'm not interested in being "not to far off moist of the time". I want a best estimation based on this specific sample". NHST does not give that answer. Bayes does.
  - jsprogrammer 10y ago
    Science only asks, "Have I observed something contrary to my theories?" For non-deductive theories, the only real approach is to count the number of times you observed something agreeing with your theories vs. the total number of times you observed something (ie. p-value).