6 ms·

> The statistical tests currently in widespread use as a criterion for publication in peer-reviewed journals guarantee that at least one result in 20 will be du

by 3JPLW 9y ago

> The statistical tests currently in widespread use as a criterion for publication in peer-reviewed journals guarantee that at least one result in 20 will be due to chance and not because the hypothesis being tested is actually true. That's wildly overly pessimistic. That would only be the case if scientists just went around, looking at the world, and came up with null-hypotheses willy nilly. That's generally not the case. There is often a _reason_ for conducting the test and a plausible cause of action. There are two possible reasons for a significant result: * The null hypothesis is correct but they got "unlucky" data (5% chance or p% chance) * There is a real effect (and the null hypothesis is actually wrong) This becomes more problematic in reproducibility tests, though, since that biases my prior towards a correct null hypothesis and now you must be very careful about pre-registration and the numbers of folks worldwide that are trying to reproduce a given experiment.

PeterisP 9y ago
"p-hacking" is also a thing, it's a reasonable possibility that they tried 20 false but plausible null hypothesis and published the one that got a significant result. If there's no pre-registration, my prior of a paper actually being true would be much lower than statistics would say simply because the incentives are aligned so that there's a motivation to exaggerate significance.
lisper 9y ago
> > at least one result in 20 will be due to chance > That's wildly overly pessimistic Why? You yourself said: > The null hypothesis is correct but they got "unlucky" data (5% chance or p% chance) How is that different from what I said? 5% is just another way of writing 1 in 20.
- 3JPLW 9y ago
  It's different because you're ignoring my second bullet point. There's a very common misconception that p-value is the percent chance the data confirm that the effect is real. That's not at all how p-values work. They are showing the _likelihood_ of generating the data were there no "effect." You make a "null hypothesis" — that is, you assume there is no effect — and you construct a distribution of the results you might see in such a case. If your results are relatively unlikely, then you say the null hypothesis is rejected at a p-value level, leading support to the fact that there is such an effect. But there's another bullet in my comment. You could _also_ see a low p-value if the null hypothesis is wrong! That would _also_ lead to a low p-value, but for a completely different reasons — reasons that are typically elucidated in the entire rest of the paper. Thus, you should be most suspicious of papers that do not have a reasonable cause of action, as that should increase your prior that the null hypothesis is actually correct. If I were to edit that post, I'd add a third bullet — that they cheated or did bad science. My objection is to you stating that the statistical tests alone guarantee at least a 1 in 20 failure rate with a p-value of .05. There are indeed many other reasons to be suspect, but the statistical tests don't guarantee this.
  - lisper 9y ago
    I think you're just misinterpreting what I mean by "failure" or "wrong result". I mean a positive result that is due to chance rather than to the null hypothesis actually being false. On that view, statistics alone guarantee a "failure" rate of at least 1 in 20. That's what choosing a threshold of 0.05 means. The on top of that baseline rate of false results you also have self-selection and occasional outright cheating that drives the failure rate higher than 1 in 20. 1 in 20 is what you get when you use a 0.05 threshold and everything else in the system works flawlessly.
    - 3JPLW 9y ago
      Ah, yes, I read you as saying that "1 out of every 20 statistical tests in every journal is guaranteed to be wrong. It's just stats, people." If what you mean to say is that "1 out of every 20 statistical tests of a false finding will demonstrate a 'significant' p-value and may subsequently get published," then we're in agreement.
      - lisper 9y ago
        I meant to say what I said. Getting a significant p-value when in fact there is no effect is a wrong result, hence, at a minimum, 1 in 20 results (on average) in a journal that uses a p-value threshold of 0.05 will be wrong.
        3JPLW 9y ago
        Then that is incorrect. The situation is simultaneously better and worse than your interpretation: > while a low P value indicates that your data are unlikely > assuming a true null, it can’t evaluate which of two > competing cases is more likely: > * The null is true but your sample was unusual. > * The null is false. You're conflating the two populations, and indeed we don't know which is which. But the P-value is only talking about the likelihood of observing your data given a true null. It only has any effect on that first population. Without knowing the size of the second population, you cannot say anything about the fraction of erroneous results in a journal based upon the statistical test alone. http://blog.minitab.com/blog/adventures-in-statistics-2/how-to-correctly-interpret-p-values http://blog.minitab.com/blog/adventures-in-statistics-2/how-... At the same time, though, converting p-values to a bayesian posterior or conditional probability that says something about the likelihood of the _hypothesis_ (like you're trying to do) is hard and generally way more pessimistic than 1 in 20 — it's more like 1 in 3. http://www.dcscience.net/Sellke-Bayarri-Berger-calibration-of-P-2001.pdf http://www.dcscience.net/Sellke-Bayarri-Berger-calibration-o...
TangoTrotFox 9y ago
Are you aware of the scale of failure here? The article (referring to the executive summary here) gives some really incredible figures. In psychology 100 reproducibility studies were only able to produce statistically significant results from 36% of prominent papers - compared to the 97% of the original studies. In biotech, a firm tried to reproduce 53 "landmark" studies in hematology and oncology - only 6 could be replicated. The reason this is being called a 'crisis' is because it's looking like the vast majority of science, particularly in the social and human physiological sciences, is junk.
- 3JPLW 9y ago
  Absolutely! I'm only addressing the common misconception in the grandparent that the p-value is the percent likelihood that the effect is real. That's not at all accurate. There are definitely other reasons that lead us to such horrifying results — especially in fields where experimenters _do_ conduct experiments seemingly at random and without a reasonable pathway of action (I'm looking at you, social psychology and friends).
Houshalter 9y ago
It's worse than just 5% because p hacking. Every analysis has many parameters, and you can choose the most significant out of thousands of possibilities. On top of this, the more interesting results are the most surprising. And therefore the least likely to be true a priori. So the chance of an interesting study being true is even lower.

by 3JPLW 9y ago

PeterisP 9y ago

"p-hacking" is also a thing, it's a reasonable possibility that they tried 20 false but plausible null hypothesis and published the one that got a significant result. If there's no pre-registration, my prior of a paper actually being true would be much lower than statistics would say simply because the incentives are aligned so that there's a motivation to exaggerate significance.

lisper 9y ago

> > at least one result in 20 will be due to chance > That's wildly overly pessimistic Why? You yourself said: > The null hypothesis is correct but they got "unlucky" data (5% chance or p% chance) How is that different from what I said? 5% is just another way of writing 1 in 20.

3JPLW 9y ago
It's different because you're ignoring my second bullet point. There's a very common misconception that p-value is the percent chance the data confirm that the effect is real. That's not at all how p-values work. They are showing the _likelihood_ of generating the data were there no "effect." You make a "null hypothesis" — that is, you assume there is no effect — and you construct a distribution of the results you might see in such a case. If your results are relatively unlikely, then you say the null hypothesis is rejected at a p-value level, leading support to the fact that there is such an effect. But there's another bullet in my comment. You could _also_ see a low p-value if the null hypothesis is wrong! That would _also_ lead to a low p-value, but for a completely different reasons — reasons that are typically elucidated in the entire rest of the paper. Thus, you should be most suspicious of papers that do not have a reasonable cause of action, as that should increase your prior that the null hypothesis is actually correct. If I were to edit that post, I'd add a third bullet — that they cheated or did bad science. My objection is to you stating that the statistical tests alone guarantee at least a 1 in 20 failure rate with a p-value of .05. There are indeed many other reasons to be suspect, but the statistical tests don't guarantee this.
- lisper 9y ago
  I think you're just misinterpreting what I mean by "failure" or "wrong result". I mean a positive result that is due to chance rather than to the null hypothesis actually being false. On that view, statistics alone guarantee a "failure" rate of at least 1 in 20. That's what choosing a threshold of 0.05 means. The on top of that baseline rate of false results you also have self-selection and occasional outright cheating that drives the failure rate higher than 1 in 20. 1 in 20 is what you get when you use a 0.05 threshold and everything else in the system works flawlessly.
  - 3JPLW 9y ago
    Ah, yes, I read you as saying that "1 out of every 20 statistical tests in every journal is guaranteed to be wrong. It's just stats, people." If what you mean to say is that "1 out of every 20 statistical tests of a false finding will demonstrate a 'significant' p-value and may subsequently get published," then we're in agreement.
    - lisper 9y ago
      I meant to say what I said. Getting a significant p-value when in fact there is no effect is a wrong result, hence, at a minimum, 1 in 20 results (on average) in a journal that uses a p-value threshold of 0.05 will be wrong.
      - 3JPLW 9y ago
        Then that is incorrect. The situation is simultaneously better and worse than your interpretation: > while a low P value indicates that your data are unlikely > assuming a true null, it can’t evaluate which of two > competing cases is more likely: > * The null is true but your sample was unusual. > * The null is false. You're conflating the two populations, and indeed we don't know which is which. But the P-value is only talking about the likelihood of observing your data given a true null. It only has any effect on that first population. Without knowing the size of the second population, you cannot say anything about the fraction of erroneous results in a journal based upon the statistical test alone. http://blog.minitab.com/blog/adventures-in-statistics-2/how-to-correctly-interpret-p-values http://blog.minitab.com/blog/adventures-in-statistics-2/how-... At the same time, though, converting p-values to a bayesian posterior or conditional probability that says something about the likelihood of the _hypothesis_ (like you're trying to do) is hard and generally way more pessimistic than 1 in 20 — it's more like 1 in 3. http://www.dcscience.net/Sellke-Bayarri-Berger-calibration-of-P-2001.pdf http://www.dcscience.net/Sellke-Bayarri-Berger-calibration-o...

TangoTrotFox 9y ago

Are you aware of the scale of failure here? The article (referring to the executive summary here) gives some really incredible figures. In psychology 100 reproducibility studies were only able to produce statistically significant results from 36% of prominent papers - compared to the 97% of the original studies. In biotech, a firm tried to reproduce 53 "landmark" studies in hematology and oncology - only 6 could be replicated. The reason this is being called a 'crisis' is because it's looking like the vast majority of science, particularly in the social and human physiological sciences, is junk.

3JPLW 9y ago
Absolutely! I'm only addressing the common misconception in the grandparent that the p-value is the percent likelihood that the effect is real. That's not at all accurate. There are definitely other reasons that lead us to such horrifying results — especially in fields where experimenters _do_ conduct experiments seemingly at random and without a reasonable pathway of action (I'm looking at you, social psychology and friends).

Houshalter 9y ago

It's worse than just 5% because p hacking. Every analysis has many parameters, and you can choose the most significant out of thousands of possibilities. On top of this, the more interesting results are the most surprising. And therefore the least likely to be true a priori. So the chance of an interesting study being true is even lower.