3 ms·

What isn't? Hypothesis Testing or the 5σ standard?

by beojan 8y ago

nonbel 8y ago
A more stringent significance level just means it costs more money or the data is much cheaper to collect. You can look up "NHST controversy", "NHST pseudoscience", etc to find many people discussing the problems for over 50 years. Here is a decent start: https://www.lesswrong.com/posts/ttvnPRTxFyru9Hh2H/against-nhst https://www.lesswrong.com/posts/ttvnPRTxFyru9Hh2H/against-nh... I've been lately seeing that this practice is infecting particle physics, which is not a good sign. I do not think they are overwhelmed yet though. Here is how it has gone in every field so far (education research, psychology, medicine, etc): Eventually the field will stop caring about all actual quantitative predictions (some barely funded crackpots on the edges will publish in ignored journals) and researchers will only test a "null hypothesis" that everyone knows is wrong (eg, in the physics case a world where the standard model is assumed true but has no higgs boson). Then upon rejecting the (known to be false) null hypothesis, they will conclude some theory only capable of vague/malleable predictions is correct. The way it works is simple, the null model acts as a strawman. The confidence in rejecting the null model is somehow translated into support for the researcher's favorite theory. How does changing the significance level address this problem?
- beojan 8y ago
  > The confidence in rejecting the null model is somehow translated into support for the researcher's favorite theory. How does changing the significance level address this problem? Not just any old favorite theory. For instance, an excess at a mass of 125 GeV doesn't provide any evidence for a new particle at 2 TeV. It does however provide evidence for a new particle at 125 GeV. > Eventually the field will stop caring about all actual quantitative predictions (some barely funded crackpots on the edges will publish in ignored journals) and researchers will only test a "null hypothesis" that everyone knows is wrong (eg, in the physics case a world where the standard model is assumed true but has no higgs boson). Then upon rejecting the (known to be false) null hypothesis, they will conclude some theory only capable of vague/malleable predictions is correct. If that were the case, we would have made a lot more discoveries by now.
  - nonbel 8y ago
    >"If that were the case, we would have made a lot more discoveries by now." In physics you still have quantitative predictions done in addition to checking the null hypothesis (which I why I said that was superfluous), if NHST adoption continues these will slowly be deprecated and replaced by only checking the null hypothesis. Seriously go read some old psychology before NHST destroyed it, they were developing quantitative theories of learning and everything [1,2]. Then go look at it now (statistically significant this and that with basically no progress for 60 years...). Also, the significance level, constraints on p-hacking, etc are adjusted so that discoveries occur at the "right" rate. I'm not sure what triggered he move to 5 sigma in physics but it was probably something like "this seems too easy" so people started carefully double checking and getting conflicting results. [1] https://www.tandfonline.com/doi/pdf/10.1080/00221309.1934.9917847 https://www.tandfonline.com/doi/pdf/10.1080/00221309.1934.99... [2] https://link.springer.com/article/10.1007/BF02289265 https://link.springer.com/article/10.1007/BF02289265
    - beojan 8y ago
      At 5σ, you lose a lot of the downsides of standard p < 0.05 hypothesis testing because the p value is so low. Nevertheless, I suppose what you would suggest is just setting limits to start with and looking for a bump there instead. That's not a bad idea, and some (Gross, e.g.) have suggested setting limits even in discovery papers.
      - nonbel 8y ago
        >"At 5σ, you lose a lot of the downsides of standard p < 0.05 hypothesis testing because the p value is so low." I don't see how this addresses any of the issues, it just makes it more expensive (ceteris paribus). As I already wrote: "A more stringent significance level just means it costs more money or the data is much cheaper to collect. ...The way it works is simple, the null model acts as a strawman. The confidence in rejecting the null model is somehow translated into support for the researcher's favorite theory. How does changing the significance level address this problem?" >"Nevertheless, I suppose what you would suggest is just setting limits to start with and looking for a bump there instead." Exactly, and the more precise the limits, the more meaningful for the theory if its consistent with the data. At the extreme, theories that allow for any value at all (as I understand the case is for every possible observation under string theory) are not supported by the data no matter how it turns out.

What isn't? Hypothesis Testing or the 5σ standard?

by beojan 8y ago

What isn't? Hypothesis Testing or the 5σ standard?

A more stringent significance level just means it costs more money or the data is much cheaper to collect. You can look up "NHST controversy", "NHST pseudoscience", etc to find many people discussing the problems for over 50 years. Here is a decent start: https://www.lesswrong.com/posts/ttvnPRTxFyru9Hh2H/against-nhst https://www.lesswrong.com/posts/ttvnPRTxFyru9Hh2H/against-nh... I've been lately seeing that this practice is infecting particle physics, which is not a good sign. I do not think they are overwhelmed yet though. Here is how it has gone in every field so far (education research, psychology, medicine, etc): Eventually the field will stop caring about all actual quantitative predictions (some barely funded crackpots on the edges will publish in ignored journals) and researchers will only test a "null hypothesis" that everyone knows is wrong (eg, in the physics case a world where the standard model is assumed true but has no higgs boson). Then upon rejecting the (known to be false) null hypothesis, they will conclude some theory only capable of vague/malleable predictions is correct. The way it works is simple, the null model acts as a strawman. The confidence in rejecting the null model is somehow translated into support for the researcher's favorite theory. How does changing the significance level address this problem?

beojan 8y ago
> The confidence in rejecting the null model is somehow translated into support for the researcher's favorite theory. How does changing the significance level address this problem? Not just any old favorite theory. For instance, an excess at a mass of 125 GeV doesn't provide any evidence for a new particle at 2 TeV. It does however provide evidence for a new particle at 125 GeV. > Eventually the field will stop caring about all actual quantitative predictions (some barely funded crackpots on the edges will publish in ignored journals) and researchers will only test a "null hypothesis" that everyone knows is wrong (eg, in the physics case a world where the standard model is assumed true but has no higgs boson). Then upon rejecting the (known to be false) null hypothesis, they will conclude some theory only capable of vague/malleable predictions is correct. If that were the case, we would have made a lot more discoveries by now.
- nonbel 8y ago
  >"If that were the case, we would have made a lot more discoveries by now." In physics you still have quantitative predictions done in addition to checking the null hypothesis (which I why I said that was superfluous), if NHST adoption continues these will slowly be deprecated and replaced by only checking the null hypothesis. Seriously go read some old psychology before NHST destroyed it, they were developing quantitative theories of learning and everything [1,2]. Then go look at it now (statistically significant this and that with basically no progress for 60 years...). Also, the significance level, constraints on p-hacking, etc are adjusted so that discoveries occur at the "right" rate. I'm not sure what triggered he move to 5 sigma in physics but it was probably something like "this seems too easy" so people started carefully double checking and getting conflicting results. [1] https://www.tandfonline.com/doi/pdf/10.1080/00221309.1934.9917847 https://www.tandfonline.com/doi/pdf/10.1080/00221309.1934.99... [2] https://link.springer.com/article/10.1007/BF02289265 https://link.springer.com/article/10.1007/BF02289265
  - beojan 8y ago
    At 5σ, you lose a lot of the downsides of standard p < 0.05 hypothesis testing because the p value is so low. Nevertheless, I suppose what you would suggest is just setting limits to start with and looking for a bump there instead. That's not a bad idea, and some (Gross, e.g.) have suggested setting limits even in discovery papers.
    - nonbel 8y ago
      >"At 5σ, you lose a lot of the downsides of standard p < 0.05 hypothesis testing because the p value is so low." I don't see how this addresses any of the issues, it just makes it more expensive (ceteris paribus). As I already wrote: "A more stringent significance level just means it costs more money or the data is much cheaper to collect. ...The way it works is simple, the null model acts as a strawman. The confidence in rejecting the null model is somehow translated into support for the researcher's favorite theory. How does changing the significance level address this problem?" >"Nevertheless, I suppose what you would suggest is just setting limits to start with and looking for a bump there instead." Exactly, and the more precise the limits, the more meaningful for the theory if its consistent with the data. At the extreme, theories that allow for any value at all (as I understand the case is for every possible observation under string theory) are not supported by the data no matter how it turns out.