3 ms·
From what I have seen it is quite simple. 1) There is this pseudo-scientific methodology called NHST (null hypothesis significance testing) devised pretty much
by nonbel 8y ago
From what I have seen it is quite simple.
1) There is this pseudo-scientific methodology called NHST (null hypothesis significance testing) devised pretty much on accident in the 1940-50s that tricks people into thinking they have discovered something interesting.
2) It is so much easier to generate these "significant" results than to study a phenomenon and come up with a model capable of making a precise prediction, then have multiple people/groups carefully collect data to check that prediction.
3) People willing to do the NHST thing can publish papers orders of magnitude faster than people not willing to.
4) Number of publications is the primary metric of success in academia.
5) People willing to do the NHST thing slowly take over the field as those who want to do real science get pushed out, retire, or leave for greener pastures.
6) Eventually the field is filled with people generating worthless papers that cannot be replicated in principle and often contradict each other.
- a) Because all that matters is you got a significant result published in a peer reviewed journal
- b) The discovery has already been made, the statistics said so, why would anyone need to replicate it?
And to preempt a common confusion. This has nothing to do with bayesian vs frequentist, testing a strawman null hypothesis can be done using bayesian math just as well as frequentist.
Some decent intros to this problem (yes, it was pointed out long ago but there is no stopping the need to publish apparently):
Meehl 1967: https://meehl.dl.umn.edu/sites/g/files/pua1696/f/074theorytestingparadox.pdf https://meehl.dl.umn.edu/sites/g/files/pua1696/f/074theoryte...
Gigerenzer 2004: https://library.mpib-berlin.mpg.de/ft/gg/GG_Mindless_2004.pdf https://library.mpib-berlin.mpg.de/ft/gg/GG_Mindless_2004.pd...
- fouc 8y agoFirst time I've heard that NHST could be a source of issues with science. I've heard about p-hacking issues or frequentist statistics getting the blame, but NHST being a root cause seems plausible in retrospect. Thanks!
- nonbel 8y agoGreat to hear you are interested in this problem. It is a huge one that goes mostly ignored. Afaict, the p-hacking/etc people usually focus on are red herrings caused by the incompatibility of NHST and scientific research. Eg, it is good thing for a scientist to analyze the data in a bunch of different ways. It is definitely not something that should be punished.
- kiliantics 8y agoCalculating p-values falls under hypothesis testing
- nonbel 8y ago1) Hypothesis testing, significance testing, and NHST are all different things 2) P-values seem ok as a way to check how well your model fits the data. The problem is testing a strawman model/hypothesis.
- kgwgk 8y agoThe article seems to be mostly about “hard” sciences, where ”discoveries” are not about getting some p<0.05
- nonbel 8y agoUnfortunately, I can see its infection of physics is in progress. And testing 0.05 vs .000005 makes no difference if you are testing a model other than the one you believe. It just makes it more expensive and time consuming to draw an inappropriate conclusion from the p-value. On the other hand, thankfully, in physics areas they usually also test actual predictions of their models. But that is part of the process you can see happened in psych, then medicine. If you sample the literature going back to 1900 or so: Stage 1: Science - Test the predictions of your models Stage 2: Hybrid - Test the predictions of your models and also strawman null/background models Stage 3: Pseudoscience - Test only strawman null/background models My guess is during stage 2 there are still people untrained in NHST in important positions who hold back the wave of BS by demanding actual science (testing predictions of models someone thinks may work). Once they retire you get stage 3. One day I want to write this all up, its really quite interesting to read the old psych and medicine papers where people were coming up with mathematical laws, etc then the slow progression to "is there a significant correlation/difference?".
- keldaris 8y agoIn my experience as a researcher in physics (obviously anecdotal to my field and not generalizable across physics) all we do are variations on Stage 1 - we either have a model, a prediction and an experiment that confirms it (these go to the best journals), or we only have a model + prediction (and couldn't do the experiment) or an experiment (but not a model for it), so we report those in the hopes that someone else will fill in the missing half. Since I mostly do theoretical soft matter work, having a model with an interesting but hard to verify prediction is the common case (which is far from useless, as experimental techniques are advancing incredibly quickly). Either way, if I brought up p-values I'd be unceremoniously laughed out of the room in my department. If I wrote up a prediction that's not just verifiable in principle but verifiable with existing (albeit sometimes new, rare or hard to implement) experimental methods, I wouldn't get published, unless it was some kind of a novel theoretical technique with further applications in that regard. I'm not saying your assertion is wrong - physics is wildly different across different fields, and I don't pretend to have any meaningful overview of all of them. I can only report that I see no signs of this supposed encroachment of pseudoscientific statistics in my physics work or the literature I read professionally - I hope it stays that way.