3 ms·
Unfortunately, I can see its infection of physics is in progress. And testing 0.05 vs .000005 makes no difference if you are testing a model other than the one
by nonbel 8y ago
Unfortunately, I can see its infection of physics is in progress. And testing 0.05 vs .000005 makes no difference if you are testing a model other than the one you believe. It just makes it more expensive and time consuming to draw an inappropriate conclusion from the p-value.
On the other hand, thankfully, in physics areas they usually also test actual predictions of their models. But that is part of the process you can see happened in psych, then medicine. If you sample the literature going back to 1900 or so:
Stage 1: Science
- Test the predictions of your models
Stage 2: Hybrid
- Test the predictions of your models and also strawman null/background models
Stage 3: Pseudoscience
- Test only strawman null/background models
My guess is during stage 2 there are still people untrained in NHST in important positions who hold back the wave of BS by demanding actual science (testing predictions of models someone thinks may work). Once they retire you get stage 3.
One day I want to write this all up, its really quite interesting to read the old psych and medicine papers where people were coming up with mathematical laws, etc then the slow progression to "is there a significant correlation/difference?".
- keldaris 8y agoIn my experience as a researcher in physics (obviously anecdotal to my field and not generalizable across physics) all we do are variations on Stage 1 - we either have a model, a prediction and an experiment that confirms it (these go to the best journals), or we only have a model + prediction (and couldn't do the experiment) or an experiment (but not a model for it), so we report those in the hopes that someone else will fill in the missing half. Since I mostly do theoretical soft matter work, having a model with an interesting but hard to verify prediction is the common case (which is far from useless, as experimental techniques are advancing incredibly quickly). Either way, if I brought up p-values I'd be unceremoniously laughed out of the room in my department. If I wrote up a prediction that's not just verifiable in principle but verifiable with existing (albeit sometimes new, rare or hard to implement) experimental methods, I wouldn't get published, unless it was some kind of a novel theoretical technique with further applications in that regard. I'm not saying your assertion is wrong - physics is wildly different across different fields, and I don't pretend to have any meaningful overview of all of them. I can only report that I see no signs of this supposed encroachment of pseudoscientific statistics in my physics work or the literature I read professionally - I hope it stays that way.
- nonbel 8y agoI'm speaking from what I saw in the LIGO and CERN (Higg's Boson) papers. They spend way too much time on rejecting some questionable model of the background rather than testing actual predictions. From media interviews, I'll also say on the theorist side of it they seem to harbor all the usual misconceptions about p-values as seen in psych and medicine. But they do eventually fit an actual prediction to the data, like a model of black holes colliding, or whatever. However, I don't personally know how many degrees of freedom they have in these fits...
- keldaris 8y agoLIGO and the Higgs boson example are both extreme outliers in physics in terms of scale, how much statistics they necessarily involve, the scrutiny and publicity they get and almost every other property. I'd also note that scientists trying to oversimplify things for a media interview almost always screw up somewhere. I wouldn't draw any conclusions about physics in general from either their papers or (especially) interviews.