3 ms·

Null hypothesis significance testing. It is a "method" of statistics that resulted from confusion between Ronald Fisher's significance testing with Jerzy Neyman

by nonbel 10y ago

Null hypothesis significance testing. It is a "method" of statistics that resulted from confusion between Ronald Fisher's significance testing with Jerzy Neyman and Egon Pearson's hypothesis testing sometime in the 1940s. None of those three are free of blame either. While there is all sorts of things wrong with NHST, the primary thing is that researchers never actually test their own hypothesis. Instead they test a "null" hypothesis that acts as a strawman, usually that some parameter equals exactly zero. Then when the strawman is "rejected", they go on to say the results support whatever theory/idea it is they favor. Rather than come up with precise predictions deduced from their idea, they leave them vague (If I am right, A will be correlated with B). Then calculations are used to reject the very precise strawman, giving an entirely false sense of rigor to a handwavy, nonsensical ritual. It is very common, and seems to get more and more popular every year despite all efforts by scientists and statisticians to stop it (because it lets you seem scientific about proving whatever you want). I was trained to think that way myself not too long ago, with absolutely zero mention of any controversy. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4347431/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4347431/ http://library.mpib-berlin.mpg.de/ft/gg/GG_Mindless_2004.pdf http://library.mpib-berlin.mpg.de/ft/gg/GG_Mindless_2004.pdf http://www.fisme.science.uu.nl/staff/christianb/downloads/meehl1967.pdf http://www.fisme.science.uu.nl/staff/christianb/downloads/me... http://andrewgelman.com/2016/09/30/why-the-garden-of-forking-paths-criticism-of-p-values-is-not-like-a-famous-borscht-belt-comedy-bit/ http://andrewgelman.com/2016/09/30/why-the-garden-of-forking...

omginternets 10y ago
>Instead they test a "null" hypothesis that acts as a strawman, usually that some parameter equals exactly zero How is "there is no difference between my groups/conditions" a strawman? This is by far the most common H0.
- nonbel 10y ago
  Because, except in rare cases, the model/theory/idea of interest does not predict "there is no difference between my groups/conditions". It predicts something else. In the rare cases a zero difference is predicted, the method is scientific. For nearly all use cases many plausible models/theories/ideas will predict "not exactly zero difference", so rejection of the hypothesis "difference = zero" is useless to the scientist. Mathematicians apparently don't get this, which is what inspired the Fisher paper I linked to. And yes, it is the most common H0. No argument there. The problem is so widespread and destructive that the mind instinctively rejects its existence as unbelievable.
  - omginternets 10y ago
    Sorry, I still don't understand. Why would an absence of effect predict anything other than a zero difference (assuming infinite samples)?
    - nonbel 10y ago
      No one predicts "an absence of effect". Don't worry, it took me years of reading the literature, running my own simulations, and having discussions in person and online. It is really difficult to accept how bad this is.
      - omginternets 10y ago
        Please stop being condescending and actually explain yourself. >No one predicts "an absence of effect" In my field, we most certainly do predict an absence of an effect, in the following sense: 1. if a manipulation does not have an effect on the system, then we expect no systematic deviations in the measurement distribution 2. point 1 will be revealed by H0 (no systematic change in measurement distribution across conditions) not being rejected. It's beginning to sound like you've encountered people who misuse or misinterpret this technique -- and I agree that there are alarmingly many such people -- but this is different from the technique being invalid.

by nonbel 10y ago

omginternets 10y ago

>Instead they test a "null" hypothesis that acts as a strawman, usually that some parameter equals exactly zero How is "there is no difference between my groups/conditions" a strawman? This is by far the most common H0.

nonbel 10y ago
Because, except in rare cases, the model/theory/idea of interest does not predict "there is no difference between my groups/conditions". It predicts something else. In the rare cases a zero difference is predicted, the method is scientific. For nearly all use cases many plausible models/theories/ideas will predict "not exactly zero difference", so rejection of the hypothesis "difference = zero" is useless to the scientist. Mathematicians apparently don't get this, which is what inspired the Fisher paper I linked to. And yes, it is the most common H0. No argument there. The problem is so widespread and destructive that the mind instinctively rejects its existence as unbelievable.
- omginternets 10y ago
  Sorry, I still don't understand. Why would an absence of effect predict anything other than a zero difference (assuming infinite samples)?
  - nonbel 10y ago
    No one predicts "an absence of effect". Don't worry, it took me years of reading the literature, running my own simulations, and having discussions in person and online. It is really difficult to accept how bad this is.
    - omginternets 10y ago
      Please stop being condescending and actually explain yourself. >No one predicts "an absence of effect" In my field, we most certainly do predict an absence of an effect, in the following sense: 1. if a manipulation does not have an effect on the system, then we expect no systematic deviations in the measurement distribution 2. point 1 will be revealed by H0 (no systematic change in measurement distribution across conditions) not being rejected. It's beginning to sound like you've encountered people who misuse or misinterpret this technique -- and I agree that there are alarmingly many such people -- but this is different from the technique being invalid.