4 ms·

> One who grasps statistical significance can better interpret health studies ... Ok, that's where I draw the line. Statistical significance is bullshit. Learn

by tgv 2y ago

> One who grasps statistical significance can better interpret health studies ... Ok, that's where I draw the line. Statistical significance is bullshit. Learning about it is as useful as learning about phlogiston. Its biggest failure is that it gives a false sense of security. It's what lead us to the reproducibility crisis. This should not have a place in an article that purports to emphasize the importance of statistics in the school curriculum.

taejavu 2y ago
Any chance you could expand on this, or point me to resources that do? It’s a very interesting take and not one I’m familiar with.
- tgv 2y ago
  There's a lot of technical stuff, but you could look into Ioanides articles [1], and the book Bernoulli's Falacy by Clayton [2] is a good introduction, also for people less involved in statistics. [1] https://medium.com/@d.incecushman/the-p-value-problem-how-dr-john-ioannidis-shook-up-science-20-years-ago-abd242ddfa23 https://medium.com/@d.incecushman/the-p-value-problem-how-dr... (this is just an example; I haven't read it) [2] https://cup.columbia.edu/book/bernoullis-fallacy/9780231199940 https://cup.columbia.edu/book/bernoullis-fallacy/97802311999...
- TrackerFF 2y ago
  Search for "p-hacking", that will give you a nice starter on the topic.
blackbear_ 2y ago
Unfortunately we will not get rid of statistical significance anytime soon. Teaching it and explaining its shortcomings is much better than sweeping it under the rug, and perhaps it will help the next generations of scientists not falling into the same trap.
reedf1 2y ago
No no no. You couldn't have gotten this more wrong. The replication crisis comes from soft sciences under-powered statistical training.
- jfengel 2y ago
  It comes from the fact that soft sciences deal with vast numbers of variables, far too many to control. They actually get a ton of statistics, trying to find a signal in so much noise. It would be great if human beings were more amenable to rigorous experiment. Failing that, we at least need to understand what these things do and don't actually mean. It's either that or give up trying to study people entirely.
  - reedf1 2y ago
    Statistics are not values you collect - but the analysis you perform. In data with many confounding variables or many degrees of freedom, the only way to be honest about what the data shows is through statistics. Statistics is what allows you to filter signal from noise. Statistical significance is a very useful tool, you just have to be honest about what it represents and under what conditions this breaks down.
    - tgv 2y ago
      A statistic is an aggregate number, not an analysis in its own. There may be some linguistic confusion here. > Statistical significance is a very useful tool What for? It answers a question in which nobody is interested. What can it do that Bayesian tools can't?
- llamaimperative 2y ago
  No it doesn’t. Common meme around here but the hard sciences are also rife with unreproducible studies. It’s an incentives problem.
  - reedf1 2y ago
    I actually agree - it's part incentives. And it is important for hard sciences to learn these lessons as well.
jasode 2y ago
>Statistical significance is bullshit. Learning about it is as useful as learning about phlogiston. The term-of-art "statistical significance" isn't bs because we ultimately have to choose what to pay attention to. Removing "statistical significance" from the vocabulary doesn't change the underlying reality about people deciding what to do based on a threshold of a number. The observed effect of <something> is either: (1) appearance of cause & effect is actually not there and just random chance or coincidence (2) a real cause & effect and not random chance Yes.. but "statistical significance" leads to "p-hacking" and other academic fraud! (https://en.wikipedia.org/wiki/Data_dredging https://en.wikipedia.org/wiki/Data_dredging) That's true but the abuses still doesn't eliminate the need to name the concept of real cause & effect vs random chance. Whatever alternative mental framework one uses to decide which of those scenarios is happening, you will arrive at something that looks like "statistical significance" even if you don't call it that. As an analogy, even though "averages" in math is misused, it doesn't mean "averages are bs".
- tgv 2y ago
  Sorry, but statistical significance has a pretty narrow meaning, and it's wrong. > Removing "statistical significance" from the vocabulary doesn't change the underlying reality about people deciding what to do based on a threshold of a number. I'm not arguing to remove it from the vocabulary. I'm arguing that that bloomin' threshold is the source of many problems. The article also isn't about methodology, it's about the secondary school curriculum. To provide pupils with a false understanding of statistics is just bad. And to teach them why it's bad, is hard.
  - dworkr 2y ago
    [dead]
sweezyjeezy 2y ago
> Statistical significance is bullshit. Learning about it is as useful as learning about phlogiston. Ok, that's where I draw the line - statistical significance is not "bullshit" - however as you say, leaning on it too hard can cause things to break quite badly. Scientists misusing it do not negate all the medical advances we have made from moving to a significance-based system. It is an absolutely essential tool for people using statistics to understand, but its limitations must be emphasised when taught, and it must be understood that it is a tool, not a conclusion. Also other alternatives should be taught more widely (e.g. Bayesian inference).
- tgv 2y ago
  We would have made better progress if we had skipped NHST. Mind you, nor Fisher, nor Pearson were in favor of it. It is a tool that should be taught after better ways, and then only to understand the past. Like phlogiston.
roenxi 2y ago
You seem to be arguing that a grasp of statistical significance helps you interpret health studies.

by tgv 2y ago

taejavu 2y ago

Any chance you could expand on this, or point me to resources that do? It’s a very interesting take and not one I’m familiar with.

tgv 2y ago
There's a lot of technical stuff, but you could look into Ioanides articles [1], and the book Bernoulli's Falacy by Clayton [2] is a good introduction, also for people less involved in statistics. [1] https://medium.com/@d.incecushman/the-p-value-problem-how-dr-john-ioannidis-shook-up-science-20-years-ago-abd242ddfa23 https://medium.com/@d.incecushman/the-p-value-problem-how-dr... (this is just an example; I haven't read it) [2] https://cup.columbia.edu/book/bernoullis-fallacy/9780231199940 https://cup.columbia.edu/book/bernoullis-fallacy/97802311999...
TrackerFF 2y ago
Search for "p-hacking", that will give you a nice starter on the topic.

blackbear_ 2y ago

Unfortunately we will not get rid of statistical significance anytime soon. Teaching it and explaining its shortcomings is much better than sweeping it under the rug, and perhaps it will help the next generations of scientists not falling into the same trap.

reedf1 2y ago

No no no. You couldn't have gotten this more wrong. The replication crisis comes from soft sciences under-powered statistical training.

jfengel 2y ago
It comes from the fact that soft sciences deal with vast numbers of variables, far too many to control. They actually get a ton of statistics, trying to find a signal in so much noise. It would be great if human beings were more amenable to rigorous experiment. Failing that, we at least need to understand what these things do and don't actually mean. It's either that or give up trying to study people entirely.
- reedf1 2y ago
  Statistics are not values you collect - but the analysis you perform. In data with many confounding variables or many degrees of freedom, the only way to be honest about what the data shows is through statistics. Statistics is what allows you to filter signal from noise. Statistical significance is a very useful tool, you just have to be honest about what it represents and under what conditions this breaks down.
  - tgv 2y ago
    A statistic is an aggregate number, not an analysis in its own. There may be some linguistic confusion here. > Statistical significance is a very useful tool What for? It answers a question in which nobody is interested. What can it do that Bayesian tools can't?
llamaimperative 2y ago
No it doesn’t. Common meme around here but the hard sciences are also rife with unreproducible studies. It’s an incentives problem.
- reedf1 2y ago
  I actually agree - it's part incentives. And it is important for hard sciences to learn these lessons as well.

jasode 2y ago

>Statistical significance is bullshit. Learning about it is as useful as learning about phlogiston. The term-of-art "statistical significance" isn't bs because we ultimately have to choose what to pay attention to. Removing "statistical significance" from the vocabulary doesn't change the underlying reality about people deciding what to do based on a threshold of a number. The observed effect of <something> is either: (1) appearance of cause & effect is actually not there and just random chance or coincidence (2) a real cause & effect and not random chance Yes.. but "statistical significance" leads to "p-hacking" and other academic fraud! (https://en.wikipedia.org/wiki/Data_dredging https://en.wikipedia.org/wiki/Data_dredging) That's true but the abuses still doesn't eliminate the need to name the concept of real cause & effect vs random chance. Whatever alternative mental framework one uses to decide which of those scenarios is happening, you will arrive at something that looks like "statistical significance" even if you don't call it that. As an analogy, even though "averages" in math is misused, it doesn't mean "averages are bs".

tgv 2y ago
Sorry, but statistical significance has a pretty narrow meaning, and it's wrong. > Removing "statistical significance" from the vocabulary doesn't change the underlying reality about people deciding what to do based on a threshold of a number. I'm not arguing to remove it from the vocabulary. I'm arguing that that bloomin' threshold is the source of many problems. The article also isn't about methodology, it's about the secondary school curriculum. To provide pupils with a false understanding of statistics is just bad. And to teach them why it's bad, is hard.
- dworkr 2y ago
  [dead]

sweezyjeezy 2y ago

> Statistical significance is bullshit. Learning about it is as useful as learning about phlogiston. Ok, that's where I draw the line - statistical significance is not "bullshit" - however as you say, leaning on it too hard can cause things to break quite badly. Scientists misusing it do not negate all the medical advances we have made from moving to a significance-based system. It is an absolutely essential tool for people using statistics to understand, but its limitations must be emphasised when taught, and it must be understood that it is a tool, not a conclusion. Also other alternatives should be taught more widely (e.g. Bayesian inference).

tgv 2y ago
We would have made better progress if we had skipped NHST. Mind you, nor Fisher, nor Pearson were in favor of it. It is a tool that should be taught after better ways, and then only to understand the past. Like phlogiston.

roenxi 2y ago

You seem to be arguing that a grasp of statistical significance helps you interpret health studies.