10 ms·

I remember my high school AP Psychology teacher mocking p=0.05 as practically meaningless. In retrospect it's funny for a psychologist to say that, but I guess

by User23 3y ago

I remember my high school AP Psychology teacher mocking p=0.05 as practically meaningless. In retrospect it's funny for a psychologist to say that, but I guess it was because he was from the more empirically minded behaviorist cognitive school and from time to time they have done actual rigorous experiments[1] (in rodents). [1] For example as described by Feynman in Cargo Cult Science.

lisper 3y ago
The problem is two-fold: 1. p=0.05 means that one result in 20 is going to be the result of chance. 2. It's generally pretty easy (especially in psychology) to do 20 experiments, cherry-pick -- and publish! -- the p=0.05 result, and throw away the others. The result is that published p=0.05 results are much more likely than 1 in 20 to be the result of chance.
- dragontamer 3y ago
  So run a meta-study upon the results published by a set of authors and double-check to make sure that their results are normally distributed across the p-values associated with their studies. These problems are solved problems in the scientific community. Just announce that regular meta-studies will be done, expectations for authors to be normally distributed is published, and publicly show off the meta-study. ------------- In any case, the discussion point you're making is well beyond the high-school level needed for a general education. If someone needs to run their own experiment (A/B testing upon their website) and cannot afford a proper set of tests/statistics, they should instead rely upon high-school level heuristics to design their personal studies. This isn't a level of study about analyzing other people's results and finding flaws in other people's (possibly maliciously seeded) results. This is a heuristic about how to run your own experiments and how to prove something to yourself at a 95% confidence level. If you want to get published in the scientific community, the level of rigor is much higher of course, but no one tries to publish a scientific paper on just a high school education (which is where I was aiming my original comment at).
  - User23 3y ago
    There’s a professor of Human Evolutionary Biology at Harvard who only has a high school diploma[1]. Needless to say he’s been published and cited many times over. [1] https://theconversation.com/profiles/louis-liebenberg-1226800 https://theconversation.com/profiles/louis-liebenberg-122680...
    - withinboredom 3y ago
      I don't know whether you're mocking them or being supportive of them or just stating a fact. Either way, education level has no bearing on subject knowledge. I know more about how computers, compilers, and software algorithms work than most post-docs and professors that I've run into in those subjects. Am I smarter than them? Nope. Do I know as many fancy big words as them? Nope. Do I care about results and communicating complex topics to normal people? Yep. Do I care more about making the company money than chasing some bug-bear to go on my resume? Yep. I fucking hate school and have no desire to ever go back. I can't put up with the bullshit, so I dropped out; I just never stopped studying and I don't need a piece of paper to affirm that fact.
      - withinboredom 3y ago
        To the people downvoting, at least rebuttal.
  - lisper 3y ago
    First, I was specifically responding to this: > I remember my high school AP Psychology teacher mocking p=0.05 as practically meaningless. and trying to explain why the OP's teacher was probably right. Second: > So run a meta-study upon the results published by a set of authors and double-check to make sure that their results are normally distributed across the p-values associated with their studies. That won't work, especially if you only run the meta-study on published results because it is all but impossible to get negative results published. Authors don't need to cherry-pick, the peer-review system does it for them. > These problems are solved problems in the scientific community. No, they aren't. These are social and political problems, not mathematical ones. And the scientific community is pretty bad at solving those. > the discussion point you're making is well beyond the high-school level needed for a general education I strongly disagree. I think everyone needs to understand this so they can approach scientific claims with an appropriate level of skepticism. Understanding how the sausage is made is essential to understanding science. And BTW, I am not some crazy anti-vaxxer climate-change denialist flat-earther. I was an academic researcher for 15 years -- in a STEM field, not psychology, and even that was sufficiently screwed up to make me change my career. I have advocated for science and the scientific method for decades. It's not science that's broken, it's the academic peer-review system, which is essentially unchanged since it was invented in the 19th century. That is what needs to change. And that has nothing to do with math and everything to do with politics and economics.
- Viliam1234 3y ago
  > p=0.05 means that one result in 20 is going to be the result of chance. You made the same mistake most people make here: you turned the arrow of the implication. It is not "successful experiment implies chance (probability 5%)" but "chance implies successful experiment (probability 5%)". What does that mean in practice? Imagine a hypothetical scientist that is fundamentally confused about something important, so all hypotheses they generate are false. Yet, using p=0.05, 5% of those hypotheses will be "confirmed experimentally". In that case, it is not 5% of the "experimentally confirmed" hypotheses that are wrong -- it is full 100%. Even without any cherry-picking. The problem is not that p=0.05 is too high. The problem is, it doesn't actually mean what most people believe it means.
  - lisper 3y ago
    I think we're actually in violent agreement here, but I just wasn't precise enough. Let me try again: p=0.05 means that one POSITIVE result in 20 is going to be the result of chance and not causality In other words: if I have some kind of intervention or treatment, and that intervention or treatment produces some result in a test group relative to a control group with p=0.05, then the odds of getting that result simply by chance and not because the treatment or intervention actually had an effect are 5%. The practical effect of this is that there are two different ways of getting a p=0.05 result: 1. Find a treatment or intervention that actually works or 2. Test ~20 different (useless) interventions. Or test one useless intervention ~20 times. A single p=0.05 result in isolation is useless because there is no way to know which of the two methods produced it. This is why replication is so important. The odds of getting a p=0.05 result by chance is 5%. But the odds of getting TWO of them in sequential trials is 0.25%, and the odds of a positive result being the result of pure chance decrease exponentially with each subsequent replication.
    - thaumasiotes 3y ago
      > Let me try again: > p=0.05 means that one POSITIVE result in 20 is going to be the result of chance and not causality No, you still didn't get it. In the example above, a full 100% of positive results, 20 out of every 20, are the result of chance and not causality. Your followup discussion is better, but your statement at the top doesn't work. (Note also that there is an interaction between p-threshold and sample size which guarantees that, if you're investigating an effect that your sample size is not large enough to detect, any statistically significant result that you get will be several times stronger than the actual effect. They're also quite likely to have the wrong sign.)
tgv 3y ago
The observation above is simply true. If you toss a coin 30 times, there's about a 5% chance that you'll end up with 10-20 ratio or one more extreme. NHST testing inverts the probability logic, makes the 5% holy, and skims over the high probability of finding something that is not equal to a specific value. That procedure is then used for theory confirmation, while it was (in another form) meant for falsification. Everything is wrong about it, even if the experimental method is flawless. Hence the reproducibility crisis.

by User23 3y ago

lisper 3y ago

The problem is two-fold: 1. p=0.05 means that one result in 20 is going to be the result of chance. 2. It's generally pretty easy (especially in psychology) to do 20 experiments, cherry-pick -- and publish! -- the p=0.05 result, and throw away the others. The result is that published p=0.05 results are much more likely than 1 in 20 to be the result of chance.

dragontamer 3y ago
So run a meta-study upon the results published by a set of authors and double-check to make sure that their results are normally distributed across the p-values associated with their studies. These problems are solved problems in the scientific community. Just announce that regular meta-studies will be done, expectations for authors to be normally distributed is published, and publicly show off the meta-study. ------------- In any case, the discussion point you're making is well beyond the high-school level needed for a general education. If someone needs to run their own experiment (A/B testing upon their website) and cannot afford a proper set of tests/statistics, they should instead rely upon high-school level heuristics to design their personal studies. This isn't a level of study about analyzing other people's results and finding flaws in other people's (possibly maliciously seeded) results. This is a heuristic about how to run your own experiments and how to prove something to yourself at a 95% confidence level. If you want to get published in the scientific community, the level of rigor is much higher of course, but no one tries to publish a scientific paper on just a high school education (which is where I was aiming my original comment at).
- User23 3y ago
  There’s a professor of Human Evolutionary Biology at Harvard who only has a high school diploma[1]. Needless to say he’s been published and cited many times over. [1] https://theconversation.com/profiles/louis-liebenberg-1226800 https://theconversation.com/profiles/louis-liebenberg-122680...
  - withinboredom 3y ago
    I don't know whether you're mocking them or being supportive of them or just stating a fact. Either way, education level has no bearing on subject knowledge. I know more about how computers, compilers, and software algorithms work than most post-docs and professors that I've run into in those subjects. Am I smarter than them? Nope. Do I know as many fancy big words as them? Nope. Do I care about results and communicating complex topics to normal people? Yep. Do I care more about making the company money than chasing some bug-bear to go on my resume? Yep. I fucking hate school and have no desire to ever go back. I can't put up with the bullshit, so I dropped out; I just never stopped studying and I don't need a piece of paper to affirm that fact.
    - withinboredom 3y ago
      To the people downvoting, at least rebuttal.
- lisper 3y ago
  First, I was specifically responding to this: > I remember my high school AP Psychology teacher mocking p=0.05 as practically meaningless. and trying to explain why the OP's teacher was probably right. Second: > So run a meta-study upon the results published by a set of authors and double-check to make sure that their results are normally distributed across the p-values associated with their studies. That won't work, especially if you only run the meta-study on published results because it is all but impossible to get negative results published. Authors don't need to cherry-pick, the peer-review system does it for them. > These problems are solved problems in the scientific community. No, they aren't. These are social and political problems, not mathematical ones. And the scientific community is pretty bad at solving those. > the discussion point you're making is well beyond the high-school level needed for a general education I strongly disagree. I think everyone needs to understand this so they can approach scientific claims with an appropriate level of skepticism. Understanding how the sausage is made is essential to understanding science. And BTW, I am not some crazy anti-vaxxer climate-change denialist flat-earther. I was an academic researcher for 15 years -- in a STEM field, not psychology, and even that was sufficiently screwed up to make me change my career. I have advocated for science and the scientific method for decades. It's not science that's broken, it's the academic peer-review system, which is essentially unchanged since it was invented in the 19th century. That is what needs to change. And that has nothing to do with math and everything to do with politics and economics.
Viliam1234 3y ago
> p=0.05 means that one result in 20 is going to be the result of chance. You made the same mistake most people make here: you turned the arrow of the implication. It is not "successful experiment implies chance (probability 5%)" but "chance implies successful experiment (probability 5%)". What does that mean in practice? Imagine a hypothetical scientist that is fundamentally confused about something important, so all hypotheses they generate are false. Yet, using p=0.05, 5% of those hypotheses will be "confirmed experimentally". In that case, it is not 5% of the "experimentally confirmed" hypotheses that are wrong -- it is full 100%. Even without any cherry-picking. The problem is not that p=0.05 is too high. The problem is, it doesn't actually mean what most people believe it means.
- lisper 3y ago
  I think we're actually in violent agreement here, but I just wasn't precise enough. Let me try again: p=0.05 means that one POSITIVE result in 20 is going to be the result of chance and not causality In other words: if I have some kind of intervention or treatment, and that intervention or treatment produces some result in a test group relative to a control group with p=0.05, then the odds of getting that result simply by chance and not because the treatment or intervention actually had an effect are 5%. The practical effect of this is that there are two different ways of getting a p=0.05 result: 1. Find a treatment or intervention that actually works or 2. Test ~20 different (useless) interventions. Or test one useless intervention ~20 times. A single p=0.05 result in isolation is useless because there is no way to know which of the two methods produced it. This is why replication is so important. The odds of getting a p=0.05 result by chance is 5%. But the odds of getting TWO of them in sequential trials is 0.25%, and the odds of a positive result being the result of pure chance decrease exponentially with each subsequent replication.
  - thaumasiotes 3y ago
    > Let me try again: > p=0.05 means that one POSITIVE result in 20 is going to be the result of chance and not causality No, you still didn't get it. In the example above, a full 100% of positive results, 20 out of every 20, are the result of chance and not causality. Your followup discussion is better, but your statement at the top doesn't work. (Note also that there is an interaction between p-threshold and sample size which guarantees that, if you're investigating an effect that your sample size is not large enough to detect, any statistically significant result that you get will be several times stronger than the actual effect. They're also quite likely to have the wrong sign.)

tgv 3y ago

The observation above is simply true. If you toss a coin 30 times, there's about a 5% chance that you'll end up with 10-20 ratio or one more extreme. NHST testing inverts the probability logic, makes the 5% holy, and skims over the high probability of finding something that is not equal to a specific value. That procedure is then used for theory confirmation, while it was (in another form) meant for falsification. Everything is wrong about it, even if the experimental method is flawless. Hence the reproducibility crisis.