3 ms·

>The results are based on a grand total of 25 people in the psilocybin group and 21 people in the SSRI group. Statistical significance is based on sample size,

>The results are based on a grand total of 25 people in the psilocybin group and 21 people in the SSRI group. Statistical significance is based on sample size, and is independent of population size. Let that sink in, doesn't matter if your population is 100K or 8 billion, when you sample you are trying to understand the probabilities in your sample, not in the population. Therefore, (think about the birthday paradox, doesn't matter how many people in the world, a few dozen in the room with you is an adequate sample), it should not surprise you that statistical significance is achieved through a much smaller sample size than most non-statisticians have intuition for.

seeknotfind 2y ago
The issue your parent comment raises is that this random group may not represent someone that's trying to evaluate what techniques can help them, regardless of the statistical significance within that group. It makes me sad that they introduce other random variables, as incontrovertible evidence of efficacy could help a lot of people!
- fsckboy 2y ago
  except it was completely non-specific about what those flaws might be, simply saying "sample is too small" when the sample is not a priori too small for a properly designed study of something that has "noticeable" effect. For example, "does alcohol get people drunk" is not hard to show on a sample of 10 people.
Wytwwww 2y ago
> few dozen in the room with you is an adequate sample It's not, though? Unless you're fine with a very high margin of error... Also the sample in studies like this is hardly ever close to being random anyway. > it should not surprise you that statistical significance is achieved through a much smaller sample size Sure... just with a very low confidence level.
- braiamp 2y ago
  Margin of error only occurs when you expect high variability and a very large population of measurements. When the set of potential measurements is of size 2, having enough statistical power can be achieved with very low sample size. Anyways, many results have survived stage 4 analysis (real world, observational, administrative documents sourced), even when their OG sample size is relatively small.
  - shakna 2y ago
    Statistical analysis in psychology is traditionally very poor, because there is an exceptionally high amount of variability. This is a known problem.
    - braiamp 2y ago
      No, psychology studies have been traditionally very poor because most of the theory was literally out of someone ass, without proper design. Newer studies have been shown to be very robust, public freakout notwithstanding.
      - shakna 2y ago
        Newer studies [0] show us that a) p-hacking is still alive and well in the psychological community. b) p-curves are not sufficient for detecting this. That isn't a lack of proper design. It's a case of statistics being abused to show significance when there is none. [0] https://psycnet.apa.org/doi/10.1027/2151-2604/a000383 https://psycnet.apa.org/doi/10.1027/2151-2604/a000383
TeaBrain 2y ago
>Therefore, (think about the birthday paradox, doesn't matter how many people in the world, a few dozen in the room with you is an adequate sample), it should not surprise you that statistical significance is achieved through a much smaller sample size than most non-statisticians have intuition for. This response on the supposed the lack of importance of a sample size is completely wrong on just about every claim. The parent comment had a valid point. Just because a population may fit a certain distribution, does not mean that any given sample size will also fit that distribution. Samples are used to ideally create a representative group of a population, that is smaller than the population. However, the sample size required to come close to a representative distribution can vary between populations and variables being examined. Also, using the birthday paradox is a terrible example and has nothing to do with statistical significance, as the so-called birthday paradox is just a simple function.
- braiamp 2y ago
  Except that sample size doesn't matter if the set of potential results/measurements of the dependent variable are very large. Someone pointed out that you can demonstrate that alcohol impairs executive functions, balance, etc. with a very small sample size, because the effects would be so large and evident that your statistical power would be. On very large variance, where the results are dichotomous in nature (can a subject walk straight in a 10 meters line, without walking outside: yes/no) can have a very small sample size. Use this calculator, set options to: two independent groups, dichotomous, group 1 = 90%, group 2 = 10%, incidence, enrollment ratio = 1, alpha = 0.05 and power = 80%. The sample size is 10, 5 for each group. https://clincalc.com/stats/samplesize.aspx https://clincalc.com/stats/samplesize.aspx
  - TeaBrain 2y ago
    That other comment on alcohol is by the same guy that made the comment that I responded to here. The same issue there is that result reliability can be influenced by the effect size of the variables being tested, but this still is far from a guarantee that the results will generalize, which is more likely to be an issue with a smaller sample. An issue with the comment I previously responded to, as I mentioned above, is that they made it out as if a small sample size could be reliable to determine a reliable statistical significance irrespective of the variables under study and tried to prove this by using an absurd analogy between statistical significance and the birthday paradox. The problem with their attempted point was that it didn't even respond to the comment above it, which pointed out that a high statistical significance is not a guarantee of reproducibility, especially with a low sample size.
risenshinetech 2y ago
This comment is a shining example of the phrase "not even wrong".
- TeaBrain 2y ago
  Just the first line alone "Statistical significance is based on sample size, and is independent of population size", was bizarrely silly in its description of the use of sample sizes, before they even got to their nonsensical analogy to the birthday paradox.
AbstractH24 2y ago
A larger sample size isn't inherently better, but if a large sample size from a diverse enough pool of people can be used to eliminate and/or identify confounding variable and distortions.

seeknotfind 2y ago

The issue your parent comment raises is that this random group may not represent someone that's trying to evaluate what techniques can help them, regardless of the statistical significance within that group. It makes me sad that they introduce other random variables, as incontrovertible evidence of efficacy could help a lot of people!

fsckboy 2y ago
except it was completely non-specific about what those flaws might be, simply saying "sample is too small" when the sample is not a priori too small for a properly designed study of something that has "noticeable" effect. For example, "does alcohol get people drunk" is not hard to show on a sample of 10 people.

Wytwwww 2y ago

> few dozen in the room with you is an adequate sample It's not, though? Unless you're fine with a very high margin of error... Also the sample in studies like this is hardly ever close to being random anyway. > it should not surprise you that statistical significance is achieved through a much smaller sample size Sure... just with a very low confidence level.

braiamp 2y ago
Margin of error only occurs when you expect high variability and a very large population of measurements. When the set of potential measurements is of size 2, having enough statistical power can be achieved with very low sample size. Anyways, many results have survived stage 4 analysis (real world, observational, administrative documents sourced), even when their OG sample size is relatively small.
- shakna 2y ago
  Statistical analysis in psychology is traditionally very poor, because there is an exceptionally high amount of variability. This is a known problem.
  - braiamp 2y ago
    No, psychology studies have been traditionally very poor because most of the theory was literally out of someone ass, without proper design. Newer studies have been shown to be very robust, public freakout notwithstanding.
    - shakna 2y ago
      Newer studies [0] show us that a) p-hacking is still alive and well in the psychological community. b) p-curves are not sufficient for detecting this. That isn't a lack of proper design. It's a case of statistics being abused to show significance when there is none. [0] https://psycnet.apa.org/doi/10.1027/2151-2604/a000383 https://psycnet.apa.org/doi/10.1027/2151-2604/a000383

TeaBrain 2y ago

>Therefore, (think about the birthday paradox, doesn't matter how many people in the world, a few dozen in the room with you is an adequate sample), it should not surprise you that statistical significance is achieved through a much smaller sample size than most non-statisticians have intuition for. This response on the supposed the lack of importance of a sample size is completely wrong on just about every claim. The parent comment had a valid point. Just because a population may fit a certain distribution, does not mean that any given sample size will also fit that distribution. Samples are used to ideally create a representative group of a population, that is smaller than the population. However, the sample size required to come close to a representative distribution can vary between populations and variables being examined. Also, using the birthday paradox is a terrible example and has nothing to do with statistical significance, as the so-called birthday paradox is just a simple function.

braiamp 2y ago
Except that sample size doesn't matter if the set of potential results/measurements of the dependent variable are very large. Someone pointed out that you can demonstrate that alcohol impairs executive functions, balance, etc. with a very small sample size, because the effects would be so large and evident that your statistical power would be. On very large variance, where the results are dichotomous in nature (can a subject walk straight in a 10 meters line, without walking outside: yes/no) can have a very small sample size. Use this calculator, set options to: two independent groups, dichotomous, group 1 = 90%, group 2 = 10%, incidence, enrollment ratio = 1, alpha = 0.05 and power = 80%. The sample size is 10, 5 for each group. https://clincalc.com/stats/samplesize.aspx https://clincalc.com/stats/samplesize.aspx
- TeaBrain 2y ago
  That other comment on alcohol is by the same guy that made the comment that I responded to here. The same issue there is that result reliability can be influenced by the effect size of the variables being tested, but this still is far from a guarantee that the results will generalize, which is more likely to be an issue with a smaller sample. An issue with the comment I previously responded to, as I mentioned above, is that they made it out as if a small sample size could be reliable to determine a reliable statistical significance irrespective of the variables under study and tried to prove this by using an absurd analogy between statistical significance and the birthday paradox. The problem with their attempted point was that it didn't even respond to the comment above it, which pointed out that a high statistical significance is not a guarantee of reproducibility, especially with a low sample size.

risenshinetech 2y ago

This comment is a shining example of the phrase "not even wrong".

TeaBrain 2y ago
Just the first line alone "Statistical significance is based on sample size, and is independent of population size", was bizarrely silly in its description of the use of sample sizes, before they even got to their nonsensical analogy to the birthday paradox.

AbstractH24 2y ago

A larger sample size isn't inherently better, but if a large sample size from a diverse enough pool of people can be used to eliminate and/or identify confounding variable and distortions.