3 ms·

We expect large overestimates of effect size in studies as a function of sample size. In statistical genetics this goes by the memorable name of the Beavis effe

We expect large overestimates of effect size in studies as a function of sample size. In statistical genetics this goes by the memorable name of the Beavis effect (https://pubmed.ncbi.nlm.nih.gov/14704201/ https://pubmed.ncbi.nlm.nih.gov/14704201/) I once mapped a modulator of lifespan in a small family of mice precisely to a chunk of chromosome 2, but using only 24 BXD family members (https://pubmed.ncbi.nlm.nih.gov/23698443/ https://pubmed.ncbi.nlm.nih.gov/23698443/). The placement of this locus is solid, but the effect size is inflated about 3 to 5X. In recent work in mice we know the effect sizes for single longevity loci max-out at about 30 days per gene/allele, but in the small study above the effect size was about 100+ days. Eye opener for me. Beavis effect evaporates with much larger sample size. In our current research on the genetics of longevity in mice we are up to 6000+ animals and have about a dozen significant hits that collectively explain roughly 125 days of life span difference. Should be in bioRxiv in next month or two Arends DA et al).

thaumasiotes 5y ago
Related: http://datacolada.org/18 http://datacolada.org/18 > we were interested in the question of how many participants a typical between-subjects psychology study needs to have an 80% chance to detect a true effect. > Turns out you need 47 participants per cell to detect that people who like eggs eat egg salad more often than those who dislike eggs.
hammock 5y ago
>We expect large overestimates of effect size in studies as a function of sample size. In statistical genetics this goes by the memorable name of the Beavis effect.... Beavis effect evaporates with much larger sample size. This idea would seem to suggest special restraint when negatively interpreting a lot of A/B test replication. It's common to get a good result with a small sample, retest with a much larger audience and get a less-impressive result. If I understand, that outcome should not necessarily be discouraging for the success of the experiment. No?
- SubiculumCode 5y ago
  Nothing works against p-values than a large sample, no matter what the power analysis says.-Folk Science Wisdom 101
  - nextos 5y ago
    Shrinking would work to some extent, by borrowing information from other variables. This is better done in a Bayesian context with multilevel models. Also weakly informative priors, which have regularizing effects. Actually these two things are a big focus area of Stan, which is the probabilistic programming language built by the group which writes that blog.

thaumasiotes 5y ago

Related: http://datacolada.org/18 http://datacolada.org/18 > we were interested in the question of how many participants a typical between-subjects psychology study needs to have an 80% chance to detect a true effect. > Turns out you need 47 participants per cell to detect that people who like eggs eat egg salad more often than those who dislike eggs.

hammock 5y ago

>We expect large overestimates of effect size in studies as a function of sample size. In statistical genetics this goes by the memorable name of the Beavis effect.... Beavis effect evaporates with much larger sample size. This idea would seem to suggest special restraint when negatively interpreting a lot of A/B test replication. It's common to get a good result with a small sample, retest with a much larger audience and get a less-impressive result. If I understand, that outcome should not necessarily be discouraging for the success of the experiment. No?

SubiculumCode 5y ago
Nothing works against p-values than a large sample, no matter what the power analysis says.-Folk Science Wisdom 101
- nextos 5y ago
  Shrinking would work to some extent, by borrowing information from other variables. This is better done in a Bayesian context with multilevel models. Also weakly informative priors, which have regularizing effects. Actually these two things are a big focus area of Stan, which is the probabilistic programming language built by the group which writes that blog.