3 ms·
>We expect large overestimates of effect size in studies as a function of sample size. In statistical genetics this goes by the memorable name of the Beavis eff
by hammock 5y ago
>We expect large overestimates of effect size in studies as a function of sample size. In statistical genetics this goes by the memorable name of the Beavis effect.... Beavis effect evaporates with much larger sample size.
This idea would seem to suggest special restraint when negatively interpreting a lot of A/B test replication. It's common to get a good result with a small sample, retest with a much larger audience and get a less-impressive result. If I understand, that outcome should not necessarily be discouraging for the success of the experiment. No?
- SubiculumCode 5y agoNothing works against p-values than a large sample, no matter what the power analysis says.-Folk Science Wisdom 101
- nextos 5y agoShrinking would work to some extent, by borrowing information from other variables. This is better done in a Bayesian context with multilevel models. Also weakly informative priors, which have regularizing effects. Actually these two things are a big focus area of Stan, which is the probabilistic programming language built by the group which writes that blog.