4 ms·

A good way to address these issues is to frame all experiments as multilevel models. See [1] for a long discussion from Andrew Gelman et al on why this is advis

by nextos 4y ago

A good way to address these issues is to frame all experiments as multilevel models. See [1] for a long discussion from Andrew Gelman et al on why this is advisable. Surprisingly, many people working in statistics still ignore the James-Stein theorem, which provides a theoretical justification for multilevel models. In layman terms, said theorem shows that if you are simultaneously estimating many random variables you should borrow information across variables [2]. Estimating them one by one is suboptimal and does not minimize the global mean squared error. Multilevel models "shrink" individual effect sizes by looking at the overall distribution of effect sizes and provide much more realistic estimates. [1] https://arxiv.org/abs/0907.2478 https://arxiv.org/abs/0907.2478 [2] https://en.wikipedia.org/wiki/Stein%27s_example https://en.wikipedia.org/wiki/Stein%27s_example

kimukasetsu 4y ago
This. Multilevel models relax the assumption of independent observations by specifying that the measures of repeated experimental units are dependent on each other. It's a way of telling your model that it has less information than it would have if all observations came from independent units. Therefore, standard errors of effects are usually larger. Otherwise, they are biased [1]. Since most researches are not aware of multilevel models, they design their experiments and aggregate their data to fit the independence assumption, which is rarely a good idea. Many are not even aware of modeling beyond hypothesis tests, and are unable or unwilling to adjust their analysis for confounding factors or non-sampling errors that arise due to experiment design flaws. Also, p-values should be deprecated, since a) nil hypothesis are strawmen at best and false by definition at worst [2] and b) they incentivize researchers to not think hard about effect sizes and uncertainty in their problems. [1] https://academic.oup.com/biomet/article/73/1/13/246001 https://academic.oup.com/biomet/article/73/1/13/246001 [2] http://www.stat.columbia.edu/~gelman/research/published/failure_of.pdf http://www.stat.columbia.edu/~gelman/research/published/fail...
- nextos 4y ago
  Many articles in e.g. Nature Genetics are cheating by hacking their p-values. These hacks would be much harder to get away with if, as a starter, they were asked to use hierarchical models and continuous explanatory variables, whenever possible. The article from Andrew Gelman you cited explains this quite well. In general the review articles and books he has co-authored are incredibly helpful to learn how to avoid common issues that plague statistical inference. We need to shift away from null hypotheses and p-values towards generative models, model selection and effect sizes. It leads to much more robust inference.

by nextos 4y ago

kimukasetsu 4y ago

This. Multilevel models relax the assumption of independent observations by specifying that the measures of repeated experimental units are dependent on each other. It's a way of telling your model that it has less information than it would have if all observations came from independent units. Therefore, standard errors of effects are usually larger. Otherwise, they are biased [1]. Since most researches are not aware of multilevel models, they design their experiments and aggregate their data to fit the independence assumption, which is rarely a good idea. Many are not even aware of modeling beyond hypothesis tests, and are unable or unwilling to adjust their analysis for confounding factors or non-sampling errors that arise due to experiment design flaws. Also, p-values should be deprecated, since a) nil hypothesis are strawmen at best and false by definition at worst [2] and b) they incentivize researchers to not think hard about effect sizes and uncertainty in their problems. [1] https://academic.oup.com/biomet/article/73/1/13/246001 https://academic.oup.com/biomet/article/73/1/13/246001 [2] http://www.stat.columbia.edu/~gelman/research/published/failure_of.pdf http://www.stat.columbia.edu/~gelman/research/published/fail...

nextos 4y ago
Many articles in e.g. Nature Genetics are cheating by hacking their p-values. These hacks would be much harder to get away with if, as a starter, they were asked to use hierarchical models and continuous explanatory variables, whenever possible. The article from Andrew Gelman you cited explains this quite well. In general the review articles and books he has co-authored are incredibly helpful to learn how to avoid common issues that plague statistical inference. We need to shift away from null hypotheses and p-values towards generative models, model selection and effect sizes. It leads to much more robust inference.