3 ms·
Sure, the chance of (unbalanced) “confounders” can be high in a small sample. But the statistical machinery you’re using is designed to handle that. If you try
by bjornsing 2y ago
Sure, the chance of (unbalanced) “confounders” can be high in a small sample. But the statistical machinery you’re using is designed to handle that. If you try to avoid it then you’re violating the underlying assumptions of that statistical machinery, no?
- mjburgess 2y agoIf the confounders are fat-tail distributed etc. then arbitrarily large samples can still be inadequate. The idea that even thousands a data points in subgroups are going to be 'well mixed' relies on extremely strong assumptions about the distribution of those traits.
- xapata 2y agoJust assumptions, not extremely strong assumptions. Ordinary least squares performs well in a large variety of cases.
- usgroup 2y agoIt doesn't work that way, and it is common to test for known imbalances, or to implement stratified sampling. As another poster said, confounders do not have to be conveniently distributed, even though that is typically assumed. It could be that a big effect is present in a confounder but collected only every N samples because it is sparse. In which case you could have large, randomly allocated but confounded samples. All these things can and do happen in randomised experiments, but it is still orders of magnitude more interpretable than what can happen in observational studies.
- bjornsing 2y agoSure, if you correctly implement stratified sampling that’s fine. But then you’ve replaced the statistical machinery with one that takes this balancing into account. If on the other hand you just redo the sampling until it seems balanced then you’ve violated the assumptions behind the standard statistical tooling.