9 ms·

Dispelling Myths about Randomisation

45 points by domofutu 2y ago

ggm 2y ago
I'd love to read similar pointers on sampling in the stream: unknown population size, semi continuous sampling. The limits of your inference and also it's strength.
- tjpnz 2y ago
  Reservoir Sampling?
dash2 2y ago
There's a lot of cases of "impure randomization" when I think balance tests are a good idea - for example, natural experiments.
IshKebab 2y ago
I feel like this article is way too confusingly written for anyone that would actually benefit from it to understand it.
- usgroup 2y ago
  Generally speaking, I've come to expect that from articles with "myths about" in the title outside of the context of the classics or anthropology.
usgroup 2y ago
> When we randomise people, these influences will still be operating on the outcome, which will vary across the people randomised to our conditions. Does randomisation mean that all these different effects are balanced somehow? No – not least because confounders do not exist in experimental studies! This is for the simple reason that a confounder is something that affects both the exposure and the outcome, and in an experimental (i.e., randomised) study we test for a difference in our outcome between the two randomised groups. I do not think this sort of word-play is useful. If your random samples are small (and even if statistically adequate) the chance of confounders in one or both groups of an A/B can be relatively high, even though the selection procedure for treatment is random. So, "No – not least because confounders do not exist in experimental studies!" is misleading if it is expected on the basis that randomisation of treatment allocation somehow makes confounding impossible. That the possibility of confounding is equally likely in both branches remains true for all sample sizes of an A/B where #A=#B and allocation is random. So, in my opinion, not a myth.
- bjornsing 2y ago
  Sure, the chance of (unbalanced) “confounders” can be high in a small sample. But the statistical machinery you’re using is designed to handle that. If you try to avoid it then you’re violating the underlying assumptions of that statistical machinery, no?
  - mjburgess 2y ago
    If the confounders are fat-tail distributed etc. then arbitrarily large samples can still be inadequate. The idea that even thousands a data points in subgroups are going to be 'well mixed' relies on extremely strong assumptions about the distribution of those traits.
    - xapata 2y ago
      Just assumptions, not extremely strong assumptions. Ordinary least squares performs well in a large variety of cases.
  - usgroup 2y ago
    It doesn't work that way, and it is common to test for known imbalances, or to implement stratified sampling. As another poster said, confounders do not have to be conveniently distributed, even though that is typically assumed. It could be that a big effect is present in a confounder but collected only every N samples because it is sparse. In which case you could have large, randomly allocated but confounded samples. All these things can and do happen in randomised experiments, but it is still orders of magnitude more interpretable than what can happen in observational studies.
    - bjornsing 2y ago
      Sure, if you correctly implement stratified sampling that’s fine. But then you’ve replaced the statistical machinery with one that takes this balancing into account. If on the other hand you just redo the sampling until it seems balanced then you’ve violated the assumptions behind the standard statistical tooling.
CodeMage 2y ago
> This approach has given us the oft-repeated (but slightly fallacious) line that 'correlation does not imply causation' (I would say that it can imply it, just often not much more). I always thought that this expression used the word "imply" in its mathematical sense, i.e. "correlation is not a sufficient condition for causation". Did I get that wrong, or did the author?
mjburgess 2y ago
> If our randomisation mechanism influences our exposure (which by definition it should) and nothing else (ditto), and we see a difference in our outcome, then this difference must have been caused by the exposure we manipulated. Just false. It's kinda sad and funny, I guess, that this article exists on a psychology site. The outcome can be caused by uncontrolled factors, and randomisation is a very limited technique for causal control. Eg., suppose traits T1...Tn are distributed throughout a population in a powerlaw fashion (eg., Zipf), so that Ti is had by (1/2^i)%. Suppose each has an effect on the relevant outcome of some non-trivial %. Then even for tens of thousands of people, you can still get "random" divisions of a population where, say, one half has all the T9 traits -- and thereby the "effect" is just due to this random segmentation. In the hard sciences causes can actually be controlled, by eg., literally placing your hand on some part of the experiment to stop it moving (or equivalent). This is the "third option" which is missing from the intro: actual science. When causes cannot be controlled, all inferences are highly provisional (, defeasible, suspect, ...). And i'd prefer we used a whole different set of methods, terminology, etc. in this case -- "speculative science" or some such
- RandomLensman 2y ago
  What is a hard science then? Not all of physics is in the "can control causes" bucket, same for chemistry, biology, medicine, ...
  - mjburgess 2y ago
    A hard science is one where you can control causes, or have an equivalent operationalizable theory of causal mechanisms, or there is a feasible strategy for the development of either. Science becomes a half-empirical speculative activity in all other cases. So yes, a lot of genetics (but not all), and so on, is a speculative science. Eg., very rarely some finite number of known genes can be given a known causal mechanism (etc.) -- in this cases, you have a Science.
    - acdha 2y ago
      This seems to be more exposing that “hard science” is an arbitrary label which is not especially useful. Genomics doesn’t magically become non-empirical because you don’t control all of the variables - empiricism is about observation, not control – and all science has some level of speculation because the entire concept is based on collecting evidence to test theories. Just because a biologist or psychologist works with more complex problems doesn’t change that underlying mechanism.
rwilson4 2y ago
As someone who works with causal inference most days, I expected to find much fault with this article. I was pleasantly surprised to find how rigorous the article is, despite some other comments here. For more information on the role of randomization in causal inference (experimental and observational), I recommend the books by Paul Rosenbaum, especially, "Observation and Experiment".

45 points by domofutu 2y ago

ggm 2y ago

I'd love to read similar pointers on sampling in the stream: unknown population size, semi continuous sampling. The limits of your inference and also it's strength.

tjpnz 2y ago
Reservoir Sampling?

dash2 2y ago

There's a lot of cases of "impure randomization" when I think balance tests are a good idea - for example, natural experiments.

IshKebab 2y ago

I feel like this article is way too confusingly written for anyone that would actually benefit from it to understand it.

usgroup 2y ago
Generally speaking, I've come to expect that from articles with "myths about" in the title outside of the context of the classics or anthropology.

usgroup 2y ago

> When we randomise people, these influences will still be operating on the outcome, which will vary across the people randomised to our conditions. Does randomisation mean that all these different effects are balanced somehow? No – not least because confounders do not exist in experimental studies! This is for the simple reason that a confounder is something that affects both the exposure and the outcome, and in an experimental (i.e., randomised) study we test for a difference in our outcome between the two randomised groups. I do not think this sort of word-play is useful. If your random samples are small (and even if statistically adequate) the chance of confounders in one or both groups of an A/B can be relatively high, even though the selection procedure for treatment is random. So, "No – not least because confounders do not exist in experimental studies!" is misleading if it is expected on the basis that randomisation of treatment allocation somehow makes confounding impossible. That the possibility of confounding is equally likely in both branches remains true for all sample sizes of an A/B where #A=#B and allocation is random. So, in my opinion, not a myth.

bjornsing 2y ago
Sure, the chance of (unbalanced) “confounders” can be high in a small sample. But the statistical machinery you’re using is designed to handle that. If you try to avoid it then you’re violating the underlying assumptions of that statistical machinery, no?
- mjburgess 2y ago
  If the confounders are fat-tail distributed etc. then arbitrarily large samples can still be inadequate. The idea that even thousands a data points in subgroups are going to be 'well mixed' relies on extremely strong assumptions about the distribution of those traits.
  - xapata 2y ago
    Just assumptions, not extremely strong assumptions. Ordinary least squares performs well in a large variety of cases.
- usgroup 2y ago
  It doesn't work that way, and it is common to test for known imbalances, or to implement stratified sampling. As another poster said, confounders do not have to be conveniently distributed, even though that is typically assumed. It could be that a big effect is present in a confounder but collected only every N samples because it is sparse. In which case you could have large, randomly allocated but confounded samples. All these things can and do happen in randomised experiments, but it is still orders of magnitude more interpretable than what can happen in observational studies.
  - bjornsing 2y ago
    Sure, if you correctly implement stratified sampling that’s fine. But then you’ve replaced the statistical machinery with one that takes this balancing into account. If on the other hand you just redo the sampling until it seems balanced then you’ve violated the assumptions behind the standard statistical tooling.

CodeMage 2y ago

> This approach has given us the oft-repeated (but slightly fallacious) line that 'correlation does not imply causation' (I would say that it can imply it, just often not much more). I always thought that this expression used the word "imply" in its mathematical sense, i.e. "correlation is not a sufficient condition for causation". Did I get that wrong, or did the author?

mjburgess 2y ago

> If our randomisation mechanism influences our exposure (which by definition it should) and nothing else (ditto), and we see a difference in our outcome, then this difference must have been caused by the exposure we manipulated. Just false. It's kinda sad and funny, I guess, that this article exists on a psychology site. The outcome can be caused by uncontrolled factors, and randomisation is a very limited technique for causal control. Eg., suppose traits T1...Tn are distributed throughout a population in a powerlaw fashion (eg., Zipf), so that Ti is had by (1/2^i)%. Suppose each has an effect on the relevant outcome of some non-trivial %. Then even for tens of thousands of people, you can still get "random" divisions of a population where, say, one half has all the T9 traits -- and thereby the "effect" is just due to this random segmentation. In the hard sciences causes can actually be controlled, by eg., literally placing your hand on some part of the experiment to stop it moving (or equivalent). This is the "third option" which is missing from the intro: actual science. When causes cannot be controlled, all inferences are highly provisional (, defeasible, suspect, ...). And i'd prefer we used a whole different set of methods, terminology, etc. in this case -- "speculative science" or some such

RandomLensman 2y ago
What is a hard science then? Not all of physics is in the "can control causes" bucket, same for chemistry, biology, medicine, ...
- mjburgess 2y ago
  A hard science is one where you can control causes, or have an equivalent operationalizable theory of causal mechanisms, or there is a feasible strategy for the development of either. Science becomes a half-empirical speculative activity in all other cases. So yes, a lot of genetics (but not all), and so on, is a speculative science. Eg., very rarely some finite number of known genes can be given a known causal mechanism (etc.) -- in this cases, you have a Science.
  - acdha 2y ago
    This seems to be more exposing that “hard science” is an arbitrary label which is not especially useful. Genomics doesn’t magically become non-empirical because you don’t control all of the variables - empiricism is about observation, not control – and all science has some level of speculation because the entire concept is based on collecting evidence to test theories. Just because a biologist or psychologist works with more complex problems doesn’t change that underlying mechanism.

rwilson4 2y ago

As someone who works with causal inference most days, I expected to find much fault with this article. I was pleasantly surprised to find how rigorous the article is, despite some other comments here. For more information on the role of randomization in causal inference (experimental and observational), I recommend the books by Paul Rosenbaum, especially, "Observation and Experiment".