3 ms·
> Then even for tens of thousands of people, you can still get "random" divisions of a population where, say, one half has all the T9 traits -- and thereby the
by kqr 2y ago
> Then even for tens of thousands of people, you can still get "random" divisions of a population where, say, one half has all the T9 traits -- and thereby the "effect" is just due to this random segmentation.
You have chosen a very high-variance distribution for the trait, so in this experiment also the sampling error would also have very high variance - big enough to capture this effect you are talking about. The article mentions this. Random sampling does not guarantee we don't get a false positive, but it lets us quantify the probability of a false positive, and pick an acceptable risk of false positive.
This probability depends on the variance of the underlying trait: a high-variance trait takes an impossibly large sample to get the same discriminatory strength as a sample of a low-variance trait.
> In the hard sciences causes can actually be controlled, by eg., literally placing your hand on some part of the experiment to stop it moving (or equivalent).
Things can be controlled in softer sciences also, it's just that it's sometimes hard to agree on which the meaningful causal antecedents are to get results that replicate out of sample. We can run into that case in hard sciences also, and we do from time to time – the typical example is how insufficient control of unknown factors caused Newton to believe relativistic effects didn't exist.
I fail to see how your "third option" is fundamentally different from the two options already specified in the article.
- mjburgess 2y agoIn my example, the point is we do not known the trait distribution -- so how are you accounting for it? You don't see a difference between actually necessarily controlling a cause by causal intervention, vs., "somehow, hopefully, on average" possible relevant causes are controlled for?
- nerdponx 2y ago> so how are you accounting for it? One way is with modeling assumptions. Recall that the T test was invented to be used in controlled beer fermentation experiments, which I think should fall into your "hard" science category. But the T test has strict distributional assumptions, without which the test is not valid and its results are meaningless. Another way is that we can improve our modeling assumptions, or at least our interpretation of modeled results, by doing basic descriptive data analysis before experimenting. Unless you are extremely starved for data, you will very quickly notice if your data follows something like a power law distribution. Then you can adjust your experimentation and modeling accordingly, even if it's just tempering your ability to draw conclusions for a randomized controlled trial. Yet another way is with replication of experiments. We are not talking about observational data here, so these experiments can be replicated. A power-law distribution, again, would be noticeable here, and we'd see a high variance in results across individual experiments. > You don't see a difference between actually necessarily controlling a cause by causal intervention, vs., "somehow, hopefully, on average" possible relevant causes are controlled for? I don't, because "actually necessarily controlling a cause by causal intervention" does not exist in real life. It's just a matter of how much the data varies around the average case, which tends to be greater in the social sciences than in the natural sciences... on average, with plenty of variation around that average.